System paths in LD_LIBRARY_PATH

Is it acceptable to specify system paths (/lib, /usr/lib, contents of ld.so.conf.d, etc.) in LD_LIBRARY_PATH? Wouldn't this cause problems? My goal is that an external program, which I can't rebuild at will, would look for libraries first in the system paths, and only if there are none, in my directories. So I want to specify the system paths first, then my directories in the LD_LIBRARY_PATH variable. I know it's better to use containers for these situations, but I'm limited in what I can do.