Repository navigation
[FEA]: Ability to provide an extra path for cuda-pathfinder to search with high priority #1054
Description
Activity
Why don't you just remove the wheel from the Python env, and let the system search kick in (through
LD_LIBRARY_PATH)? This is not a typical request especially for library teams who own both C and Python libraries. If you don't keep the hygiene of your dev env, at one point things will just break and we can't be blamed.Why don't you just remove the wheel from the Python env, and let the system search kick in (through
LD_LIBRARY_PATH)? This is not a typical request especially for library teams who own both C and Python libraries. If you don't keep the hygiene of your dev env, at one point things will just break and we can't be blamed.We could certainly do that. It's just one extra step to make sure
pip installnever pulls back in those dependencies. I agree that it is very much a niche/developer-only feature and may not be worth doing if it only serves a small value.pip install --no-deps ...will help you avoid installing dependencies. Alternatively, you can organize non-nvSHMEM dependencies into a, say,requirements-dev-core.txtfile, and thenpip install -r requirements-dev-core.txtto just avoid nvSHMEM.I'd prefer us to not commit to any overwrite mechanisms for now (preferably forever). I don't want to repeat the nightmare of having
RPATHvsRUNPATHand having to worry about the search precedence. The less factors participating in our search logic, the better.Reacted by Ben GlickUnderstandable. This is not a dealbreaker for us (or probably anyone else) if we can't have it.
I agree we shouldn't reorder standard paths, but I agree with @benhg that there's value in having an "escape hatch" of sorts to override pathfinder behavior. This will inevitably come in handy when someone needs to pull in a debug build of a library or something else. I think if we have pathfinder specific environment variables that allow controlling on a per library and per component (library vs headers vs executables vs etc.) basis that it would be a valuable addition.
Hi @benhg, I wanted to call out that the just-merged PR #2680 includes features relevant to this issue.
PR #2680 adds descriptor-driven product installation roots for cuDNN and NCCL. Existing Pathfinder APIs use these automatically for both dynamic-library and header discovery:
CUDNN_PATH=<root>supports the declared cuDNN layouts:lib/lib64plusincludeon Linux;bin/x64orbinplusincludeon Windows x64; and the verified standalone archive'sbin/arm64plusincludelayout on native Windows ARM64.NCCL_HOME=<root>supportslib,lib64, andbuild/libfor the NCCL dynamic library on Linux, plusincludeandbuild/includefor headers. This covers installed/extracted prefixes as well as an NCCL source/build tree.
This resembles the third idea in this issue, but there is an important difference: these variables are conventional-layout fallbacks, not developer overrides.
For dynamic libraries, the effective load precedence remains:
- An already-loaded library.
- A wheel or Conda candidate.
- Native OS loader search.
- The product root (
CUDNN_PATHorNCCL_HOME). - Generic CUDA roots and later fallbacks such as Program Files.
For headers, wheel and Conda candidates likewise precede the product root; the product root then precedes generic CUDA and system locations. A product variable must point to an installation root with one of the declared layouts, and an unset, stale, or non-matching root falls through normally.
So PR #2680 does not resolve the original nvSHMEM development use case here, mainly:
- An nvSHMEM-specific product root is still missing.
- Even an analogous product root at the current precedence would not supersede an installed nvSHMEM wheel.
- It does not accept an arbitrary library file or directory as a highest-priority override.
- It does not provide fail-loud semantics for a configured override that cannot be used.
Do you think this issue should stay open for a genuine preemptive override, or would the current product-root model be sufficient if we added equivalent nvSHMEM metadata? It could let pathfinder discover a conventional nvSHMEM development or installation tree whenever no higher-priority wheel or Conda candidate is present.
Is this a duplicate?
Area
cuda.pathfinder
Is your feature request related to a problem? Please describe.
The cuda-pathfinder's search path prioritizes libraries from wheels. For example,
libnvshmem_host.socomes fromlibnvidia-nvshmem-cuXXwheel. If I'm working from a development branch of nvshmem, I may need to manually copy libraries over thesite-packagesdirectory in order to get my custom libraries to be opened.LD_LIBRARY_PATHis much lower in the priority order (on purpose), so that won't work either.Describe the solution you'd like
A few different ideas come to mind:
<LIBNAME_UPPER_CASE>_PATH_OVERRIDEso each library has an ability to be overridden independentlyCUDA_PATHFINDER_TOP_PRIORITY_PATHwhich applies to all pathfinder searches<library>_HOMEenvironment variable, and prioritizing it above other search pathsDescribe alternatives you've considered
I think I listed all 3 above. The current hacky WAR is to copy .so files around on the system.
Additional context
No response