Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1937

mixed binary/venv/conda installs


    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major Major
    • master, 5.1.0, 5.0.7
    • Affects Version/s: master, 5.0.6
    • Component/s: Build Infrastructure
    • None
    • Environment:
      mix of conda install and pegasus system package install

      Hi Karan,

      I wanted to ask you about a complication we've been dealing with since
      some recent python, and condor upgrades.

      Our use case is that the pegasus executables (pegasus-plan etc.) are
      installed by the sysadmin in /usr using (I think) `dnf install
      pegasus`. We then run code installed in some virtualenv or conda env,
      which includes the pegasus-wms.api package. That environment will be
      running a "modern" version of python (3.9+), whereas RHEL 8 linux is
      using an old python (3.6) which is what pegasus tools at that level

      The divergence between these python versions means that if we run
      pegasus-plan without setting the PEGASUS_PYTHON variable, we will
      get a PYTHONPATH that includes a mix of python3.9+ packages and
      python3.6 packages in /usr, and these are now too divergent to be at
      all compatible.

      If, however, we set PEGASUS_PYTHON=/usr/bin/python3 then the pegasus
      dependencies installed in $PEGASUS_HOME/lib64/pegasus/externals are
      not in the PYTHONPATH, and the code fails.

      The only workaround I've found for this is to create a standalone
      environment, conda/pip install pegasus dependencies, install pegasus
      into it from source and then point PEGASUS_PYTHON to that. Or try to
      manually identify and get sys-admins to install pegasus dependencies
      from lib64/pegasus/externals into the standard lib path.

      What we want is to be able to use pegasus tools installed in /usr,
      while isolating the python version pegasus needs from everything else.
      What's the best way to do that? My hunch is that we need some new flag
      that would say:

      • I want the systemwide version of python3 for pegasus (/usr/bin/python3)
      • I do not want to see the systemwide python paths for anything
        except pegasus
      • I do need PEGASUS_HOME/lib64/pegasus/externals ... or I need pegasus
        to install its dependencies in a way that we can use them.

      Does this make sense?


            rynge Mats Rynge
            iwharry Ian Harry
            2 Start watching this issue