Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1943

mixed binary/conda installs broken

XMLWordPrintable

      However, using `export PEGASUS_UPDATE_PYTHONPATH=0` as suggested here https://pegasus.isi.edu/documentation/user-guide/installation.html#mixing-environments-system-venv-conda hits some issues:
        * First, I needed to install the complete pegasus-wms package. pegasus-wms.api (as we use currently) does not work because Pegasus.db is not installed, and seems required if setting this option (although Pegasus.db exists in the systemwide install, we no longer use it). We can add the bigger package if needed though.
        * Then, when submitting a workflow (using pegasus.api to plan and submit a workflow at generation time) the PEGASUS_UPDATE_PYTHONPATH=0 variable is not passed to the running workflow and when pegasus-plan is invoked to run subworkflows, it will fail. If I stop the workflow, and then restart again with pegasus-run, the variable is picked up. It would definitely be good to figure out a fix to this one as "plan and submit when generating a workflow" is now our default use case.
       * Finally we then see failures in jobs mid-workflow that look something like "Integrity check: No checksum in the meta data for H1L1V1-BANK2HDF-1186740100-3400.hdf". Indeed there is no checksum in the metadata for that file (or any of the other inputs). The test workflow is definitely a bit weird, as we look to test a number of features rather than do something sensible, and I think this is probably related to using many different "sites" for various jobs.

            Assignee:
            rynge Mats Rynge
            Reporter:
            iwharry Ian Harry
            Watchers:
            1 Start watching this issue

              Created:
              Updated: