Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1893

need to explicitly mount sharedfilesystem dir into container when using shared filesystem as staging site for nonsharedfs

XMLWordPrintable

      For alpha fold workflow zaiyan was trying to mimic this setup
      https://pegasus.isi.edu/documentation/user-guide/data-transfers.html#hybrid-shared-filesystem-with-non-shared-filesystem-semantics

      however, the compute jobs that are launched by the container fail because of failure in data transfer inside the container

      LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
      2022-12-14 11:07:20,089 INFO: --------------------------------------------------------------------------------
      2022-12-14 11:07:20,089 INFO: Starting transfers - attempt 1
      2022-12-14 11:07:22,111 ERROR: Expected local file does not exist: /scitech/shared/scratch-90-days/mzalam/alpha2/mzalam/pegasus/Alphafold-workflow/run0034/00/00/GA98.fasta
      2022-12-14 11:07:22,111 ERROR: Expe

      The planner created this in pegasus lite
      printf "\n###################### Staging in input data and executables ######################\n" 1>&2

      1. stage in data and executables
        pegasus-transfer --threads 1 1>&2 << 'eof'
        [
        Unknown macro: { "type"}

        ,
        ...

      however, the singularity exec does not have the directory mounted
      singularity exec --no-home --bind $PWD:/srv singularity-container.sif /srv/jackhmmer_uniref90_ID0000001-cont.sh

            Assignee:
            vahi Karan Vahi
            Reporter:
            vahi Karan Vahi
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: