Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1313

allow for direct use of singularity containers on CVMFS on compute site

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: master
    • Fix Version/s: master, 4.9.0
    • Component/s: pegasus-plan
    • Labels:
      None

      Description

      Hi Karan,

      I've been experimenting with the current singularity implementation in 4.8.4 and hit the following problem. I want to use Brian's images from /cvmfs/singularity.opensciencegrid.org/pycbc/pycbc-el7:latest as they are pre-staged to all of our nodes, but I get the error

      2018.10.12 09:06:43.922 EDT: [FATAL ERROR]
      [1]: Unable to instantiate Transformation Catalog at edu.isi.pegasus.planner.catalog.transformation.TransformationFactory.loadInstance(TransformationFactory.java:172)
      [2]: edu.isi.pegasus.planner.catalog.transformation.impl.Text caught edu.isi.pegasus.planner.parser.ScannerException line 28: at edu.isi.pegasus.planner.parser.TransformationCatalogTextParser.parse(TransformationCatalogTextParser.java:236)
      [3]: Invalid suffix .opensciencegrid.org/pycbc/pycbc-el7:latest determined singularity image url url -> file:///cvmfs/singularity.opensciencegrid.org/pycbc/pycbc-el7:latest , protocol -> file , host -> , path -> /cvmfs/singularity.opensciencegrid.org/pycbc/pycbc-el7:latest , url-prefix -> file:// at edu.isi.pegasus.planner.catalog.transformation.classes.Container.computeLFN(Container.java:180)

      It looks like Pegasus doesn't like the file URL in my transformation catalog:

      tr pegasus::findrange:4.0 {

       site condorpool {

         pfn "gsiftp://sugwg-condor.phy.syr.edu/usr/bin/pegasus-keg"
         arch "x86_64"
         os "linux"

         # INSTALLED means pfn refers to path in the container.
         # STAGEABLE means the executable can be staged into the container
         type "STAGEABLE"

         #optional attribute to specify the container to use
         container "centos-pycbc"
       }
      }

      cont centos-pycbc{
          # can be either docker or singularity
          type "singularity"

          # URL to image in a docker|singularity hub OR
          # URL to an existing docker image exported as a tar file or singularity image
          image "file:///cvmfs/singularity.opensciencegrid.org/pycbc/pycbc-el7:latest"

          # optional site attribute to tell pegasus which site tar file
          # exists. useful for handling file URL's correctly
          image_site "pycbc"
      }

      Cheers,
      Duncan.

        Attachments

          Activity

            People

            • Assignee:
              vahi Karan Vahi
              Reporter:
              dbrown Duncan Brown
            • Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: