Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1135

pegasus.transfer.bypass.input.staging breaks symlinking on the local site

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 4.7.1
    • master, 4.8.0, 4.7.3
    • Pegasus Planner
    • None

    Description

      In order to run on OSG, we need to set
      ```
      pegasus.transfer.bypass.input.staging=True
      ```
      however this breaks symlinking on the local site. Without this option set, local jobs can access their frame data with symlinks:
      ```
       { "type": "transfer",
         "id": 1,
         "src_urls": [
           { "site_label": "local", "url": "file:///home/dbrown/projects/osg/local-workflow-1/output/local-site-scratch/work/./main_ID0000001/./113494/H-H1_HOFT_C02-1134940160-4096.gwf", "checkpoint": "false" }
         ],
         "dest_urls": [
           { "site_label": "local", "url": "symlink://$PWD/113494/H-H1_HOFT_C02-1134940160-4096.gwf" }
         ] }
      ```

      However, when I set this, these become copy jobs:
      ```
       { "type": "transfer",
         "id": 1,
         "src_urls": [
           { "site_label": "local", "url": "file:///cvmfs/oasis.opensciencegrid.org/ligo/frames/O1/H1_HOFT_C02/H/1134/H-H1_HOFT_C02-1134940160-4096.gwf", "priority": 400, "checkpoint": "false" },
         ],
         "dest_urls": [
           { "site_label": "local", "url": "file://$PWD/113494/H-H1_HOFT_C02-1134940160-4096.gwf" }
         ] }

      This is a blocker for us using OSG.

      Attachments

        Activity

          People

            vahi Karan Vahi
            dbrown Duncan Brown
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: