Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1135

pegasus.transfer.bypass.input.staging breaks symlinking on the local site

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 4.7.1
    • Fix Version/s: master, 4.8.0, 4.7.3
    • Component/s: pegasus-plan
    • Labels:
      None

      Description

      In order to run on OSG, we need to set
      ```
      pegasus.transfer.bypass.input.staging=True
      ```
      however this breaks symlinking on the local site. Without this option set, local jobs can access their frame data with symlinks:
      ```
       { "type": "transfer",
         "id": 1,
         "src_urls": [
           { "site_label": "local", "url": "file:///home/dbrown/projects/osg/local-workflow-1/output/local-site-scratch/work/./main_ID0000001/./113494/H-H1_HOFT_C02-1134940160-4096.gwf", "checkpoint": "false" }
         ],
         "dest_urls": [
           { "site_label": "local", "url": "symlink://$PWD/113494/H-H1_HOFT_C02-1134940160-4096.gwf" }
         ] }
      ```

      However, when I set this, these become copy jobs:
      ```
       { "type": "transfer",
         "id": 1,
         "src_urls": [
           { "site_label": "local", "url": "file:///cvmfs/oasis.opensciencegrid.org/ligo/frames/O1/H1_HOFT_C02/H/1134/H-H1_HOFT_C02-1134940160-4096.gwf", "priority": 400, "checkpoint": "false" },
         ],
         "dest_urls": [
           { "site_label": "local", "url": "file://$PWD/113494/H-H1_HOFT_C02-1134940160-4096.gwf" }
         ] }

      This is a blocker for us using OSG.

        Attachments

          Activity

            People

            • Assignee:
              vahi Karan Vahi
              Reporter:
              dbrown Duncan Brown
            • Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: