Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1885

Allow bypass of input files in CondorIO mode to be similar to behavior for nonsharedfs

    XMLWordPrintable

Details

    Description

      Thanks for the followup! My main concern here is that the file is declared with two PFNs in the site catalog:

          pfns:
          - site: local
            pfn: file:///cvmfs/oasis.opensciencegrid.org/ligo/frames/O3/V1Online/V-V1Online-12399/V-V1Online-1239948000-2000.gwf
          - site: osg
            pfn: file:///cvmfs/oasis.opensciencegrid.org/ligo/frames/O3/V1Online/V-V1Online-12399/V-V1Online-1239948000-2000.gwf

      In the current case the site=local file is used, and then the file is correctly transferred with condorIO. This involves a transfer from the LIGO cluster to whatever remote OSG cluster we are running on. However, the site=osg file is visible on the remote cluster and should be used directly. So I contend that there should not be any condor file IO for this file and the planner should pick the site=osg URL for this file, not the site=local URL.

      I think the stage-in question is less important. There are a few steps currently being done here:

      1) The file is transferred from the location under /cvmfs to the local site scratch on site=local as a stage-in job.
      2) The file is transferred to the OSG site via condor IO when the job starts.
      3) The file is moved from the condor transfer location, to the directory pegasus expects the file to be in within the job's .sh script.

      While stage (1) and (3) seem to be adding complexity, they aren't really a problem as long as they are *symlinks* (or moves) and not *copies*.

      Does that make sense? The behaviour if transferring a file from site=local to site=osg seems correct, but I think the file should have been used directly from site=osg in this case.

      Attachments

        1. deep-lfn.sub
          0.8 kB
        2. test.sh
          0.1 kB

        Activity

          People

            vahi Karan Vahi
            iwharry Ian Harry
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: