Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1323

pegasus transfer should not try and transfer a file that does not exist

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.9.1
    • Fix Version/s: 5.0.0, 4.9.1
    • Component/s: pegasus-transfer
    • Labels:
      None

      Description

      Pegasus transfer tries to run the transfer program regardless of whether or not the output file exists. This confuses the user, as the error messages from e.g. stashcp are relevant, but that's not where the error is. If pegasus-transfer is moving a local file to another location, it would be better to check that the file exists before calling the transfer program and exit with a more helpful error message if it does not.

      ############################ Staging out output files ############################
      2018-11-06 11:49:32,578 INFO: Reading URL pairs from stdin
      2018-11-06 11:49:32,581 INFO: 1 transfers loaded
      2018-11-06 11:49:32,581 INFO: PATH=/srv/pegasus.BVvWUhhMr/pegasus-4.9.1dev/bin:/cvmfs/connect.opensciencegrid.org/modules/spack/bin:/opt/pycbc/pycbc-software/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/cvmfs/connect.opensciencegrid.org/modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-4.8.5-ri3pcc3a7osub6d2qf2acir52w2prrbr/bin
      2018-11-06 11:49:32,581 INFO: LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64
      2018-11-06 11:49:32,814 INFO: --------------------------------------------------------------------------------
      2018-11-06 11:49:32,815 INFO: Starting transfers - attempt 1
      2018-11-06 11:49:34,819 INFO: Tool found: stashcp Version: N/A Path: /bin/stashcp
      2018-11-06 11:49:34,819 INFO: /bin/stashcp '/srv/pegasus.BVvWUhhMr/112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf' 'stash:///user/dbrown/o1-open-catalog/analysis/analysis-test/o1-analysis-test-latest-LOSC_16_V1/output/osgconnect-scratch/work/./o1-analysis-test-latest-LOSC_16_V1-main_ID0000001/./112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf'
      2018-11-06 11:49:35,451 INFO: Traceback (most recent call last):
        File "/bin/stashcp", line 502, in <module>
          main()
        File "/bin/stashcp", line 494, in main
          result = doStashCpSingle(sourceFile=source, destination=destination, debug=args.debug)
        File "/bin/stashcp", line 118, in doStashCpSingle
          return doWriteBack(sourceFile, destination)
        File "/bin/stashcp", line 65, in doWriteBack
          statinfo = os.stat(source)
      OSError: [Errno 2] No such file or directory: '/srv/pegasus.BVvWUhhMr/112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf'
      2018-11-06 11:49:35,452 ERROR: Command exited with non-zero exit code (1): /bin/stashcp ...

        Attachments

          Activity

            People

            • Assignee:
              rynge Mats Rynge
              Reporter:
              dbrown Duncan Brown
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: