Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1323

pegasus transfer should not try and transfer a file that does not exist

    XMLWordPrintable

Details

    Description

      Pegasus transfer tries to run the transfer program regardless of whether or not the output file exists. This confuses the user, as the error messages from e.g. stashcp are relevant, but that's not where the error is. If pegasus-transfer is moving a local file to another location, it would be better to check that the file exists before calling the transfer program and exit with a more helpful error message if it does not.

      ############################ Staging out output files ############################
      2018-11-06 11:49:32,578 INFO: Reading URL pairs from stdin
      2018-11-06 11:49:32,581 INFO: 1 transfers loaded
      2018-11-06 11:49:32,581 INFO: PATH=/srv/pegasus.BVvWUhhMr/pegasus-4.9.1dev/bin:/cvmfs/connect.opensciencegrid.org/modules/spack/bin:/opt/pycbc/pycbc-software/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/cvmfs/connect.opensciencegrid.org/modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-4.8.5-ri3pcc3a7osub6d2qf2acir52w2prrbr/bin
      2018-11-06 11:49:32,581 INFO: LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64
      2018-11-06 11:49:32,814 INFO: --------------------------------------------------------------------------------
      2018-11-06 11:49:32,815 INFO: Starting transfers - attempt 1
      2018-11-06 11:49:34,819 INFO: Tool found: stashcp Version: N/A Path: /bin/stashcp
      2018-11-06 11:49:34,819 INFO: /bin/stashcp '/srv/pegasus.BVvWUhhMr/112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf' 'stash:///user/dbrown/o1-open-catalog/analysis/analysis-test/o1-analysis-test-latest-LOSC_16_V1/output/osgconnect-scratch/work/./o1-analysis-test-latest-LOSC_16_V1-main_ID0000001/./112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf'
      2018-11-06 11:49:35,451 INFO: Traceback (most recent call last):
        File "/bin/stashcp", line 502, in <module>
          main()
        File "/bin/stashcp", line 494, in main
          result = doStashCpSingle(sourceFile=source, destination=destination, debug=args.debug)
        File "/bin/stashcp", line 118, in doStashCpSingle
          return doWriteBack(sourceFile, destination)
        File "/bin/stashcp", line 65, in doWriteBack
          statinfo = os.stat(source)
      OSError: [Errno 2] No such file or directory: '/srv/pegasus.BVvWUhhMr/112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf'
      2018-11-06 11:49:35,452 ERROR: Command exited with non-zero exit code (1): /bin/stashcp ...

      Attachments

        Activity

          People

            rynge Mats Rynge
            dbrown Duncan Brown
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: