pegasus transfer should not try and transfer a file that does not exist

XMLWordPrintable

      Pegasus transfer tries to run the transfer program regardless of whether or not the output file exists. This confuses the user, as the error messages from e.g. stashcp are relevant, but that's not where the error is. If pegasus-transfer is moving a local file to another location, it would be better to check that the file exists before calling the transfer program and exit with a more helpful error message if it does not.

                                                            1. Staging out output files ############################
                                                              2018-11-06 11:49:32,578 INFO: Reading URL pairs from stdin
                                                              2018-11-06 11:49:32,581 INFO: 1 transfers loaded
                                                              2018-11-06 11:49:32,581 INFO: PATH=/srv/pegasus.BVvWUhhMr/pegasus-4.9.1dev/bin:/cvmfs/connect.opensciencegrid.org/modules/spack/bin:/opt/pycbc/pycbc-software/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/cvmfs/connect.opensciencegrid.org/modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-4.8.5-ri3pcc3a7osub6d2qf2acir52w2prrbr/bin
                                                              2018-11-06 11:49:32,581 INFO: LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.0.090/mkl/lib/intel64
                                                              2018-11-06 11:49:32,814 INFO: --------------------------------------------------------------------------------
                                                              2018-11-06 11:49:32,815 INFO: Starting transfers - attempt 1
                                                              2018-11-06 11:49:34,819 INFO: Tool found: stashcp Version: N/A Path: /bin/stashcp
                                                              2018-11-06 11:49:34,819 INFO: /bin/stashcp '/srv/pegasus.BVvWUhhMr/112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf' 'stash:///user/dbrown/o1-open-catalog/analysis/analysis-test/o1-analysis-test-latest-LOSC_16_V1/output/osgconnect-scratch/work/./o1-analysis-test-latest-LOSC_16_V1-main_ID0000001/./112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf'
                                                              2018-11-06 11:49:35,451 INFO: Traceback (most recent call last):
                                                              File "/bin/stashcp", line 502, in <module>
                                                              main()
                                                              File "/bin/stashcp", line 494, in main
                                                              result = doStashCpSingle(sourceFile=source, destination=destination, debug=args.debug)
                                                              File "/bin/stashcp", line 118, in doStashCpSingle
                                                              return doWriteBack(sourceFile, destination)
                                                              File "/bin/stashcp", line 65, in doWriteBack
                                                              statinfo = os.stat(source)
                                                              OSError: [Errno 2] No such file or directory: '/srv/pegasus.BVvWUhhMr/112846/L1-INSPIRAL_FULL_DATA_JOB1-1128466800-3299.hdf'
                                                              2018-11-06 11:49:35,452 ERROR: Command exited with non-zero exit code (1): /bin/stashcp ...

            Assignee:
            Mats Rynge
            Reporter:
            Duncan Brown
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: