Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-375

pegasus-transfer fails on scp if the destination URL is a file

    XMLWordPrintable

    Details

      Description

      When I use SCP for pegasus stage-out, the destination URL is a file and pegasus-transfer fails like so:

      2011-05-03 13:57:25,132 INFO: Reading URL pairs from stdin
      2011-05-03 13:57:25,133 INFO: PATH=/usr/local/globus/default/bin:/usr/local/pegasus/default/bin:/usr/bin:/bin
      2011-05-03 13:57:25,133 INFO: LD_LIBRARY_PATH=/usr/local/globus/default/lib:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/../lib/amd64
      2011-05-03 13:57:25,133 INFO: Executing scp commands
      Traceback (most recent call last):
        File "/usr/local/pegasus/default/bin/pegasus-transfer", line 738, in <module>
          scp()
        File "/usr/local/pegasus/default/bin/pegasus-transfer", line 437, in scp
          stats_add(url_pair.dst_path)
        File "/usr/local/pegasus/default/bin/pegasus-transfer", line 595, in stats_add
          s = os.stat(filename)
      OSError: [Errno 2] No such file or directory: '/submit/diamond/outputs/f.b1'

      This happens because the scp() function tries to stat the file before it has been transferred.

      I fixed it like so:

      --- pegasus-transfer 2011-05-03 14:43:34.000000000 -0700
      +++ pegasus-transfer-new 2011-05-03 14:43:51.000000000 -0700
      @@ -434,12 +434,13 @@
                   prepare_local_dir(os.path.dirname(url_pair.dst_path))
                   cmd += " " + url_pair.src_host + ":" + url_pair.src_path
                   cmd += " " + url_pair.dst_path
      - stats_add(url_pair.dst_path)
               else:
                   cmd += " " + url_pair.src_path
                   cmd += " " + url_pair.dst_host + ":" + url_pair.dst_path
                   stats_add(url_pair.src_path)
               myexec(cmd, 1*60*60, False, delay_errors)
      + if url_pair.dst_proto == "file":
      + stats_add(url_pair.dst_path)
       
       
       def webget():

        Attachments

          Activity

            People

            • Assignee:
              rynge-page Mats Rynge [X] (Inactive)
              Reporter:
              gideon Gideon Juve (Inactive)
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: