Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-610

Batch scp transfers in pegasus-transfer

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.6.0
    • Labels:
      None

      Description

      Currently the scp implementation of pegasus-transfer does one transfer at a time. We should try to optimize it to do transfers in batches somehow.

      I'm not sure how this would work, but it may be as simple as specifying several sources for each destination directory. We would have to validate that the source file name is the same as the destination file name.

      So if you have:

      scp://host/A/file1 file:///B/file1
      scp://host/A/file2 file:///B/file2

      It should be possible to do:

      mkdir -p /B
      scp host:/A/file1 host:/A/file2 /B/

      We would have to test to see if this is faster or not.

        Attachments

          Activity

            People

            • Assignee:
              rynge Mats Rynge
              Reporter:
              gideon Gideon Juve (Inactive)
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: