Currently the scp implementation of pegasus-transfer does one transfer at a time. We should try to optimize it to do transfers in batches somehow.

I'm not sure how this would work, but it may be as simple as specifying several sources for each destination directory. We would have to validate that the source file name is the same as the destination file name.

So if you have:

scp://host/A/file1 file:///B/file1
scp://host/A/file2 file:///B/file2

It should be possible to do:

mkdir -p /B
scp host:/A/file1 host:/A/file2 /B/

We would have to test to see if this is faster or not.