When pegasus-cleanup is given a URL like this:

s3://gideon@amazon/pegasus-wms-scratch/gideon/pegasus/montage/20141007T100752-0800

with the option --recusive, then it needs to call this pegasus-s3 command:

pegasus-s3 rm s3://gideon@amazon/pegasus-wms-scratch/gideon/pegasus/montage/20141007T100752-0800/*

otherwise the set of keys will not be removed. Note that we are appending '/*' to the URL. Only append '/' if the URL doesn't already end in '/'.

I think it is probably safe to fix this in pegasus-cleanup without changing pegasus-plan because a) pegasus-plan does not know about s3, and b) pegasus-plan probably only uses --recursive for leaf cleanup, so that's a good way to tell if the URL is a S3 key prefix rather than a full key. Note that we wouldn't want to add '/*' to a regular key, because that would cause the key to not be deleted and it could cause other, unintended keys to be deleted.

We should make this fix in the 4.4.x branch.