pegasus-cleanup needs to add wildcards to s3:// URLs when --recursive is used

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major
    • 4.4.1
    • Affects Version/s: master, 4.4.0
    • None

      When pegasus-cleanup is given a URL like this:

      s3://gideon@amazon/pegasus-wms-scratch/gideon/pegasus/montage/20141007T100752-0800

      with the option --recusive, then it needs to call this pegasus-s3 command:

      pegasus-s3 rm s3://gideon@amazon/pegasus-wms-scratch/gideon/pegasus/montage/20141007T100752-0800/*

      otherwise the set of keys will not be removed. Note that we are appending '/*' to the URL. Only append '/' if the URL doesn't already end in '/'.

      I think it is probably safe to fix this in pegasus-cleanup without changing pegasus-plan because a) pegasus-plan does not know about s3, and b) pegasus-plan probably only uses --recursive for leaf cleanup, so that's a good way to tell if the URL is a S3 key prefix rather than a full key. Note that we wouldn't want to add '/*' to a regular key, because that would cause the key to not be deleted and it could cause other, unintended keys to be deleted.

      We should make this fix in the 4.4.x branch.

            Assignee:
            Mats Rynge
            Reporter:
            Gideon Juve
            Archiver:
            Rajiv Mayani

              Created:
              Updated:
              Resolved:
              Archived: