Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-790

pegasus-cleanup needs to add wildcards to s3:// URLs when --recursive is used

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: master, 4.4.0
    • Fix Version/s: 4.4.1
    • Labels:
      None

      Description

      When pegasus-cleanup is given a URL like this:

      s3://gideon@amazon/pegasus-wms-scratch/gideon/pegasus/montage/20141007T100752-0800

      with the option --recusive, then it needs to call this pegasus-s3 command:

      pegasus-s3 rm s3://gideon@amazon/pegasus-wms-scratch/gideon/pegasus/montage/20141007T100752-0800/*

      otherwise the set of keys will not be removed. Note that we are appending '/*' to the URL. Only append '/' if the URL doesn't already end in '/'.

      I think it is probably safe to fix this in pegasus-cleanup without changing pegasus-plan because a) pegasus-plan does not know about s3, and b) pegasus-plan *probably* only uses --recursive for leaf cleanup, so that's a good way to tell if the URL is a S3 key prefix rather than a full key. Note that we wouldn't want to add '/*' to a regular key, because that would cause the key to not be deleted and it could cause other, unintended keys to be deleted.

      We should make this fix in the 4.4.x branch.

        Attachments

          Activity

            People

            • Assignee:
              rynge Mats Rynge
              Reporter:
              gideon Gideon Juve (Inactive)
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: