Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-791

Use bulk delete in pegasus-s3

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: master, 4.5.0
    • Fix Version/s: 4.5.0
    • Labels:
      None

      Description

      Right now when you use the pegasus-s3 rm command, it deletes the keys one-by-one. Amazon added a bulk delete interface that enables keys to be deleted in batches of 1,000. We should use that interface to speed up the deletion process. In boto this is invoked with Bucket.delete_keys([key1,key2]). We could also do parallel deletes for very large sets of keys.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gideon Gideon Juve (Inactive)
                Reporter:
                gideon Gideon Juve (Inactive)
              • Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: