Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-791

Use bulk delete in pegasus-s3

    XMLWordPrintable

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • master, 4.5.0
    • 4.5.0
    • None

    Description

      Right now when you use the pegasus-s3 rm command, it deletes the keys one-by-one. Amazon added a bulk delete interface that enables keys to be deleted in batches of 1,000. We should use that interface to speed up the deletion process. In boto this is invoked with Bucket.delete_keys([key1,key2]). We could also do parallel deletes for very large sets of keys.

      Attachments

        Issue Links

          Activity

            People

              gideon Gideon Juve (Inactive)
              gideon Gideon Juve (Inactive)
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: