Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-641 Improve Monitord performance (size and time) for large workflows
  3. PM-646

improve the expunge operation in monitord in replay mode

    XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: master
    • Fix Version/s: master, 4.1
    • Component/s: Monitord
    • Labels:
      None

      Description

      In case of large SCEC MySQL databases, the expunge process ( deleting already populated data in the mysql db for a workflow ) takes a long time, resulting in a spike in the memory usage.

      This is because of the way expunge has been implemented in the netlogger SQL Alchemy code.
      It triggers individual select statements , the results of which are then invidually deleted.

      The Expunge code already does things specific to a database backend
      1) for sqlite it just overwrites

      Hence, for mysql it is possible to rely on database triggers in the schema definition to do cascaded deletes

        Attachments

          Activity

            People

            • Assignee:
              vahi Karan Vahi
              Reporter:
              vahi Karan Vahi
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: