unbounded recursion in database loader for dashboard in monitord

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major
    • master, 4.9.0, 4.8.3
    • Affects Version/s: master, 4.8.2
    • Component/s: Monitord
    • None
    • Environment:
      1.8M building SimCenter workflow

      8 02:56:11,756:ERROR:Pegasus.db.dashboard_loader.DashboardLoader(189): Connection problem during commit: (OperationalError) database is locked None None - reattempting batch
      2018-06-18 02:56:16,786:ERROR:Pegasus.db.dashboard_loader.DashboardLoader(189): Connection problem during commit: (OperationalError) database is locked None None - reattempting batch
      2018-06-18 02:56:21,816:ERROR:Pegasus.db.dashboard_loader.DashboardLoader(189): Connection problem during commit: (OperationalError) database is locked None None - reattempting batch
      2018-06-18 02:56:26,847:ERROR:Pegasus.db.dashboard_loader.DashboardLoader(189): Connection problem during commit: (OperationalError) database is locked None None - reattempting batch
      2018-06-18 02:56:31,889:ERROR:Pegasus.db.dashboard_loader.DashboardLoader(189): Connection problem during commit: (OperationalError) database is locked None None - reattempting batch
      2018-06-18 02:56:31,903:WARNING:Pegasus.monitoring.workflow(147): DASHBOARD DB NL-LOAD-ERROR --> ce9881fb-1eb8-4adb-99eb-614222c01942 - simcenter-0
      2018-06-18 02:56:31,903:WARNING:Pegasus.monitoring.workflow(148): error sending event to dashboard db: xwf.end -->

      {'status': 0, 'xwf__id': 'ce9881fb-1eb8-4adb-99eb-614222c01942', 'reason': None, 'restart_count': 0, 'ts': 1529310851.0}

      2018-06-18 02:56:32,072:WARNING:Pegasus.monitoring.workflow(149): Traceback (most recent call last):
        File "/usr/lib64/python2.7/site-packages/Pegasus/monitoring/workflow.py", line 142, in output_to_dashboard_db
          self._dashboard_sink.send(event, kwargs)
        File "/usr/lib64/python2.7/site-packages/Pegasus/monitoring/event_output.py", line 167, in send
          self._db.process(d)
        File "/usr/lib64/python2.7/site-packages/Pegasus/db/dashboard_loader.py", line 91, in process
          self.check_flush(increment=True)
        File "/usr/lib64/python2.7/site-packages/Pegasus/db/base_loader.py", line 48, in check_flush
          self.hard_flush()
        File "/usr/lib64/python2.7/site-packages/Pegasus/db/dashboard_loader.py", line 191, in hard_flush
          self.hard_flush()
      ...
      ..
      RuntimeError: maximum recursion depth exceeded

            Assignee:
            Karan Vahi
            Reporter:
            Mats Rynge
            Archiver:
            Rajiv Mayani

              Created:
              Updated:
              Resolved:
              Archived: