Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-458

stampede api's should provide a close connection method

XMLWordPrintable

      While running pegasus plots on a synthetic scec run I see eventually an operational error saying too many connections()

      The SCEC run has around 77 sub workflows. The plotting code for each sub workflow has to initialize an object of the Stampede API , that opens a connection in SQLAlchemy. I think in this case, the connections are not being closed automatically.
      Probably an explicit close connection method should be provided?

      2011-08-05 15:17:23,154:pegasus-plots:create_charts:356: INFO: Generating graphs/charts for the workflow 34a04ade-26c3-440f-8cdf-d3e39d39abc0 ...
      .....
      2011-08-05 15:17:25,455:pegasus-plots:create_charts:356: INFO: Generating graphs/charts for the workflow 86139274-e17d-4328-b941-4547d13d6630 ...

      2011-08-05 15:17:26,457:pegasus-plots:create_charts:356: INFO: Generating graphs/charts for the workflow 6756b543-7641-45a3-9889-a0639133824b ...
      2011-08-05 15:17:26,638:populate.py:get_wf_stats:354: WARNING: Traceback (most recent call last):
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/Pegasus/plots_stats/plots/populate.py", line 350, in get_wf_stats
      workflow_stampede_stats = StampedeStatistics(global_db_url , expand)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/netlogger/analysis/workflow/stampede_statistics.py", line 164, in _init_
      SQLAlchemyInit._init_(self, connString, initializeToPegasusDB)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/netlogger/analysis/modules/base.py", line 129, in __init_
      initFunction(self.db, self.metadata, kw=dialect_kw)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/netlogger/analysis/schema/stampede_schema.py", line 408, in initializeToPegasusDB
      metadata.create_all(db)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/schema.py", line 2119, in create_all
      bind.create(self, checkfirst=checkfirst, tables=tables)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/engine/base.py", line 1701, in create
      connection=connection, **kwargs)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/engine/base.py", line 1738, in _run_visitor
      conn = self.contextual_connect(close_with_result=False)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/engine/base.py", line 1832, in contextual_connect
      self.pool.connect(),
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/pool.py", line 183, in connect
      return _ConnectionFairy(self).checkout()
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/pool.py", line 370, in _init_
      rec = self._connection_record = pool.get()
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/pool.py", line 214, in get
      return self.do_get()
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/pool.py", line 733, in do_get
      con = self.create_connection()
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/pool.py", line 148, in create_connection
      return _ConnectionRecord(self)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/pool.py", line 254, in _init_
      self.connection = self.__connect()
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/pool.py", line 320, in __connect
      connection = self.__pool._creator()
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/engine/strategies.py", line 76, in connect
      return dialect.connect(*cargs, **cparams)
      File "/data/scratch/vahi/software/install/pegasus/default/lib/python/sqlalchemy/engine/default.py", line 249, in connect
      return self.dbapi.connect(*cargs, **cparams)
      File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.3-py2.6-linux-x86_64.egg/MySQLdb/_init_.py", line 81, in Connect
      return Connection(*args, **kwargs)
      File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.3-py2.6-linux-x86_64.egg/MySQLdb/connections.py", line 187, in _init_
      super(Connection, self)._init_(*args, **kwargs2)
      OperationalError: (OperationalError) (1040, 'Too many connections') None None
      vahi@obelix:/data/scratch/vahi/examples/synthetic-scec/dags/vahi/pegasus/Stampede-Test$ logger.warning(traceback

            Assignee:
            mmgoode Monte Goode
            Reporter:
            vahi Karan Vahi
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: