Planner should publish metrics about the workflow to the Pegasus Metrics Server

XMLWordPrintable

    • Type: New Feature
    • Resolution: Fixed
    • Priority: Major
    • master, 4.2
    • Affects Version/s: master
    • Component/s: Pegasus Planner
    • None

      The metrics will be pushed out in JSON format to the metrics server.

      The planner will honor two environment variables

      a) PEGASUS_METRICS

      A boolean value ( true | false ) indicating whether metrics reporting is turned ON/OFF

      b) PEGASUS_USER_METRICS_SERVER

      A comma separated list of URL's to metric servers to report metrics to
      This can be set by users to enable reporting to their own servers.

      c) PEGASUS_METRICS_SERVER
      Can be used to change the default server used to report the metrics to.

      In addition, they will be written out to a file in the submit directory.
      Sample Output
      {
      "client": "pegasus-plan",
      "version": "4.2.0cvs",
      "start_time": "Dec 18, 2012 2:30:32 PM",
      "end_time": "Dec 18, 2012 2:30:32 PM",
      "duration": 0.62,
      "exitcode": 1,
      "error": "java.lang.RuntimeException: TransferEngine.java: Can\u0027t determine a location to transfer input file for lfn david.f.a for job preprocess_ID000001\n\tat edu.isi.pegasus.planner.refiner.TransferEngine.getFilesFromRC(TransferEngine.java:1216)\n\tat edu.isi.pegasus.planner.refiner.TransferEngine.processParents(TransferEngine.java:592)\n\tat edu.isi.pegasus.planner.refiner.TransferEngine.addTransferNodes(TransferEngine.java:371)\n\tat edu.isi.pegasus.planner.refiner.MainEngine.runPlanner(MainEngine.java:238)\n\tat edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:612)\n\tat edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:353)\n\tat edu.isi.pegasus.planner.client.CPlanner.main(CPlanner.java:250)\n",
      "root.wf.uid": "b6f1053e-508a-47da-b057-bd3e82f5f416",
      "wf.uid": "b6f1053e-508a-47da-b057-bd3e82f5f416",
      "wf_metrics":

      { "compute_tasks": 4, "dax_tasks": 0, "dag_tasks": 0, "compute_jobs": 4, "clustered_jobs": 0, "si_tx_jobs": 0, "so_tx_jobs": 0, "inter_tx_jobs": 0, "reg_jobs": 0, "cleanup_jobs": 0, "create_dir_jobs": 0, "dax_jobs": 0, "dag_jobs": 0, "chmod_jobs": 0, "total_jobs": 4 }

      }

            Assignee:
            Karan Vahi
            Reporter:
            Karan Vahi
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: