Planner should publish metrics about the workflow to the Pegasus Metrics Server

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Type: New Feature
    • Resolution: Fixed
    • Priority: Major
    • master, 4.2
    • Affects Version/s: master
    • Component/s: Pegasus Planner
    • None

      The metrics will be pushed out in JSON format to the metrics server.

      The planner will honor two environment variables

      a) PEGASUS_METRICS

      A boolean value ( true | false ) indicating whether metrics reporting is turned ON/OFF

      b) PEGASUS_USER_METRICS_SERVER

      A comma separated list of URL's to metric servers to report metrics to
      This can be set by users to enable reporting to their own servers.

      c) PEGASUS_METRICS_SERVER
      Can be used to change the default server used to report the metrics to.

      In addition, they will be written out to a file in the submit directory.
      Sample Output
      {
      "client": "pegasus-plan",
      "version": "4.2.0cvs",
      "start_time": "Dec 18, 2012 2:30:32 PM",
      "end_time": "Dec 18, 2012 2:30:32 PM",
      "duration": 0.62,
      "exitcode": 1,
      "error": "java.lang.RuntimeException: TransferEngine.java: Can\u0027t determine a location to transfer input file for lfn david.f.a for job preprocess_ID000001\n\tat edu.isi.pegasus.planner.refiner.TransferEngine.getFilesFromRC(TransferEngine.java:1216)\n\tat edu.isi.pegasus.planner.refiner.TransferEngine.processParents(TransferEngine.java:592)\n\tat edu.isi.pegasus.planner.refiner.TransferEngine.addTransferNodes(TransferEngine.java:371)\n\tat edu.isi.pegasus.planner.refiner.MainEngine.runPlanner(MainEngine.java:238)\n\tat edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:612)\n\tat edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:353)\n\tat edu.isi.pegasus.planner.client.CPlanner.main(CPlanner.java:250)\n",
      "root.wf.uid": "b6f1053e-508a-47da-b057-bd3e82f5f416",
      "wf.uid": "b6f1053e-508a-47da-b057-bd3e82f5f416",
      "wf_metrics":

      { "compute_tasks": 4, "dax_tasks": 0, "dag_tasks": 0, "compute_jobs": 4, "clustered_jobs": 0, "si_tx_jobs": 0, "so_tx_jobs": 0, "inter_tx_jobs": 0, "reg_jobs": 0, "cleanup_jobs": 0, "create_dir_jobs": 0, "dax_jobs": 0, "dag_jobs": 0, "chmod_jobs": 0, "total_jobs": 4 }

      }

            Assignee:
            Karan Vahi
            Reporter:
            Karan Vahi
            Archiver:
            Rajiv Mayani

              Created:
              Updated:
              Resolved:
              Archived: