Integrate Pegasus with SRM to execute workflows on OSG

XMLWordPrintable

    • Type: New Feature
    • Resolution: Fixed
    • Priority: Major
    • 3.0
    • Affects Version/s: None
    • Component/s: Pegasus Planner
    • None

      Relevant email from Kent Blackburn as to the requriements

      I see this a a doomed exercise! The $DATA areas are too small to be
      utilized for these workflows. We are just re-inventing a model that does
      not work on the OSG for LIGO. The Caltech cluster does support accessing
      the Storage Element(SE)/SRM from the worker nodes (directly). Many OSG
      sites do although there are different usage cases (dcache API, etc.) but
      those are not expected to cause the workflow any issues at this time.

      The SRM API has been tested by hand and can continue to be tested by
      hand very easily (srmcopy) I don't see how this gets more complex when
      used in a workflow. And as I noted in my first paragraph, srmcopy should
      not be needed.

      There will be some OSG sites where the data has to be staged from the SE
      to the worker nodes, but we can worry about them after we get this
      working. I am mostly interested in getting data prestaged to OSG SEs and
      making those OSG sites look like the LIGO Data Grid (no data movement
      necessary). This will make the LIGO scientist much more interested in
      using Pegasus and using the OSG!

            Assignee:
            Karan Vahi
            Reporter:
            Karan Vahi
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: