add functionality to the Python api so that you can do SubWorkflow(Workflow("name"))

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major
    • master
    • Affects Version/s: master
    • Component/s: Workflow API Libraries
    • None

      It could be useful to add a workflow into an existing workflow, without having to create a SubWorkflow job, and have a separate workflow yaml file.

      For example, Ian Harry (LIGO/PyCBC), described the following scenario:
      I have some root workflow A, and it has a SubWorkflow job B. Everything in B is known before running A. That is, there is nothing dynamic about the hierarchical workflow. The workflow is separated for organizational purposes (no 'generate_workflow' type job is run before the SubWorkflow job). Instead of first writing the workflow_b.yml file, and adding it to the SubWorkflow job, it would easier (and have less files) to just do something along the lines of this:

      w1 = Workflow()
      w.add_jobs(one)
      w2 = Workflow()
      w2.add_jobs(two, three)
      w1.add_jobs( w2 )
      w1.add_jobs( four )

      1. w1 should serialize w2 and internally manage it as w1.add_subwf( serialized(w2) ), and
      2. four should auto depend on w2 if four uses output of three or four

      notes: https://docs.google.com/document/d/1e6rfkJDiXN3eKhSma7V_PmihfsgcaK9hisfOtPK6KrY/edit

            Assignee:
            Ryan Tanaka [X] (Inactive)
            Reporter:
            Ryan Tanaka [X] (Inactive)
            Archiver:
            Rajiv Mayani

              Created:
              Updated:
              Resolved:
              Archived: