add functionality to the Python api so that you can do SubWorkflow(Workflow("name"))

XMLWordPrintable

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major
    • master
    • Affects Version/s: master
    • Component/s: Workflow API Libraries
    • None

      It could be useful to add a workflow into an existing workflow, without having to create a SubWorkflow job, and have a separate workflow yaml file.

      For example, Ian Harry (LIGO/PyCBC), described the following scenario:
      I have some root workflow A, and it has a SubWorkflow job B. Everything in B is known before running A. That is, there is nothing dynamic about the hierarchical workflow. The workflow is separated for organizational purposes (no 'generate_workflow' type job is run before the SubWorkflow job). Instead of first writing the workflow_b.yml file, and adding it to the SubWorkflow job, it would easier (and have less files) to just do something along the lines of this:

      w1 = Workflow()
      w.add_jobs(one)
      w2 = Workflow()
      w2.add_jobs(two, three)
      w1.add_jobs( w2 )
      w1.add_jobs( four )

      1. w1 should serialize w2 and internally manage it as w1.add_subwf( serialized(w2) ), and
      2. four should auto depend on w2 if four uses output of three or four

      notes: https://docs.google.com/document/d/1e6rfkJDiXN3eKhSma7V_PmihfsgcaK9hisfOtPK6KrY/edit

            Assignee:
            Ryan Tanaka (Inactive)
            Reporter:
            Ryan Tanaka (Inactive)
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: