data dependencies between dax jobs and compute jobs in a workflow

XMLWordPrintable

      One of the disconnects right now for hierarchal workflows is that pegasus does not automatically take care of data dependencies between the dax jobs ( that result in a new workflow being run) and a compute job in the DAX.

      For example a workflow W, with jobs D -> C where D is a dax job and C is a compute job and the sub workflow invoked by D creates a file, that job C requires. Currently, the planner fails.

      The proposal to fix this is , to allow users to designate output files in the DAX job, like any other job in the workflow. Pegasus will ensure that when the sub workflow runs the output files required by job C are transferred to the scratch directory for the workflow W of which C is part of.

            Assignee:
            Karan Vahi
            Reporter:
            Karan Vahi
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: