Output Mapper in conjunction with data dependencies between sub workflow jobs

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      I have made a small modification of this test workflow here:

      https://github.com/spxiwh/pegasus_subflow_example/

      which fails in the same way as our big workflows. I think I've also seen where the problem is. You are using the output.map file to move files back from sub-workflows to the main workflow. We are using the output.map file to stage files out, in some cases directly from the sub-workflow. These two files are conflicting with each other, it seems I cannot get pegasus to read both of them, which either results in files not being staged out, which should be, or I get failures because the main workflow cannot find files from the sub-workflow ..... However, my take on this is that this almost works. If we can just figure out some way to handle an internal-to-the-subworkflow output.map and a stage-back-to-the-main-workflow output.map I think this would "just work" (and a secondary question, can I put a file into both, or do I have to be careful to avoid that). ..... I hope the workflow will explain this much better than I've managed to do with words!

            Assignee:
            Karan Vahi
            Reporter:
            Duncan Brown
            Archiver:
            Rajiv Mayani

              Created:
              Updated:
              Resolved:
              Archived: