-
Type: Bug
-
Resolution: Fixed
-
Priority: Major
-
Affects Version/s: master, 5.0.0
-
Component/s: Planner: hierarchal workflows, Planner: Mappers
-
None
I have made a small modification of this test workflow here:
https://github.com/spxiwh/pegasus_subflow_example/
which fails in the same way as our big workflows. I think I've also seen where the problem is. You are using the output.map file to move files back from sub-workflows to the main workflow. We are using the output.map file to stage files out, in some cases directly from the sub-workflow. These two files are conflicting with each other, it seems I cannot get pegasus to read both of them, which either results in files not being staged out, which should be, or I get failures because the main workflow cannot find files from the sub-workflow ..... However, my take on this is that this almost works. If we can just figure out some way to handle an internal-to-the-subworkflow output.map and a stage-back-to-the-main-workflow output.map I think this would "just work" (and a secondary question, can I put a file into both, or do I have to be careful to avoid that). ..... I hope the workflow will explain this much better than I've managed to do with words!