Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
master, 5.0.0
-
None
Description
I have made a small modification of this test workflow here:
https://github.com/spxiwh/pegasus_subflow_example/
which fails in the same way as our big workflows. I think I've also seen where the problem is. You are using the output.map file to move files back from sub-workflows to the main workflow. We are using the output.map file to stage files out, in some cases directly from the sub-workflow. These two files are conflicting with each other, it seems I cannot get pegasus to read *both* of them, which either results in files not being staged out, which should be, or I get failures because the main workflow cannot find files from the sub-workflow ..... However, my take on this is that this *almost* works. If we can just figure out some way to handle an internal-to-the-subworkflow output.map and a stage-back-to-the-main-workflow output.map I think this would "just work" (and a secondary question, can I put a file into *both*, or do I have to be careful to avoid that). ..... I hope the workflow will explain this much better than I've managed to do with words!
https://github.com/spxiwh/pegasus_subflow_example/
which fails in the same way as our big workflows. I think I've also seen where the problem is. You are using the output.map file to move files back from sub-workflows to the main workflow. We are using the output.map file to stage files out, in some cases directly from the sub-workflow. These two files are conflicting with each other, it seems I cannot get pegasus to read *both* of them, which either results in files not being staged out, which should be, or I get failures because the main workflow cannot find files from the sub-workflow ..... However, my take on this is that this *almost* works. If we can just figure out some way to handle an internal-to-the-subworkflow output.map and a stage-back-to-the-main-workflow output.map I think this would "just work" (and a secondary question, can I put a file into *both*, or do I have to be careful to avoid that). ..... I hope the workflow will explain this much better than I've managed to do with words!
