Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1795

Output Mapper in conjunction with data dependencies between sub workflow jobs

    XMLWordPrintable

Details

    Description

       I have made a small modification of this test workflow here:

      https://github.com/spxiwh/pegasus_subflow_example/

      which fails in the same way as our big workflows. I think I've also seen where the problem is. You are using the output.map file to move files back from sub-workflows to the main workflow. We are using the output.map file to stage files out, in some cases directly from the sub-workflow. These two files are conflicting with each other, it seems I cannot get pegasus to read *both* of them, which either results in files not being staged out, which should be, or I get failures because the main workflow cannot find files from the sub-workflow ..... However, my take on this is that this *almost* works. If we can just figure out some way to handle an internal-to-the-subworkflow output.map and a stage-back-to-the-main-workflow output.map I think this would "just work" (and a secondary question, can I put a file into *both*, or do I have to be careful to avoid that). ..... I hope the workflow will explain this much better than I've managed to do with words!

      Attachments

        Activity

          People

            vahi Karan Vahi
            dbrown Duncan Brown
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: