Intermediate outputs in a clustered job get sent back to staging site when they are not used by subsequent jobs outside of the cluster and when stage_out has been set to false for those files

XMLWordPrintable

    • Type: Bug
    • Resolution: Won't Fix
    • Priority: Minor
    • master, 5.0.2
    • Affects Version/s: master, 5.0.1
    • Component/s: Pegasus Planner
    • None

      Intermediate outputs within a clustered job, which are not used outside of that cluster as inputs to other jobs and when stage_out=False for those intermediate outputs, are still being sent back to the staging site. If the expectation is that they shouldn't be sent back, then this is a bug.

      An example of this is illustrated in cluster-staging-pegasus.tar.gz. See workflow.py for the example workflow, and pegasus/test-cluster/run0001/00/00/merge_cluster.sh to see what transfers would be done.

            Assignee:
            Karan Vahi
            Reporter:
            Ryan Tanaka (Inactive)
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: