Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1838

Intermediate outputs in a clustered job get sent back to staging site when they are not used by subsequent jobs outside of the cluster and when stage_out has been set to false for those files

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Minor
    • master, 5.0.2
    • master, 5.0.1
    • Pegasus Planner
    • None

    Description

      Intermediate outputs within a clustered job, which are not used outside of that cluster as inputs to other jobs and when stage_out=False for those intermediate outputs, are still being sent back to the staging site. If the expectation is that they shouldn't be sent back, then this is a bug.

      An example of this is illustrated in cluster-staging-pegasus.tar.gz. See workflow.py for the example workflow, and pegasus/test-cluster/run0001/00/00/merge_cluster.sh to see what transfers would be done.

      Attachments

        Activity

          People

            vahi Karan Vahi
            tanaka Ryan Tanaka (Inactive)
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: