Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-739

Data resuse / cleanup conflict

XMLWordPrintable

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major Major
    • master, 4.4.0, 4.3.2
    • Affects Version/s: None
    • Component/s: None
    • None

      The following (and tarball) is from a workflow with most tasks removed due to data reuse. The cleanup algorithm is getting confused because a file listed as transfer=True in a job exists on a remote site, but not the output site. A staging job is added to transfer from the remote site to the output site, but then a cleanup job is being added.

      2014.02.20 16:20:08.605 PST: [DEBUG] Clustering 2 cleanup nodes at level 2 with cluster size 1
      2014.02.20 16:20:08.605 PST: [DEBUG] Creating a clustered cleanup job named clean_up_staging_level_2_0 consisting of 1 nodes
      2014.02.20 16:20:08.605 PST: [DEBUG] Creating a clustered cleanup job named clean_up_staging_level_2_1 consisting of 1 nodes
      2014.02.20 16:20:08.605 PST: [DEBUG] For cleanup job clean_up_staging_level_2_0 the associated compute job is create_dir_pipeline_0_staging
      2014.02.20 16:20:08.606 PST: [FATAL ERROR] java.lang.RuntimeException: Unable to determine cleanup url for lfn HN001_addrepl.bam at site null
      at edu.isi.pegasus.planner.refiner.cleanup.Cleanup.createCleanupJob(Cleanup.java:213)
      at edu.isi.pegasus.planner.refiner.cleanup.InPlace.addCleanUpJobs(InPlace.java:532)
      at edu.isi.pegasus.planner.refiner.cleanup.InPlace.addCleanupJobs(InPlace.java:298)
      at edu.isi.pegasus.planner.refiner.CleanupEngine.addCleanupJobs(CleanupEngine.java:69)
      at edu.isi.pegasus.planner.refiner.MainEngine.runPlanner(MainEngine.java:279)
      at edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:646)
      at edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:366)
      at edu.isi.pegasus.planner.client.CPlanner.main(CPlanner.java:254)

            Assignee:
            vahi Karan Vahi
            Reporter:
            rynge-page Mats Rynge [X] (Inactive)
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: