Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-739

Data resuse / cleanup conflict

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: master, 4.4.0, 4.3.2
    • Component/s: None
    • Labels:
      None

      Description


      The following (and tarball) is from a workflow with most tasks removed due to data reuse. The cleanup algorithm is getting confused because a file listed as transfer=True in a job exists on a remote site, but not the output site. A staging job is added to transfer from the remote site to the output site, but then a cleanup job is being added.

      2014.02.20 16:20:08.605 PST: [DEBUG] Clustering 2 cleanup nodes at level 2 with cluster size 1
      2014.02.20 16:20:08.605 PST: [DEBUG] Creating a clustered cleanup job named clean_up_staging_level_2_0 consisting of 1 nodes
      2014.02.20 16:20:08.605 PST: [DEBUG] Creating a clustered cleanup job named clean_up_staging_level_2_1 consisting of 1 nodes
      2014.02.20 16:20:08.605 PST: [DEBUG] For cleanup job clean_up_staging_level_2_0 the associated compute job is create_dir_pipeline_0_staging
      2014.02.20 16:20:08.606 PST: [FATAL ERROR] java.lang.RuntimeException: Unable to determine cleanup url for lfn HN001_addrepl.bam at site null
              at edu.isi.pegasus.planner.refiner.cleanup.Cleanup.createCleanupJob(Cleanup.java:213)
              at edu.isi.pegasus.planner.refiner.cleanup.InPlace.addCleanUpJobs(InPlace.java:532)
              at edu.isi.pegasus.planner.refiner.cleanup.InPlace.addCleanupJobs(InPlace.java:298)
              at edu.isi.pegasus.planner.refiner.CleanupEngine.addCleanupJobs(CleanupEngine.java:69)
              at edu.isi.pegasus.planner.refiner.MainEngine.runPlanner(MainEngine.java:279)
              at edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:646)
              at edu.isi.pegasus.planner.client.CPlanner.executeCommand(CPlanner.java:366)
              at edu.isi.pegasus.planner.client.CPlanner.main(CPlanner.java:254)
       

        Attachments

          Activity

            People

            • Assignee:
              vahi Karan Vahi
              Reporter:
              rynge-page Mats Rynge [X] (Inactive)
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: