Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-302

pegasus-transfer should handle optional files



    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: 3.1
    • Labels:


      In certain cases, montage code does not generate an output file even though the job itself succeeds.

      Because of this the stage out jobs fail as the input does not exist ( as they are never created in the first place )

      None of the underlying transfer tools we use ( like guc etc ) support the functionality of transferring as many files as possible. They fail on
      first error. That makes it difficult to incorporate the behaviour in pegasus-transfer,
      unless each transfer is invoked separately internally.

      However we probably should consider doing this only( separate invocation of underlying transfer tool ) for the optional files

      Another issue in this case would be that we could detect an error but there is no clear way to detect whether it
      was because of a missing source file, unless there are specific exitcodes for this case in GUC and other tools we use in pegasus-transfer

      For this there is a placeholder in the DAX to mark file transfer as optional.
      We need to come up with a way to let pegasus-transfer know that a particular file transfer is optional in the input file passed to it




            • Assignee:
              rynge-page Mats Rynge [X] (Inactive)
              vahi Karan Vahi
            • Watchers:
              0 Start watching this issue


              • Created: