Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-302

pegasus-transfer should handle optional files

XMLWordPrintable

    • Type: Icon: Improvement Improvement
    • Resolution: Won't Fix
    • Priority: Icon: Major Major
    • 3.1
    • Affects Version/s: None
    • None

      In certain cases, montage code does not generate an output file even though the job itself succeeds.

      Because of this the stage out jobs fail as the input does not exist ( as they are never created in the first place )

      None of the underlying transfer tools we use ( like guc etc ) support the functionality of transferring as many files as possible. They fail on
      first error. That makes it difficult to incorporate the behaviour in pegasus-transfer,
      unless each transfer is invoked separately internally.

      However we probably should consider doing this only( separate invocation of underlying transfer tool ) for the optional files

      Another issue in this case would be that we could detect an error but there is no clear way to detect whether it
      was because of a missing source file, unless there are specific exitcodes for this case in GUC and other tools we use in pegasus-transfer

      For this there is a placeholder in the DAX to mark file transfer as optional.
      We need to come up with a way to let pegasus-transfer know that a particular file transfer is optional in the input file passed to it

            Assignee:
            rynge-page Mats Rynge [X] (Inactive)
            Reporter:
            vahi Karan Vahi
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: