-
Type: Improvement
-
Resolution: Won't Fix
-
Priority: Major
-
Affects Version/s: None
-
Component/s: Planner: Transfer Module
-
None
In certain cases, montage code does not generate an output file even though the job itself succeeds.
Because of this the stage out jobs fail as the input does not exist ( as they are never created in the first place )
None of the underlying transfer tools we use ( like guc etc ) support the functionality of transferring as many files as possible. They fail on
first error. That makes it difficult to incorporate the behaviour in pegasus-transfer,
unless each transfer is invoked separately internally.
However we probably should consider doing this only( separate invocation of underlying transfer tool ) for the optional files
Another issue in this case would be that we could detect an error but there is no clear way to detect whether it
was because of a missing source file, unless there are specific exitcodes for this case in GUC and other tools we use in pegasus-transfer
For this there is a placeholder in the DAX to mark file transfer as optional.
We need to come up with a way to let pegasus-transfer know that a particular file transfer is optional in the input file passed to it