-
Type: New Feature
-
Resolution: Unresolved
-
Priority: Major
-
Affects Version/s: master, 5.0.7
-
Component/s: Planner: Containers, Planner: Transfer Module
-
None
As part of PM-1321 , in Pegasus, for containerized jobs, the data for the job gets pulled in when the container starts from within the container. This requires the container to have the pegasus worker package deployed within the container. While the pegasus worker package does get deployed inside the container at runtime, certain python dependencies still need to be fulfilled in the container build file. This approach was selected to allow users to use transfer tools whose dependencies are not fulfilled on the hostos.
However, there are legitimate cases for the supporting data transfers on the Host OS also for containerized jobs. For example, for ML applications using tensor flow etc that have ntasks set to > 1.