Support transfers for a job on the HOST OS instead of from within the container

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      As part of PM-1321 , in Pegasus, for containerized jobs, the data for the job gets pulled in when the container starts from within the container. This requires the container to have the pegasus worker package deployed within the container. While the pegasus worker package does get deployed inside the container at runtime, certain python dependencies still need to be fulfilled in the container build file.  This approach was selected to allow users to use transfer tools whose dependencies are not fulfilled on the hostos. 

      However, there are legitimate cases for the supporting data transfers on the Host OS also for containerized jobs. For example, for ML applications using tensor flow etc that have ntasks set to > 1. 

            Assignee:
            Karan Vahi
            Reporter:
            Karan Vahi
            Archiver:
            Rajiv Mayani

              Created:
              Updated:
              Archived: