Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1321

Move transfer staging into the container rather than the host OS


      Hi Karan, Mats,

      I traced the problem of the missing df to the following:

      1. I was submitting from a RHEL7 machine which only has /usr/bin in PATH (on RHEL7, /bin is a symlink to /usr/bin). I had getenv = 2. True in the submit file.
      3. The execute machine was Ubutu which has both /bin and /usr/bin. df is in /bin.
      4. The grid start script couldn't find df, as /bin was not in PATH.

      The solution was to set a profile that forces getenv = False for this site. However, I then ran into the problem that this is a bare bones Ubuntu system so it doesn't have gfal-copy, etc. so pegasus-transfer fails.

      I have a pull request here that implements a solution:


      My fix the same as Karan's PM-1190 fix for checking the file integrity: I moved the input and output transfers inside the wrapper script that's run in the container, not the wrapper script that's run on the host. Only the container itself is staged outside the container.

      I'm running a few tests now, but the .sh files constructed look reasonable to me. Can you try running this through the standard Pegasus test suite?


        There are no Sub-Tasks for this issue.

            vahi Karan Vahi
            dbrown Duncan Brown
            2 Start watching this issue