Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1298 symlinking when running with containers
  3. PM-1305

Integrity checking breaks for symlinks in containers with different mount points

XMLWordPrintable

      currently integrity checking is invoked in PegasusLite outside of the container invocation, because of which integrity checks fail when the existing data is mounted in the container in a different directory.

      for example our bamboo tests case fail now because of this

      var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN
      28-Sep-2018 10:04:16
      28-Sep-2018 10:04:16 ##############[Pegasus Lite] Figuring out the worker package to use ##############
      28-Sep-2018 10:04:16 2018-09-28 10:03:08: The job contained a Pegasus worker package
      28-Sep-2018 10:04:16
      28-Sep-2018 10:04:16 ###############[Pegasus Lite] Staging in input data and executables ###############
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,720 INFO: Reading URL pairs from stdin
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: 3 transfers loaded
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: PATH=/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/pegasus-4.9.0dev/bin:/usr/local/bin:/usr/bin:/bin
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: LD_LIBRARY_PATH=
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,815 INFO: --------------------------------------------------------------------------------
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,815 INFO: Starting transfers - attempt 1
      28-Sep-2018 10:04:16 2018-09-28 10:03:11,992 INFO: Tool found: docker Version: 18.06.1 Path: /usr/bin/docker
      28-Sep-2018 10:04:16 2018-09-28 10:03:12,002 INFO: /usr/bin/docker pull 'centos:7' && /usr/bin/docker save -o '/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/centos-base' 'centos:7'
      28-Sep-2018 10:04:16 2018-09-28 10:03:17,503 INFO: 7: Pulling from library/centos
      28-Sep-2018 10:04:16 Digest: sha256:6f6d986d425aeabdc3a02cb61c02abb2e78e57357e92417d6d58332856024faf
      28-Sep-2018 10:04:16 Status: Image is up to date for centos:7
      28-Sep-2018 10:04:16 2018-09-28 10:03:17,766 INFO: Tool found: gfal-copy Version: 2.15.4 Path: /usr/bin/gfal-copy
      28-Sep-2018 10:04:16 2018-09-28 10:03:17,766 INFO: gfal-copy -f -p 'gsiftp://bamboo.isi.edu/lfs1/software/bamboo/data/xml-data/build-dir/PEGASUS-WT49-T044F/test/core/044-docker-nonsharedfs-symlink/staging-site/scratch/044-docker-nonsharedfs-symlink/bamboo/pegasus/diamond/run0001/00/00/diamond-preprocess-4_0' 'file:///var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/diamond-preprocess-4_0'
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,164 INFO: Copying 83064 bytes gsiftp://bamboo.isi.edu/lfs1/software/bamboo/data/xml-data/build-dir/PEGASUS-WT49-T044F/test/core/044-docker-nonsharedfs-symlink/staging-site/scratch/044-docker-nonsharedfs-symlink/bamboo/pegasus/diamond/run0001/00/00/diamond-preprocess-4_0 => file:///var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/diamond-preprocess-4_0
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,165 INFO: ln -f -s '/existing/data/f.a' '/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/f.a'
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,185 INFO: --------------------------------------------------------------------------------
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,185 INFO: Stats: Total 3 transfers, 198.7 MB transferred in 9 seconds. Rate: 21.0 MB/s (167.9 Mb/s)
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: Between sites cartman_data->condorpool : 1 transfers, 81.1 KB transferred in 9 seconds. Rate: 8.6 KB/s (68.6 Kb/s)
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: Between sites condorpool->condorpool : 2 transfers, 198.7 MB transferred in 9 seconds. Rate: 21.0 MB/s (167.9 Mb/s)
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: All transfers completed successfully.
      28-Sep-2018 10:04:16
      28-Sep-2018 10:04:16 ##############[Pegasus Lite] Setting the xbit for executables staged ##############
      28-Sep-2018 10:04:16
      28-Sep-2018 10:04:16 ##############[Pegasus Lite] Checking file integrity for input files ##############
      28-Sep-2018 10:04:16 Integrity check: File f.a does not exist
      28-Sep-2018 10:04:16 Integrity check: f.a: Expected checksum (ca8ed5988cb4ca0b67c45fd80fd17423aba2a066ca8a63a4e1c6adab067a3e92) does not match the calculated checksum (None) (timing: 0.032)

            Assignee:
            vahi Karan Vahi
            Reporter:
            vahi Karan Vahi
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: