Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1298 symlinking when running with containers
  3. PM-1305

Integrity checking breaks for symlinks in containers with different mount points

    XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: master
    • Fix Version/s: master, 4.9.0
    • Component/s: Pegasus Lite, pegasus-plan
    • Labels:
      None

      Description

      currently integrity checking is invoked in PegasusLite outside of the container invocation, because of which integrity checks fail when the existing data is mounted in the container in a different directory.

      for example our bamboo tests case fail now because of this

      var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN
      28-Sep-2018 10:04:16
      28-Sep-2018 10:04:16 ##############[Pegasus Lite] Figuring out the worker package to use ##############
      28-Sep-2018 10:04:16 2018-09-28 10:03:08: The job contained a Pegasus worker package
      28-Sep-2018 10:04:16
      28-Sep-2018 10:04:16 ###############[Pegasus Lite] Staging in input data and executables ###############
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,720 INFO: Reading URL pairs from stdin
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: 3 transfers loaded
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: PATH=/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/pegasus-4.9.0dev/bin:/usr/local/bin:/usr/bin:/bin
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: LD_LIBRARY_PATH=
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,815 INFO: --------------------------------------------------------------------------------
      28-Sep-2018 10:04:16 2018-09-28 10:03:09,815 INFO: Starting transfers - attempt 1
      28-Sep-2018 10:04:16 2018-09-28 10:03:11,992 INFO: Tool found: docker Version: 18.06.1 Path: /usr/bin/docker
      28-Sep-2018 10:04:16 2018-09-28 10:03:12,002 INFO: /usr/bin/docker pull 'centos:7' && /usr/bin/docker save -o '/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/centos-base' 'centos:7'
      28-Sep-2018 10:04:16 2018-09-28 10:03:17,503 INFO: 7: Pulling from library/centos
      28-Sep-2018 10:04:16 Digest: sha256:6f6d986d425aeabdc3a02cb61c02abb2e78e57357e92417d6d58332856024faf
      28-Sep-2018 10:04:16 Status: Image is up to date for centos:7
      28-Sep-2018 10:04:16 2018-09-28 10:03:17,766 INFO: Tool found: gfal-copy Version: 2.15.4 Path: /usr/bin/gfal-copy
      28-Sep-2018 10:04:16 2018-09-28 10:03:17,766 INFO: gfal-copy -f -p 'gsiftp://bamboo.isi.edu/lfs1/software/bamboo/data/xml-data/build-dir/PEGASUS-WT49-T044F/test/core/044-docker-nonsharedfs-symlink/staging-site/scratch/044-docker-nonsharedfs-symlink/bamboo/pegasus/diamond/run0001/00/00/diamond-preprocess-4_0' 'file:///var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/diamond-preprocess-4_0'
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,164 INFO: Copying 83064 bytes gsiftp://bamboo.isi.edu/lfs1/software/bamboo/data/xml-data/build-dir/PEGASUS-WT49-T044F/test/core/044-docker-nonsharedfs-symlink/staging-site/scratch/044-docker-nonsharedfs-symlink/bamboo/pegasus/diamond/run0001/00/00/diamond-preprocess-4_0 => file:///var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/diamond-preprocess-4_0
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,165 INFO: ln -f -s '/existing/data/f.a' '/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/f.a'
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,185 INFO: --------------------------------------------------------------------------------
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,185 INFO: Stats: Total 3 transfers, 198.7 MB transferred in 9 seconds. Rate: 21.0 MB/s (167.9 Mb/s)
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: Between sites cartman_data->condorpool : 1 transfers, 81.1 KB transferred in 9 seconds. Rate: 8.6 KB/s (68.6 Kb/s)
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: Between sites condorpool->condorpool : 2 transfers, 198.7 MB transferred in 9 seconds. Rate: 21.0 MB/s (167.9 Mb/s)
      28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: All transfers completed successfully.
      28-Sep-2018 10:04:16
      28-Sep-2018 10:04:16 ##############[Pegasus Lite] Setting the xbit for executables staged ##############
      28-Sep-2018 10:04:16
      28-Sep-2018 10:04:16 ##############[Pegasus Lite] Checking file integrity for input files ##############
      28-Sep-2018 10:04:16 Integrity check: File f.a does not exist
      28-Sep-2018 10:04:16 Integrity check: f.a: Expected checksum (ca8ed5988cb4ca0b67c45fd80fd17423aba2a066ca8a63a4e1c6adab067a3e92) does not match the calculated checksum (None) (timing: 0.032)

        Attachments

          Activity

            People

            • Assignee:
              vahi Karan Vahi
              Reporter:
              vahi Karan Vahi
            • Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: