-
Type: Sub-task
-
Resolution: Fixed
-
Priority: Major
-
Affects Version/s: master
-
Component/s: Integrity Checking, Pegasus Lite, Pegasus Planner
-
None
currently integrity checking is invoked in PegasusLite outside of the container invocation, because of which integrity checks fail when the existing data is mounted in the container in a different directory.
for example our bamboo tests case fail now because of this
var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN
28-Sep-2018 10:04:16
28-Sep-2018 10:04:16 ##############[Pegasus Lite] Figuring out the worker package to use ##############
28-Sep-2018 10:04:16 2018-09-28 10:03:08: The job contained a Pegasus worker package
28-Sep-2018 10:04:16
28-Sep-2018 10:04:16 ###############[Pegasus Lite] Staging in input data and executables ###############
28-Sep-2018 10:04:16 2018-09-28 10:03:09,720 INFO: Reading URL pairs from stdin
28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: 3 transfers loaded
28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: PATH=/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/pegasus-4.9.0dev/bin:/usr/local/bin:/usr/bin:/bin
28-Sep-2018 10:04:16 2018-09-28 10:03:09,721 INFO: LD_LIBRARY_PATH=
28-Sep-2018 10:04:16 2018-09-28 10:03:09,815 INFO: --------------------------------------------------------------------------------
28-Sep-2018 10:04:16 2018-09-28 10:03:09,815 INFO: Starting transfers - attempt 1
28-Sep-2018 10:04:16 2018-09-28 10:03:11,992 INFO: Tool found: docker Version: 18.06.1 Path: /usr/bin/docker
28-Sep-2018 10:04:16 2018-09-28 10:03:12,002 INFO: /usr/bin/docker pull 'centos:7' && /usr/bin/docker save -o '/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/centos-base' 'centos:7'
28-Sep-2018 10:04:16 2018-09-28 10:03:17,503 INFO: 7: Pulling from library/centos
28-Sep-2018 10:04:16 Digest: sha256:6f6d986d425aeabdc3a02cb61c02abb2e78e57357e92417d6d58332856024faf
28-Sep-2018 10:04:16 Status: Image is up to date for centos:7
28-Sep-2018 10:04:16 2018-09-28 10:03:17,766 INFO: Tool found: gfal-copy Version: 2.15.4 Path: /usr/bin/gfal-copy
28-Sep-2018 10:04:16 2018-09-28 10:03:17,766 INFO: gfal-copy -f -p 'gsiftp://bamboo.isi.edu/lfs1/software/bamboo/data/xml-data/build-dir/PEGASUS-WT49-T044F/test/core/044-docker-nonsharedfs-symlink/staging-site/scratch/044-docker-nonsharedfs-symlink/bamboo/pegasus/diamond/run0001/00/00/diamond-preprocess-4_0' 'file:///var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/diamond-preprocess-4_0'
28-Sep-2018 10:04:16 2018-09-28 10:03:19,164 INFO: Copying 83064 bytes gsiftp://bamboo.isi.edu/lfs1/software/bamboo/data/xml-data/build-dir/PEGASUS-WT49-T044F/test/core/044-docker-nonsharedfs-symlink/staging-site/scratch/044-docker-nonsharedfs-symlink/bamboo/pegasus/diamond/run0001/00/00/diamond-preprocess-4_0 => file:///var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/diamond-preprocess-4_0
28-Sep-2018 10:04:16 2018-09-28 10:03:19,165 INFO: ln -f -s '/existing/data/f.a' '/var/lib/condor/execute/dir_14184/pegasus.XPXOSGxTN/f.a'
28-Sep-2018 10:04:16 2018-09-28 10:03:19,185 INFO: --------------------------------------------------------------------------------
28-Sep-2018 10:04:16 2018-09-28 10:03:19,185 INFO: Stats: Total 3 transfers, 198.7 MB transferred in 9 seconds. Rate: 21.0 MB/s (167.9 Mb/s)
28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: Between sites cartman_data->condorpool : 1 transfers, 81.1 KB transferred in 9 seconds. Rate: 8.6 KB/s (68.6 Kb/s)
28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: Between sites condorpool->condorpool : 2 transfers, 198.7 MB transferred in 9 seconds. Rate: 21.0 MB/s (167.9 Mb/s)
28-Sep-2018 10:04:16 2018-09-28 10:03:19,186 INFO: All transfers completed successfully.
28-Sep-2018 10:04:16
28-Sep-2018 10:04:16 ##############[Pegasus Lite] Setting the xbit for executables staged ##############
28-Sep-2018 10:04:16
28-Sep-2018 10:04:16 ##############[Pegasus Lite] Checking file integrity for input files ##############
28-Sep-2018 10:04:16 Integrity check: File f.a does not exist
28-Sep-2018 10:04:16 Integrity check: f.a: Expected checksum (ca8ed5988cb4ca0b67c45fd80fd17423aba2a066ca8a63a4e1c6adab067a3e92) does not match the calculated checksum (None) (timing: 0.032)