Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1179 Integrity checking in Pegasus
  3. PM-1251

pegasus-transfer to checksum files


      For integrity checking in Pegasus, we have cases where a user may not have checksums available for all inputs. In that case, we want to be able to compute the checksums for raw inputs as part of the workflow.

      Two Options are possible
      1) One solution is to add jobs to the workflow to compute checksums for these input files.
      2) pegasus-transfer while pulling or pushing the data computes the checksums.

      2) has the added advantage of trying to compute checksums when a file is pulled for the workflow. In case of 1) the separate jobs are decoupled from the transfer jobs and may always need to pull in files to compute the checksums

            rynge Mats Rynge
            vahi Karan Vahi
            2 Start watching this issue