The last couple of weeks we have discussed several times improvements to the PegasusLite worker package handling. The current logic for when PegasusLite starts is:

1. If the job came with a package, use it.
A warning will be issued if the arch mismatch

2. If a location was specified in the job env, use it iff:
The full version match between the planner and the worker package

3. Download a package from http://download.pegasus.isi.edu

The problem is now that we have two conflicting improvements requests. The first one is to skip step 3 and let the workflow fail. The reasoning is that 3. hides issues from the user, and failing will highlight the issue and give the user a chance to fix it.

A second improvement is to tighten 1 and 2 to make sure we can execute workflows in a more mixed execution environment. The idea is that the workflow might ship a default worker package with the jobs, but if the job lands on a different os/version, we will fall back to 3 and download the correct worker package. The downside is that we will see more hits on download.pegasus.isi.edu, but the upside is that Pegasus will work out of the box in more cases.

I think we can accommodate these two requirements by introducing a new setting to specify if downloading worker packages should be allowed or not. The planner will propagate the setting to the PegasusLite .sh, just like we do with the version today.

The new logic in this case would be:

1. If the job came with a package, use it iff:
The arch/os/version matches fully

2. If a location was specified in the job env, use it iff:
The full version match between the planner and the worker package

3. Download a package from http://download.pegasus.isi.edu iff:
pegasus_allow_worker_download is true

My suggestion is that pegasus_allow_worker_download defaults to true to allow a better user experience.