Currently, when cleanup is enabled, we set job priorities for user jobs based on level in the workflow. The problem is that we leave some Pegasus added jobs at priority 0. This causes starvation of new workflows entering a busy system, and will not overlap data staging with computation jobs correctly. My suggestion is that

1) we always set job priorities, cleanup enabled or disabled

2) we set the default priorities:

Cleanup: 1000
Stage out: 900
Dirmanager: 800
Stage in: 700
Compute jobs: level based like we have now

This follows the the current level based thinking (give priority to workflows which are further along) but also overlap data staging and computation (data jobs have higher priority, but the assumptions is that there are relatively few data staging jobs compared to compute jobs)