-
Type: Bug
-
Resolution: Won't Fix
-
Priority: Major
-
Affects Version/s: 4.7.4
-
Component/s: Pegasus Planner
-
None
-
Environment:linux ubuntu 64
Hi,When i use pegaus to submit task:
pegasus-plan \
--conf /home/ubuntu/pipline/pegasusrc \
--sites condorlocal,condorpool \
--output-site local \
--dir /home/ubuntu/pipline/dags \
--dax /home/ubuntu/pipline/dax_lianchuang.xml \
--submit
I meet a error message below:
2019.01.16 08:21:20.837 UTC: [WARNING] File URL (file:///home/ubuntu/pipline/worker/run.py,
2019.01.16 08:21:20.837 UTC: [FATAL ERROR]
[1] java.lang.RuntimeException: Unable to select a Physical Filename (PFN) for file with logical filename (LFN) as diamond-run-1.0 for preferred site condorlocalwith runTransferOnLocalSite - true amongst [(file:///home/ubuntu/pipline/worker/run.py,{site=condorpool}
)] at edu.isi.pegasus.planner.refiner.TransferEngine.getFilesFromRC(TransferEngine.java:1412)
My excecuatable is:
run_cmd = Executable(namespace="diamond", name="run", version="1.0", os="linux", arch="x86_64")
run_cmd.addPFN(PFN("file://" + work_dir + '/worker/run.py', config.CONDOR_POOL_SITE))
run_cmd.installed = False
diamond.addExecutable(run_cmd)
My config file is:
pegasus.catalog.site=XML
pegasus.catalog.site.file=/home/ubuntu/pipline/sites.xml
#
#pegasus.dir.useTimestamp=true
pegasus.dir.storage.deep=false
pegasus.condor.logs.symlink=false
#
pegasus.data.configuration = condorio
pegasus.monitord.notifications=true
my sites.xml is:
<?xml version="1.0" encoding="UTF-8"?>
<sitecatalog xmlns="http://pegasus.isi.edu/schema/sitecatalog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://pegasus.isi.edu/schema/sitecatalog http://pegasus.isi.edu/schema/sc-4.0.xsd" version="4.0">
<site handle="local" arch="x86_64" os="LINUX">
<directory type="shared-scratch" path="/home/ubuntu/pipline/work">
<file-server operation="all" url="file:///home/ubuntu/pipline/work"/>
</directory>
<directory type="local-storage" path="/home/ubuntu/pipline/outputs">
<file-server operation="all" url="file:///home/ubuntu/pipline/outputs"/>
</directory>
</site>
<site handle="condorpool" arch="x86_64" os="LINUX">
<profile namespace="pegasus" key="style" >condor</profile>
<profile namespace="condor" key="universe" >vanilla</profile>
</site>
<site handle="condorlocal" arch="x86_64" os="LINUX">
<profile namespace="pegasus" key="style" >condor</profile>
<profile namespace="condor" key="universe" >local</profile>
</site>
</sitecatalog>
my dax.xml is:
<?xml version="1.0" encoding="UTF-8"?>
<!-- generated: 2019-01-17 06:59:57.156656 -->
<!-- generated by: ubuntu -->
<!-- generator: python -->
<adag xmlns="http://pegasus.isi.edu/schema/DAX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://pegasus.isi.edu/schema/DAX http://pegasus.isi.edu/schema/dax-3.6.xsd" version="3.6" name="diamond">
<metadata key="name">pipline</metadata>
<metadata key="createdby">wukan</metadata>
<executable name="undemand_reqeust" namespace="diamond" version="1.0" arch="x86_64" os="linux">
<pfn url="file:///home/ubuntu/pipline/ec2.sh" site="condorlocal"/>
</executable>
<executable name="scan" namespace="diamond" version="1.0" arch="x86_64" os="linux">
<pfn url="file:///home/ubuntu/pipline/scaner.py" site="condorlocal"/>
</executable>
<executable name="wes" namespace="diamond" version="1.0" arch="x86_64" os="linux" installed="false">
<pfn url="file:///home/ubuntu/pipline/lianchuang/worker/wes_pipeline.py" site="condorpool"/>
</executable>
<executable name="term" namespace="diamond" version="2.0" arch="x86_64" os="linux">
<pfn url="file:///home/ubuntu/pipline/term.py" site="condorlocal"/>
</executable>
<executable name="spot_request" namespace="diamond" version="1.0" arch="x86_64" os="linux">
<pfn url="file:///home/ubuntu/pipline/ec2_spot.sh" site="condorlocal"/>
</executable>
<job id="ID0000001" namespace="diamond" name="spot_request" version="1.0">
<argument>1 eu-west-1 eu-west-1c r4.xlarge 50 /home/ubuntu/pipline/OrganizationAccountAccessRole OrganizationAccountAccessRole ami-0fbc756353d10a5ac pipline_worker</argument>
<profile namespace="env" key="HOME">/home/ubuntu</profile>
</job>
<job id="ID0000002" namespace="diamond" name="wes" version="1.0">
<argument>A2018102201 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102201_BH5VGHDSXX_S1_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102201_BH5VGHDSXX_S1_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
</job>
<job id="ID0000003" namespace="diamond" name="wes" version="1.0">
<argument>A2018102202 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102202_BH5VGHDSXX_S2_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102202_BH5VGHDSXX_S2_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
</job>
<job id="ID0000004" namespace="diamond" name="wes" version="1.0">
<argument>A2018102203 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102203_BH5VGHDSXX_S3_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102203_BH5VGHDSXX_S3_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
</job>
<job id="ID0000005" namespace="diamond" name="wes" version="1.0">
<argument>A2018102204 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102204_BH5VGHDSXX_S4_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102204_BH5VGHDSXX_S4_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
</job>
<job id="ID0000006" namespace="diamond" name="wes" version="1.0">
<argument>A2018102205 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102205_BH5VGHDSXX_S5_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102205_BH5VGHDSXX_S5_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
</job>
<job id="ID0000007" namespace="diamond" name="wes" version="1.0">
<argument>A2018102206 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102206_BH5VGHDSXX_S6_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102206_BH5VGHDSXX_S6_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
</job>
<job id="ID0000008" namespace="diamond" name="wes" version="1.0">
<argument>A2018102207 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102207_BH5VGHDSXX_S7_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102207_BH5VGHDSXX_S7_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
</job>
<job id="ID0000009" namespace="diamond" name="wes" version="1.0">
<argument>A2018102208 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102208_BH5VGHDSXX_S8_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102208_BH5VGHDSXX_S8_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
</job>
</adag>
the full content of sites.xml pegasusrc ,and dax.xml is attached below.