Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1343

pegasus submit problem

XMLWordPrintable

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major Major
    • master
    • Affects Version/s: 4.7.4
    • Component/s: Pegasus Planner
    • None
    • Environment:
      linux ubuntu 64

      Hi,When i use pegaus to submit task:
      pegasus-plan \
      --conf /home/ubuntu/pipline/pegasusrc \
      --sites condorlocal,condorpool \
      --output-site local \
      --dir /home/ubuntu/pipline/dags \
      --dax /home/ubuntu/pipline/dax_lianchuang.xml \
      --submit

      I meet a error message below:
      2019.01.16 08:21:20.837 UTC: [WARNING] File URL (file:///home/ubuntu/pipline/worker/run.py,

      {site=condorpool}) not included as the site attribute is a mismatch to the site name (condorlocal) allowLocalFileURLs true
      2019.01.16 08:21:20.837 UTC: [FATAL ERROR]
      [1] java.lang.RuntimeException: Unable to select a Physical Filename (PFN) for file with logical filename (LFN) as diamond-run-1.0 for preferred site condorlocalwith runTransferOnLocalSite - true amongst [(file:///home/ubuntu/pipline/worker/run.py,{site=condorpool}

      )] at edu.isi.pegasus.planner.refiner.TransferEngine.getFilesFromRC(TransferEngine.java:1412)

      My excecuatable is:
      run_cmd = Executable(namespace="diamond", name="run", version="1.0", os="linux", arch="x86_64")
      run_cmd.addPFN(PFN("file://" + work_dir + '/worker/run.py', config.CONDOR_POOL_SITE))
      run_cmd.installed = False
      diamond.addExecutable(run_cmd)

      My config file is:
      pegasus.catalog.site=XML
      pegasus.catalog.site.file=/home/ubuntu/pipline/sites.xml
      #
      #pegasus.dir.useTimestamp=true
      pegasus.dir.storage.deep=false
      pegasus.condor.logs.symlink=false
      #
      pegasus.data.configuration = condorio
      pegasus.monitord.notifications=true

      my sites.xml is:
      <?xml version="1.0" encoding="UTF-8"?>
      <sitecatalog xmlns="http://pegasus.isi.edu/schema/sitecatalog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://pegasus.isi.edu/schema/sitecatalog http://pegasus.isi.edu/schema/sc-4.0.xsd" version="4.0">
      <site handle="local" arch="x86_64" os="LINUX">
      <directory type="shared-scratch" path="/home/ubuntu/pipline/work">
      <file-server operation="all" url="file:///home/ubuntu/pipline/work"/>
      </directory>
      <directory type="local-storage" path="/home/ubuntu/pipline/outputs">
      <file-server operation="all" url="file:///home/ubuntu/pipline/outputs"/>
      </directory>
      </site>
      <site handle="condorpool" arch="x86_64" os="LINUX">
      <profile namespace="pegasus" key="style" >condor</profile>
      <profile namespace="condor" key="universe" >vanilla</profile>
      </site>
      <site handle="condorlocal" arch="x86_64" os="LINUX">
      <profile namespace="pegasus" key="style" >condor</profile>
      <profile namespace="condor" key="universe" >local</profile>
      </site>
      </sitecatalog>

      my dax.xml is:

      <?xml version="1.0" encoding="UTF-8"?>
      <!-- generated: 2019-01-17 06:59:57.156656 -->
      <!-- generated by: ubuntu -->
      <!-- generator: python -->
      <adag xmlns="http://pegasus.isi.edu/schema/DAX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://pegasus.isi.edu/schema/DAX http://pegasus.isi.edu/schema/dax-3.6.xsd" version="3.6" name="diamond">
      <metadata key="name">pipline</metadata>
      <metadata key="createdby">wukan</metadata>
      <executable name="undemand_reqeust" namespace="diamond" version="1.0" arch="x86_64" os="linux">
      <pfn url="file:///home/ubuntu/pipline/ec2.sh" site="condorlocal"/>
      </executable>
      <executable name="scan" namespace="diamond" version="1.0" arch="x86_64" os="linux">
      <pfn url="file:///home/ubuntu/pipline/scaner.py" site="condorlocal"/>
      </executable>
      <executable name="wes" namespace="diamond" version="1.0" arch="x86_64" os="linux" installed="false">
      <pfn url="file:///home/ubuntu/pipline/lianchuang/worker/wes_pipeline.py" site="condorpool"/>
      </executable>
      <executable name="term" namespace="diamond" version="2.0" arch="x86_64" os="linux">
      <pfn url="file:///home/ubuntu/pipline/term.py" site="condorlocal"/>
      </executable>
      <executable name="spot_request" namespace="diamond" version="1.0" arch="x86_64" os="linux">
      <pfn url="file:///home/ubuntu/pipline/ec2_spot.sh" site="condorlocal"/>
      </executable>
      <job id="ID0000001" namespace="diamond" name="spot_request" version="1.0">
      <argument>1 eu-west-1 eu-west-1c r4.xlarge 50 /home/ubuntu/pipline/OrganizationAccountAccessRole OrganizationAccountAccessRole ami-0fbc756353d10a5ac pipline_worker</argument>
      <profile namespace="env" key="HOME">/home/ubuntu</profile>
      </job>
      <job id="ID0000002" namespace="diamond" name="wes" version="1.0">
      <argument>A2018102201 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102201_BH5VGHDSXX_S1_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102201_BH5VGHDSXX_S1_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
      </job>
      <job id="ID0000003" namespace="diamond" name="wes" version="1.0">
      <argument>A2018102202 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102202_BH5VGHDSXX_S2_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102202_BH5VGHDSXX_S2_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
      </job>
      <job id="ID0000004" namespace="diamond" name="wes" version="1.0">
      <argument>A2018102203 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102203_BH5VGHDSXX_S3_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102203_BH5VGHDSXX_S3_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
      </job>
      <job id="ID0000005" namespace="diamond" name="wes" version="1.0">
      <argument>A2018102204 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102204_BH5VGHDSXX_S4_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102204_BH5VGHDSXX_S4_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
      </job>
      <job id="ID0000006" namespace="diamond" name="wes" version="1.0">
      <argument>A2018102205 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102205_BH5VGHDSXX_S5_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102205_BH5VGHDSXX_S5_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
      </job>
      <job id="ID0000007" namespace="diamond" name="wes" version="1.0">
      <argument>A2018102206 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102206_BH5VGHDSXX_S6_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102206_BH5VGHDSXX_S6_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
      </job>
      <job id="ID0000008" namespace="diamond" name="wes" version="1.0">
      <argument>A2018102207 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102207_BH5VGHDSXX_S7_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102207_BH5VGHDSXX_S7_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
      </job>
      <job id="ID0000009" namespace="diamond" name="wes" version="1.0">
      <argument>A2018102208 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102208_BH5VGHDSXX_S8_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102208_BH5VGHDSXX_S8_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
      </job>
      </adag>

      the full content of sites.xml pegasusrc ,and dax.xml is attached below.

        1. sites.xml
          1 kB
        2. pegasusrc
          0.2 kB
        3. dax_lianchuang.xml
          27 kB

            Assignee:
            vahi Karan Vahi
            Reporter:
            wekay102200 kan.wu
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: