Uploaded image for project: 'Pegasus'
  1. Pegasus
  2. PM-1343

pegasus submit problem

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.7.4
    • Fix Version/s: None
    • Component/s: pegasus-plan
    • Labels:
      None
    • Environment:
      linux ubuntu 64

      Description

      Hi,When i use pegaus to submit task:
      pegasus-plan \
      --conf /home/ubuntu/pipline/pegasusrc \
      --sites condorlocal,condorpool \
      --output-site local \
      --dir /home/ubuntu/pipline/dags \
      --dax /home/ubuntu/pipline/dax_lianchuang.xml \
      --submit

       I meet a error message below:
      2019.01.16 08:21:20.837 UTC: [WARNING] File URL (file:///home/ubuntu/pipline/worker/run.py,{site=condorpool}) not included as the site attribute is a mismatch to the site name (condorlocal) allowLocalFileURLs true
      2019.01.16 08:21:20.837 UTC: [FATAL ERROR]
       [1] java.lang.RuntimeException: Unable to select a Physical Filename (PFN) for file with logical filename (LFN) as diamond-run-1.0 for preferred site condorlocalwith runTransferOnLocalSite - true amongst [(file:///home/ubuntu/pipline/worker/run.py,{site=condorpool})] at edu.isi.pegasus.planner.refiner.TransferEngine.getFilesFromRC(TransferEngine.java:1412)


      My excecuatable is:
          run_cmd = Executable(namespace="diamond", name="run", version="1.0", os="linux", arch="x86_64")
          run_cmd.addPFN(PFN("file://" + work_dir + '/worker/run.py', config.CONDOR_POOL_SITE))
          run_cmd.installed = False
          diamond.addExecutable(run_cmd)

      My config file is:
      pegasus.catalog.site=XML
      pegasus.catalog.site.file=/home/ubuntu/pipline/sites.xml
      #
      #pegasus.dir.useTimestamp=true
      pegasus.dir.storage.deep=false
      pegasus.condor.logs.symlink=false
      #
      pegasus.data.configuration = condorio
      pegasus.monitord.notifications=true

      my sites.xml is:
      <?xml version="1.0" encoding="UTF-8"?>
          <sitecatalog xmlns="http://pegasus.isi.edu/schema/sitecatalog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://pegasus.isi.edu/schema/sitecatalog http://pegasus.isi.edu/schema/sc-4.0.xsd" version="4.0">
              <site handle="local" arch="x86_64" os="LINUX">
                  <directory type="shared-scratch" path="/home/ubuntu/pipline/work">
                      <file-server operation="all" url="file:///home/ubuntu/pipline/work"/>
                  </directory>
                  <directory type="local-storage" path="/home/ubuntu/pipline/outputs">
                      <file-server operation="all" url="file:///home/ubuntu/pipline/outputs"/>
                  </directory>
              </site>
              <site handle="condorpool" arch="x86_64" os="LINUX">
                  <profile namespace="pegasus" key="style" >condor</profile>
                  <profile namespace="condor" key="universe" >vanilla</profile>
              </site>
              <site handle="condorlocal" arch="x86_64" os="LINUX">
                  <profile namespace="pegasus" key="style" >condor</profile>
                  <profile namespace="condor" key="universe" >local</profile>
              </site>
          </sitecatalog>

      my dax.xml is:

      <?xml version="1.0" encoding="UTF-8"?>
      <!-- generated: 2019-01-17 06:59:57.156656 -->
      <!-- generated by: ubuntu -->
      <!-- generator: python -->
      <adag xmlns="http://pegasus.isi.edu/schema/DAX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://pegasus.isi.edu/schema/DAX http://pegasus.isi.edu/schema/dax-3.6.xsd" version="3.6" name="diamond">
              <metadata key="name">pipline</metadata>
              <metadata key="createdby">wukan</metadata>
              <executable name="undemand_reqeust" namespace="diamond" version="1.0" arch="x86_64" os="linux">
                      <pfn url="file:///home/ubuntu/pipline/ec2.sh" site="condorlocal"/>
              </executable>
              <executable name="scan" namespace="diamond" version="1.0" arch="x86_64" os="linux">
                      <pfn url="file:///home/ubuntu/pipline/scaner.py" site="condorlocal"/>
              </executable>
              <executable name="wes" namespace="diamond" version="1.0" arch="x86_64" os="linux" installed="false">
                      <pfn url="file:///home/ubuntu/pipline/lianchuang/worker/wes_pipeline.py" site="condorpool"/>
              </executable>
              <executable name="term" namespace="diamond" version="2.0" arch="x86_64" os="linux">
                      <pfn url="file:///home/ubuntu/pipline/term.py" site="condorlocal"/>
              </executable>
              <executable name="spot_request" namespace="diamond" version="1.0" arch="x86_64" os="linux">
                      <pfn url="file:///home/ubuntu/pipline/ec2_spot.sh" site="condorlocal"/>
              </executable>
              <job id="ID0000001" namespace="diamond" name="spot_request" version="1.0">
                      <argument>1 eu-west-1 eu-west-1c r4.xlarge 50 /home/ubuntu/pipline/OrganizationAccountAccessRole OrganizationAccountAccessRole ami-0fbc756353d10a5ac pipline_worker</argument>
                      <profile namespace="env" key="HOME">/home/ubuntu</profile>
              </job>
              <job id="ID0000002" namespace="diamond" name="wes" version="1.0">
                      <argument>A2018102201 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102201_BH5VGHDSXX_S1_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102201_BH5VGHDSXX_S1_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
              </job>
              <job id="ID0000003" namespace="diamond" name="wes" version="1.0">
                      <argument>A2018102202 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102202_BH5VGHDSXX_S2_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102202_BH5VGHDSXX_S2_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
              </job>
              <job id="ID0000004" namespace="diamond" name="wes" version="1.0">
                      <argument>A2018102203 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102203_BH5VGHDSXX_S3_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102203_BH5VGHDSXX_S3_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
              </job>
              <job id="ID0000005" namespace="diamond" name="wes" version="1.0">
                      <argument>A2018102204 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102204_BH5VGHDSXX_S4_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102204_BH5VGHDSXX_S4_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
              </job>
              <job id="ID0000006" namespace="diamond" name="wes" version="1.0">
                      <argument>A2018102205 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102205_BH5VGHDSXX_S5_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102205_BH5VGHDSXX_S5_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
              </job>
              <job id="ID0000007" namespace="diamond" name="wes" version="1.0">
                      <argument>A2018102206 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102206_BH5VGHDSXX_S6_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102206_BH5VGHDSXX_S6_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
              </job>
              <job id="ID0000008" namespace="diamond" name="wes" version="1.0">
                      <argument>A2018102207 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102207_BH5VGHDSXX_S7_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102207_BH5VGHDSXX_S7_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
              </job>
              <job id="ID0000009" namespace="diamond" name="wes" version="1.0">
                      <argument>A2018102208 sz-hapdeliver GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102208_BH5VGHDSXX_S8_L002_R1_001.fastq.gz GMPHP18F-02/Novaseq/HP18K003SYnova-1/A2018102208_BH5VGHDSXX_S8_L002_R2_001.fastq.gz role-biz-gene-data data/lianchuang/output-1 wukan 4 20 oss OrganizationAccountAccessRole</argument>
              </job>
      </adag>

      the full content of sites.xml pegasusrc ,and dax.xml is attached below.
                                                  

        Attachments

        1. dax_lianchuang.xml
          27 kB
        2. pegasusrc
          0.2 kB
        3. sites.xml
          1 kB

          Activity

            People

            • Assignee:
              vahi Karan Vahi
              Reporter:
              wekay102200 kan.wu
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: