Parameter lists

Passing multiple inputs to a workflow

Using the Viash VDSL3 Nextflow platform, an optional --param_list argument can be passed to OpenPipelines workflows. The --param_list argument enables passing multiple inputs to a workflow, resulting in a multi-event nextflow channel.

Event-specific and global parameters

The creation of multi-event Nextflow channels has the advantage of passing event-specific parameters for each input file. In the following param_list file, a multi-event Nextflow channel is created for two input files: foo.txt and bar.txt. For each input, an id and additional parameters that are specific for each input can be passed as event-specific parameters (here: event_param).

$cat param_list.yaml
- id: foo
  input: foo.txt
  event_param: lorem
- id: bar
  input: bar.txt
  event_param: ipsum

nextflow run ... --param_list param_list.yaml

The event-specific parameters in the param_list file can be combined with global parameters that apply to all the input events. For example, combining the above param_list file with a global parameter --global_param:

nextflow run ... --param_list param_list.yaml --global_param baz

will result in the following events being processed:

id: foo, input: foo.txt, event_param: lorem, global_param: baz
id: bar, input: bar.txt, event_param: ipsum, global_param: baz

Note that event-sepcific parameters defined in the param_list will always overwrite global parameters. For example, running

nextflow run ... --param_list param_list.yaml --global_param baz

with the following param_list.yaml:

$cat param_list.yaml
- id: foo
  input: foo.txt
- id: bar
  input: bar.txt
  global_param: ipsum

Will result in the following events being processed:

id: foo, input: foo.txt, event_param: lorem
id: bar, input: bar.txt, event_param: ipsum

Other `param_list` file types

In addition to passing the param_list as a yaml file as seen in the examples above, it can also be passed as a json or csv file.

JSON file

The following example shows how to use a json file as a parameter list.

$ cat param_list.json
[
    {
        "id": "foo",
        "input": "foo.txt",
        "event_param": "lorem"
    },
    {
        "id": "bar",
        "input": "bar.txt",
        "event_param": "ipsum"
    }
]

nextflow run ... --param_list param_list.json

CSV file

The following example shows how to use a csv file as a parameter list.

$ cat param_list.csv
id,input,event_param
foo,foo.txt,lorem
bar,bar.txt,ipsum

nextflow run ... --param_list param_list.csv

Passing `param_list` to Nextflow’s `params-file`

The param_list can also be passed via Nextflow’s params-file, such that both the event-specific and global parameters are passed via a file. The following example shows how to use a params-file yamle file containing the param_list.

$ cat params-file.yaml
param_list:
  - id: foo
    input: foo.txt
    event_param: lorem
  - id: bar
    input: bar.txt
    event_param: ipsum
global_param: baz

nextflow run ... -params-file params-file.yaml

Resolving paths

All files defined in the param_list are relative to the location of the param_list file. For example, with a param_list.yaml file located in the data directory:

$ cat /data/param_list.yaml
- id: foo
  input: foo.txt
- id: bar
  input: /path/to/bar.txt

This will result in the following parameter sets being processed:

id: foo, input: /data/foo.txt
id: bar, input: /path/to/bar.txt

Note that this also works when the param list is located on a remote location, such as an S3 bucket. In that case, the files in the param list are relative to the location of the param list on the remote location.