Submitting Jobs
One of the novel features provided by Oliver is the ease with which workflow
inputs, options, and labels can be set dynamically on the command line.
Typically when submitting a workflow, one must craft and POST one or more workflowInputs
,
workflowOptions
, and labels
JSON files to Cromwell. This can be cumbersome
— especially when you have thousands of workflows to run.
Dynamic Parameter Parsing
Oliver provides an improvement over this experience for a few keys reasons:
- Inputs, options, and labels can be loaded from files or as key-value pairs on the command line.
- Inputs, options, and labels are processed sequentially and overwrite arguments that were provided earlier on the command line.
- Oliver distinguishes between inputs, options, and labels based on the prefix provided.
For instance, consider the following command:
oliver submit workflow.wdl defaults.json sample_name=SJBALL101_D
Here, Oliver loads workflow.wdl
and begins preparing the workflowInputs
(inputs), workflowOptions
(options), and labels
(labels) JSON objects — all of which are initialized as {}
.
- Oliver loads the JSON object defined in
defaults.json
. For each key-value in that object, it adds the pair toworkflowInputs
, overwriting the pair if it already exists. In this case, the JSON object is empty, soworkflowInputs
now mirrors what was contained indefaults.json
. - Oliver recognizes that you have provided
sample_name=SJBALL101_D
as a key value pair and adds this pair toworkflowInputs
, overwriting the key if it existed.
In this way, you can dynamically set and overwrite inputs with ease to satisfy your use case.
Prefixes
This process works similarly for all parameter types (inputs, options, or labels). To distinguish them to Oliver, you need to use the following prefixes:
Parameter Type | Prefix | Example |
---|---|---|
Input | <none> |
key=value |
Option | @ |
@key=value |
Label | % |
%key=value |
For example, consider the following command and note the process Oliver follows in the comments:
oliver submit workflow.wdl \
default-inputs.json \ # loads all values in the JSON file to the inputs object.
@default-options.json \ # loads all values in the JSON file to the options object.
%default-labels.json \ # loads all values in the JSON file to the labels object.
input_key=value \ # adds `input_key=value` to the inputs object (overwrites the value if `input_key` set in default-inputs.json).
@option_key=value \ # adds `option_key=value` to the options object (overwrites the value if `option_key` set in default-options.json).
%label_key=value \ # adds `label_key=value` to the labels object (overwrites the value if `label_key` set in default-labels.json).
Inspecting Parameters with Dry Runs
Often, it can be useful to inspect how Oliver is parsing the parameters you are passing on the command line. You can easily do so by specifying the dry-run argument.
oliver submit workflow.wdl defaults.json sample_name=SJBALL101_D --dry-run
Job Names and Groups
You can submit jobs with a "Oliver Job Name" (-j
) and "Oliver Job Group" (-g
) to mimic the capabilities of an HPC. Under the hood, Oliver adds these as labels to the workflow. In most Oliver commands, you can then specify these options to restrict results.
For instance, if you wished to submit a group of samples with the same job name and then track their status:
# Submit jobs
for SAMPLE in $SAMPLES; do
echo oliver submit $WORKFLOW_URL $DEFAULT_INPUTS_FILE \
input_bam=${SAMPLE}.bam \
output_prefix=${SAMPLE}. \
-j ${SAMPLE} -g CohortAlpha
done
# Check status of this job group
oliver status -g CohortAlpha