6 Advanced Protocols

Advanced protocols can contain many more elements than the simple protocols described in the previous chapter. In this chapter we’ll discuss parameters, objects, data properties, variable scope, substitution, directives, and template functions.

6.1 Parameters

Parameters are named values that you define at the beginning of a protocol. You can then use the parameter name instead of the value later in the protocol. In this partial example, a VOLUME parameter is defined which is used in the pipetter.pipette step:

parameters:
  VOLUME:
    description: "amount of water to dispense"
    value: 200 ul
...
steps:
  1:
    command: pipetter.pipette
    sources: water
    destinations: plate1(all)
    volumes: $#VOLUME

A parameter should be given a description and a value. Notice that $# prefixes VOLUME in the step; $# tells Roboliq to substitute in a parameter value. Substitution is discussed in more detail later in this chapter.

6.2 Objects

There are several more object types you might want to use in advanced protocols:

Data: for defining a data table. The Data type facilitates complex experimental designs, and you can read more about it in the next section and in the chapter on Design Tables.

Variable: For defining references to other variables. Variables are not particularly useful in Roboliq, but they could potentially be used to easily switch between objects in case you have, for example, two water sources water1 and water2. In that case you could have a water variable whose value to set to the source you want to use:

objects:
  water1: ...
  water2: ...
  water:
    description: "water source"
    type: Variable
    value: water1
steps:
  1:
    command: pipetter.pipette
    sources: water
    destinations: plate1(all)
    volumes: 50 ul

Template: You can use a template to define re-usable steps. Here is a toy example for a template named dispenseToPlate1 which creates a pipetter.pipette command that transfers a volume of water to all wells on plate1:

objects:
  plate1:
    type: Plate
    ...
  dispenseToPlate1:
    description: "template to dispense `volume` ul of water to all wells on plate1"
    type: Template
    template:
      command: pipetter.pipette
      sources: water
      destinations: plate1(all)
      volumes: "{{volume}}"
steps:
  1:
    command: system.call
    name: dispenseToPlate1
    params:
      volume: 10 ul
  2:
    command: system.call
    name: dispenseToPlate1
    params:
      volume: 50 ul

This protocol will first dispense 10ul to all wells, then another 50ul to all wells – not actually a useful protocol, but it illustrates the point. In the system.call command, the name of the template is specified along with the parameters template parameters. Templates are expanded using the Handlebars template engine, which is the reason for the “{{” and “}}” delimiters in the line volumes: "{{volume}}".

6.3 Data

Roboliq supports data tables to enable complex experiments. Conceptually, a data table is like a spread sheet of rows and named columns, where each row represents some “thing”, a each column represents a property. In Roboliq, a data table is an array of JSON objects: each object is a row, and each property is a column. Normally all the objects will have the same set of properties (but this is not required).

Data tables are supported by a Data type, a data property, a data() directive, and a set of data.* commands.

6.3.1 Data type

You can define a data table using the Data object type. Here’s an example where each row has a well, a liquid source, and a volume:

objects:
  data1:
    type: Data
    value:
      - {well: A01, volume: 10 ul, source: liquid1}
      - {well: B01, volume: 10 ul, source: liquid2}
      - {well: A02, volume: 20 ul, source: liquid1}
      - {well: B02, volume: 20 ul, source: liquid2}

You can define as many data tables as you want in a protocol.

6.3.2 data property

After defining a data table, you need to “activate” it for usage. This is done using the data property, which understands the following parameters:

• source: this is the name of a Data object.
• where: this is an optional boolean mathjs expression that is evaluated for each data row – only rows for which the expression evaluates to true are activated.
• orderBy: an optional array of column names for ordering rows. The ordering behavior is the same as the _.sortBy function in lodash.

Any step can be given a data property to make a table available in that step and its sub-steps. Here’s an example application:

steps:
  1:
    data: {source: data1}
    command: pipetter.pipette
    sources: $$source
    destinationPlate: plate1
    destinations: $$well
    volumes: $$volume

The data property activates our data table data1. The command pipetter.pipette can now access the data columns by using the $$-prefix along with the column name. So the above example is essentially equivalent to this:

steps:
  1:
    command: pipetter.pipette
    sources: [liquid1, liquid2, liquid1, liquid2]
    destinationPlate: plate1
    destinations: [A01, B01, A02, B02]
    volumes: [10 ul, 10 ul, 20 ul, 20 ul]

6.3.3 data.* commands

Roboliq’s data.* commands provide two commands for more sophisticated handling of data tables: data.forEachRow and data.forEachGroup.

data.forEachRow lets you run a series of steps on each row of the data table. For each row, the command activates a new data table containing only that row, and it runs its sub-steps using that new table. Here’s a toy example:

steps:
  1:
    data: {source: data1}
    command: data.forEachRow
    steps:
      1:
        command: pipetter.pipette
        sources: $source
        destinationPlate: plate1
        destinations: $well
        volumes: $volume
      2:
        command: fluorescenceReader.measurePlate
        object: plate1
        output:
          joinKey: well

In this case, the sub-steps will be repeated 4 times, once for each row. That mean each well will be dispensed into and measured before moving onto the next well. Notice that here only a single $-prefix was used rather than the double $$-prefix for the column variables. When a data table is activated, Roboliq will check if any of the columns have all the same value; if so, that property and value will be automatically added to the current scope (see the next section about Scope). Scope variables are accessible via the $-prefix. Since the data.forEachRow command activates each row individually, all of its columns will be added to the scope.

data.forEachGroup lets you operate on groups of rows at a time. You provide a groupBy property for it to group by, and then for each group it activates a new data table with those rows and runs its sub-steps using that new table. Here’s another toy example:

steps:
  1:
    data: {source: data1}
    command: data.forEachGroup
    groupBy: source
    steps:
      1:
        command: pipetter.pipette
        sources: $source
        destinationPlate: plate1
        destinations: $$well
        volumes: $$volume
      2:
        command: fluorescenceReader.measurePlate
        object: plate1
        output:
          joinKey: well

Since there are two unique source values in data1, the data.forEachGroup command will create two new data tables for it sub-steps:

First table:

- {well: A01, volume: 10 ul, source: liquid1}
- {well: A02, volume: 20 ul, source: liquid1}

Second table:

- {well: B01, volume: 10 ul, source: liquid2}
- {well: B02, volume: 20 ul, source: liquid2}

So the sub-steps will be repeated twice, once for new data table. Notice that here we used the single $-prefix for source, but the double $$-prefix for well and volume. Because the values of the well and volume columns are not the same in all rows, they are not automatically added to the scape, and we can’t use the single $-prefix.

6.3.4 data() directive

The data() directive lets you assign a modified version of your data table to a property value. It will be easier to explain this in the section on Substitution, so we’ll postpone the discussion till the end of this chapter.

6.4 Scope

Scope is the set of currently active variables in a step. These usually come from one of two sources: 1) the data directive and commands, as discussed above, or 2) a loop command like system.repeat, which lets you add an index variable to the scope of the sub-steps.

The scope is a kind of stacked-tower structure. When a step pushes variables into scope, they are available to that step’s command and all substeps; however, they are not available to sibling or parent steps.

6.5 Substitution

Substitution lets you work with parameters and data tables by inserting their values into the protocol. Roboliq supports three forms: template substitution, scope substitution, and directive substitution.

6.5.1 Scope $ substitution

In scope substition, an expression starting with $ is replaced with a value from the scope. There are various forms of replacement, which we’ll dive into now.

$#...: pre-scope substitution for parameter values

Parameters are not actually part of the scope, and they are accessible outside of steps as well. This means that they can be used in other parameter values and in object definitions, which is not the case for normal scope variables. You can substitute the value of a parameter named MYPARAM by with $#MYPARAM.

${...}: javascript expression

Roboliq will substitute in the result of a JavaScript expression. The JavaScript expression has access to:

• The scope variables
• _: the lodash module and its many functions.
• math: the mathjs module and its many functions.

Note that any value JSON value may be returned, whether it’s a string, number, boolean, array, or object.

$(...): mathjs calculation

The mathjs module provides a fairly broad range of math operations and is able to handle of units, such as volume. The mathjs expression has access to the current scope variables.

$...: scope value substitution

Here you just name the scope variable, and Roboliq will substitute in its value.

If you have activated a data table using the data property, then you can use $colName to get an array of all the values in the column named colName. Furthermore, if any of the data columns are filled with the same value, then that value is added to the scope as $colName_ONE, where colName is the actual name of the column. For example, if the active data table has a column named plate whose entries are all plate1, then $plate1_ONE = "plate1".

NOTE: Scope substitution can only be used as a parameter value, but not as a parameter name or part of a longer string. The following uses are invalid:

• text: "Hello, $name": Roboliq only supports scope substitution for an entire value, so the name value will not be substituted into this text.
  You can use template substitution for this purpose instead.
• $myparam: 4: Roboliq does not support scope substitution for property names. You can use template substitution for this purpose instead.

Examples

Let’s look at examples of $-substitutions. Consider this protocol:

roboliq: v1
parameters:
  TEXT: { value: "Hello, World" }
objects:
  data1:
    type: Data
    value:
      - {a: 1, b: 1}
      - {a: 1, b: 2}
steps:
  1:
    data: data1
    command: system.echo
    value:
      javascript: "${`${TEXT} ${a} ${__step.command}`}"
      math: "$(a * 10)"
      scopeParameter: $TEXT
      scopeColumn: $b
      scopeOne: $a_ONE
      scopeData: $__data[0].b
      scopeObjects: $__objects.data1.type
      scopeParameters: $__parameters.TEXT.value
      scopeStep: $__step.command

The system.echo command will output the object described in its value parameter. The resulting value is this:

javascript: "Hello, World 1 system.echo"
math: 10
scopeParameter: "Hello, World"
scopeColumn: [1, 2]
scopeOne: 1
scopeData: 1
scopeObjects: "Data"
scopeParameters: "Hello, World"
scopeStep: "system.echo"

6.5.2 Template ` substitution

Template substitution uses the Handlebars template engine to manipulate text. Template substitution occurs on strings that start and end with a tick (`). Here’s a simple example that produces a new string:

text: "`Hello, {{name}}`"

If the name in the current scope is “John”, then this will set text: "Hello, John".

If the template substitution result is enclosed by braces or brackets, Roboliq will attempt to parse it as a JSON object. Here’s a trivial example that turns a template substitution into a command, assuming that name is currently in scope:

1: `{command: "system._echo", text: "Hello, {{name}}"}`

6.5.3 Directive () substitution

Directives are substitution functions. The main one is the data() directive, which was briefly mentioned above in the Data section. The other directives are also closely related to Data objects, and they are discussed more in the chapter on Design Tables.

The data() directive lets you assign a modified version of your data table to a property value. The directive can take several properties:

• where: same as for the data property, this lets you select a subset of rows in the active data table.
• map: each row in the data table will be mapped to this value. This is how you can transform your rows.
• summarize: like map, but for summarizing all the rows into a single row. Summarize has a particularity: all column names are pushed into the current scope as arrays, and they are not overwritten by a single common value even if the column only contains a single value.
• join: a string separator that will be used to join all elements of the array (see Array#join in some JavaScript documentation).
• head: if set to true,

Let’s consider some examples using the data1 table from above. Here is the table:

- {well: A01, volume: 10 ul, source: liquid1}
- {well: B01, volume: 10 ul, source: liquid2}
- {well: A02, volume: 20 ul, source: liquid1}
- {well: B02, volume: 20 ul, source: liquid2}

and here are the examples:

Directive:
data(): {where: 'source == "liquid1"'}}
Result:

- {well: "A01", volume: "10 ul", source: "liquid1"}
- {well: "A02", volume: "20 ul", source: "liquid1"}

Directive:
data(): {map: '$volume'}
Result:

["10 ul", "10 ul", "20 ul", "20 ul"]

Directive:
data(): {where: 'source == "liquid1"', map: '$(volume * 2)'}
Result:

["20 ul", "40 ul"]

Directive:
data(): {map: {well: "$well"}}
Result:

- {well: "A01"}
- {well: "B01"}
- {well: "A02"}
- {well: "B02"}

Directive:
data(): {map: "$well", join: ","}
Result:

"A01,B01,A02,B02"

Directive:
data(): {summarize: {totalVolume: '$(sum(volume))'}}
Result:

- {totalVolume: "60 ul"}

Directive:
data(): {groupBy: "source", summarize: {source: '${source[0]}', totalVolume: '$(sum(volume))'}}
Result:

- {source: "liquid1", totalVolume: "30 ul"}
- {source: "liquid2", totalVolume: "30 ul"}