Typical workflow structure

The number of nodes in the standard library can be quite daunting for a new user, so let’s go through a few common use cases to get acquainted with some of the types of nodes that you can find.

Importing data

To import data from a file or a database you first need to add a Datasource or Datasources node (Library/Sympathy/Datasources in the library view) to the workflow. Configure the Datasource(s) node to point to where your data is located. Connect it to a type node like Table(s) (Library/Sympathy/Data/Table in the library view) or ADAF(s) (Library/Sympathy/Data/ADAF in the library view) and you should be good to go. The type nodes can often automatically detect the file format and read the file without any additional configuration, but sometimes you need to open the configuration GUI, manually choose the file format, and complete some configuration specific for that file format.

Prepare data

Typically, data needs to be prepared by removing invalid values and unwanted noise from the data before it is analyzed. This may also include removing irrelevant columns to save execution time and storage space.

When working with Tables two basic nodes useful for washing data are: Select rows in Table and Select columns in Table. Their function is fairly self explanatory.

Analyze data

There are three different approaches to analyzing data in Sympathy. The fastest and easiest is to use the Calculator node. The Calculator node supports small computations and is far from feature-complete at this stage. It only operates on Table data.

The second approach is to use the function selector f(x) nodes. The function selector supports both Table (e.g. F(x) Table) and ADAF (e.g. F(x) ADAF) data. The f(x) nodes are typically used to define functions that will be used in many different workflows.

The third approach is to write a full Sympathy node. This requires more work but is necessary to implement custom behaviour beyond that which is possible in the f(x) or Calculator nodes. Refer to the Node writing tutorial for information about how to write full Sympathy nodes.

Export data as plots or reports

Exporting is useful to store intermediate or final results from a workflow.

The output from any function node can easily be exported by connecting an export node, such as, for example, Export Tables - when dealing with table data, and Export ADAFs for ADAF data. Notice that the exporter names are in plural, which means that they work on list type input. To export table data using Export Tables, the Table to Tables node can be used to produce the desired table list type. The export nodes are different from the import nodes in that they do not use an external data source, instead, the output location is set in the nodes configuration. Export nodes exist for many of the same file formats as the import nodes, making it possible to do import, analysis and then export back to the original input source.

For visualization, a few different nodes are available for plotting and reporting. The most powerful set of plotting and reporting nodes are in the reporting library.

Working with ADAF

Many of the nodes in the standard library are only available for Table data. If your data is more naturally represented as ADAF you can still use those nodes by letting them work on the tables that make up the ADAF. For instance if I have imported some data as an ADAF, but I want to remove some of the time series from one of the rasters. The node ADAF to Table let’s me get the relevant raster as a table and I can then use the node Select columns in Table to remove some of the columns. As a last step I can use the node Update ADAF with Table to place the modified Table back into the ADAF.

Working with ADAF

Example of working with ADAF. This workflow can be found in <sympathy install directory>/Sympathy/Doc/workflows/ADAF example.syx.

Control structures

Perhaps you have noticed that some common programming control structures are missing in Sympathy. Things like loops and if-statements are instead implemented in a more data-centric way.

Conditional execution

There is currently no way to branch a flow and only execute a single branch. Instead you can use filters and selectors to guide the data into different branches.

Looping

There is also no explicit way to loop in Sympathy. What you can do though is to use Lists. Most list nodes implicitly loop over all the incoming data. For example Select columns in Tables will loop over all the tables in the input and do the selection for each of them.