What’s new

News in 1.3.5

Node/plugin changes:

  • Calculations in Calculator Table can be deselected for output enabling better support for intermediary calculations. This also enables intermediary calculations to have different lengths from output columns.
  • The input table(s) in Calculator Table can be easily copied over to the output table(s) with the new Copy Input parameter. Calculations with the same column name override columns from the input table(s).
  • MATLAB nodes and Matlab Calculator have gotten better cross-platform compatibility.
  • Matlab Calculator has been updated with the same GUI and (almost) the same functionality as Calculator Tables.
  • Matlab Table and Matlab Tables have gotten a new simplified format. See the documentation for details on how to use that. This format can also be imported and exported in Table and Export Tables respectively. A Table-like API is planned for a future release. The API that currently resides in Sympathy/Matlab will also be deprecated in a later release, in favor of the new format. The old nodes are left for compatibility, so current flows and scripts will still work.
  • The generic Empty node allows to specify the data type of the output port. The previous, specific, Empty-nodes have been deprecated.
  • Rename column nodes have more consistent priority rules when more that one column are renamed to the same name.
  • Extract lambda nodes are more robust with regard to corrupt flows. One corrupt flow should no longer stop the nodes from extracting other lambdas.
  • New node: Heatmap calculation useful for feeding the heatmap in Figure(s) from Table(s).
  • New features for heatmaps in Figure(s) from Table(s): logarithmic color scales and Z labels.
  • Datasource and other nodes where you specify a file path can specify paths relative to its own workflow or the top workflow. This can make a difference when working with linked subflows.
  • Datasources GUI is no longer slowed down when searching large folder structures. If the search takes to long it is aborted, and to get the full results the node has to be executed.
  • The table name used for the output in HJoin Table can now be selected.
  • Fixes to extract flows as lambdas so that workflow environment variables and flow name are set correctly.
  • TimeSync ADAFs can now use integer timebases and correctly displays datetimes in the plot.
  • Assert Equal Table now treats NaNs as equal.
  • Improved config gui and handling of NaN values, masked values and non-ascii binary data in VSplit Table(s).
  • A new node has been added HJoin ADAFs pairwise.
  • When zooming and panning in Plot Table and using datetime as X axis, the current time span in the plot is displayed.
  • SQL importer plugin can use SQLAlchemy and provide betters autodetection of existing tables.
  • SQL exporter plugin can use SQLAlchemy.
  • Improved documentation generation with support for libraries on different drives or on unicode paths.

Platform:

  • Nodes have gotten dynamic port icons that display the actual types.
  • Color of textfields can now be changed.
  • A textfields can be moved by dragging on any part of it. It is now edited by double clicking it or by right clicking and choosing “Edit”.
  • The table viewer and any viewer which uses that component (i.e. ADAF viewer) can now be transposed for better viewing of long column names and tables with few rows but many columns.
  • Table viewer now supports copying values and/or column names as a table or as text.
  • The viewer can now show histograms for more types of data.
  • Allow maximizing subflow configurations.
  • Linked flows can now be placed on a different drive than their parent flows.
  • combo_editor for string parameters can now have an empty list of options.
  • Invalid subflows are more reliably shown as invalid (gray). Now any subflow which looks executable should be executable.
  • Subflows show an error indicator if they contain any nodes that are not found in the node library. This should make such nodes much easier to find.
  • Better feedback when trying to open a non-existing or corrupt workflow.
  • The platform can handle a larger number of linked files without running into the OS limit.
  • An Advanced tab has been added to Preferences, with one option to limit the number of concurrent nodes that may be executed, and one option to display warnings about deprecated nodes.
  • New preference option to set number of concurrent worker processes. This may help with performance for heavily branched flows.
  • Python 3 support for files created with the node and function wizards.
  • Library wizard can create subdirectories.
  • Spyder can’t handle files on file paths contaning non-ascii characters, and will fail to start when trying to debug nodes. An error message is now displayed to notify the user of this.
  • Improved stability of type inference.
  • File datasources always store absolute paths.
  • Database datasources can use SQLalchemy in addition to ODBC.

News in 1.3.4

Sympathy for Data version 1.3.4 offers improvements to existing nodes, including several new plot types for the figure nodes and overall polish.

Node/plugin changes:

  • Figure nodes have been massively improved with several new plot types (scatter/bar plots/histograms/heatmaps), improved gui, etc
  • Extended figure export node with plugin exporter structure as for other types and choice of specifying image size in mm and dpi
  • Reporting Nodes have been improved with rulers in layout window, pdf exporting and auto creation of tree structures
  • Calculator, allows accessing the input table directly under the name “table” allowing for a way to test if a column exists. The node was also extended with the json module in the execution context
  • ca.changed now correctly returns empty array for empty input
  • Added functions ca.global_min and ca.global_max to standard calculator plugin. These handle empty input as you would expect
  • Interpolate ADAF nodes have improved handling of missing values and resampling of zero-length signals
  • Datasource and exporter/importer of SQL can use SQLAlchemy
  • Pad List input can be different types of lists
  • Predicate nodes have new editors for writing code
  • VJoin nodes can mask missing values
  • MDF importer creates MDF_datetime metadata
  • Assert Equal Table allows approximate comparison of floats
  • Added documentation for internal nodes (Apply, Map, etc.)

APIs:

  • Made it possible to specify viewer and icon for custom types (TypeAlias). For details, see Creating a custom data type
  • Only scanning Libraries for plugins, PYTHONPATH is no longer included
  • Scalar parameters can use the new combobox editor. See All parameters example for an example
  • Code parameter editor for string parameters. See Editors for details and All parameters example for an example
  • Allow Controllers to trigger on user-specified value. For an example of this see Controller example
  • Implemented cols() and added documentation for col/cols and Column class. See Table API
  • Added attrs property to Table API
  • Expose dtypes module in sympathy.api

UI:

  • Improved look and feel of wizards
  • Library wizard has new examples
  • Node wizard can select tags
  • Show filename in flow tab unless flow label has been explicitly set by user. This means that a flow created in 1.3.4 will have no flow label when opened in older versions.

Platform:

  • More robust checks of port types
  • Masked arrays

Deprecated nodes:

  • Raw Tables nodes
  • Scatter 3D ADAF

News in 1.3.3

Sympathy for Data version 1.3.3 offers improvements to existing nodes, the table viewer and automatic parameter validation when configuring nodes.

GUI:

  • Behaviour change of “?” wildcard in Table viewer search bar to match single character only
  • General improvements of Table viewer GUI
  • General improvements of parameter validation

New nodes/plugins:

Changes in nodes/plugins:

  • Allow unicode characters in Calculator node
  • Improved default behaviour of Calculator node
  • Improved rescaling of preview plot in Filter ADAFs node
  • Improved XLSX export output compatibility
  • Extract Lambdas can be configured when connected
  • Improved performance of VSplit Table
  • Improved bounds checking for calculator functions shift_seq_start and shift_seq_end
  • Improve gui in Manually Create Table. Now allows removing selected rows/columns as well as changing name and datatype of existing columns
  • Improved handling of bad timebases in interpolation nodes

APIs:

  • Added value_changed propagation to parameters
  • Made verify_parameters validate every change to configured parameters, for nodes with generated configurations

Miscellaneous:

  • Fixed update method for tuple type
  • Data viewer can once again be run stand alone
  • Updated icons

News in 1.3.2

Sympathy for Data version 1.3.2 offers several new and prominent features, such as the ability to specify libraries used by workflows, new window handling which brings open, but minimized, configurations and viewers into focus, a reworked save dialog that properly detects changes in subflows and many improvements to existing nodes.

GUI

  • Raise open Configuration/Settings/Viewer windows on consecutive clicks
  • Improved save confirmation for workflows
  • Improvements to the function wizard. Including updating it to work with the new generic F(x) nodes

New features

  • Flows can now specify libraries and python paths in the Info dialog. These are added to the global library/python paths when loading the flow
  • New error message box for node dialogs for showing validation errors/messages in node configurations
  • Support for storing masked arrays, but not every node can handle them correctly

New nodes/plugins

Changes in nodes/plugins

  • Renamed Plot to Figure for nodes using the Figure type
  • Figure Compressor, Layout Figures in Subplots: added auto recolor and auto rescale
  • Improved datetime handling in Figure nodes
  • MDF exporter plugin: encode unicode columns instead of ignoring them
  • Convert columns in Table(s): converts string dates to either UTC or Naive datetimes. Choosing UTC, localized times will be converted to UTC. Choosing naive, the time zone info in the input is simply ignored. Old nodes will automatically use UTC
  • Improved performance of Select rows in Table(s)
  • Select rows nodes better handles values without explicit type annotation
  • Improved error handling in lookup nodes
  • Calculator plugin: Make sure that result is always correct length in changed_up, changed_down, and shift_array
  • Filter ADAFs: added parameter validation and error messages. Filter design is computed and shown on parameter changes
  • Changed the visible name for importer and exporter plugins for ADAF and Table to SyData
  • Removes matlab settings from Matlab Table nodes and put them into global Preferences dialog
  • Renamed calculator nodes to Calculator Table(s)
  • CSV Exporter plugin: improved writing of datetime columns
  • Improve handling of missing units in interpolate nodes

APIs

  • Extended Table API and added Column object
  • Change default value for attribute 'unit' to always be empty string in ADAFs
  • Added ParameterView base class for generated and custom GUIs to API. Custom GUIs can override the methods and properties to customize the behavior. Inheriting from ParameterView will be required in the future versions

Miscellaneous

  • Added support for signing the Installer/Uninstaller
  • Extended searchbar functionality for the Table viewer
  • Always write generated files in the right directory
  • Fix overrides not saved in syx files
  • Non-linked subflows inherit their parents $SY_FLOW_FILEPATH and $SY_FLOW_DIR
  • Improve performance of type inference

News in 1.3.1

Sympathy for Data version 1.3.1 offers several new and prominent features such as an improved data viewer with embedded plot, a new figure datatype and many new nodes as well as improved performance and stability.

New features

  • Improved Data viewer with embedded plotting of signals.
  • Overhaul of subflow configuration: Split into settings and configuration. Removed grouping. Only allow selecting shallow nodes/flows. Added Wizard configuration mode. Optionally override parameters of linked subflows. Should be somewhat backwards compatible
  • Added Figure-type. Passes serialized matplotlib figures between nodes
  • Added tuple-type
  • Better handling of broken links/nodes missing from library and changed port types due to subflow changes
  • F(x) function wizard
  • Allow setting flow name, description, version, author, and copyright information in flow info dialog. Also improved handling of flow labels all around
  • Expose more environment variables from workflow
  • New command-line option: --nocapture for debugging

New nodes

Node changes

Exporters/Importers changes

  • ADAF Importer was extended with option to link to imported content
  • MDF Importer can handle zip-files that include a single MDF-file as input
  • Gzip Exporter binary writes files correctly
  • ATF Importer supports a wider range of files
  • Export tables nodes will now create output folders if necessary
  • Increased compression for exported sydata-files produces smaller files

Optimizations

API changes

  • Added MATLAB API for writing scripts executed by the Matlab node
  • Added update method to Attributes class. (ADAF API)
  • Added support for placeholder text in lineedit_editor in parameter helper
  • Added visibility and enable/disable slots to ParameterValueWidget

Bug fixes

  • Fixed name and type of output port of Report Apply nodes
  • Fixed a bug where save file dialog wouldn’t show up at all when trying to save subflow on Windows, if the subflow label contained some specific unallowed characters
  • Made sure that aborting a subflow doesn’t also abort nodes outside of the subflow
  • Fixed a bug where linked subflows were sometimes inserted with absolute path

Stability

  • Improved reliability when working with lambdas, maps and apply nodes

Deprecated nodes

Deprecated nodes don’t show up in the library view, but can still be used in workflows.

  • Type specific versions of list operation nodes (such as Get Item Table and Append ADAF).
  • Old FilterADAFs node

News in 1.3 series

Sympathy for Data version 1.3.0 offers several new and prominent features such as generic types, higher order functions and much improved support for linked subflows.

Many small improvements were made to the standard node library. Nodes will often cope better with empty input data and deliver informative, but less detailed, feedback.

Nodes from 1.2.x should be compatible with 1.3.0 but there are new, more succinct, ways of writing nodes for 1.3.x that are not backwards compatible with 1.2.x. When writing new nodes, consider which older versions of the platform that will be used.

New features

New nodes

User interface

  • Right-click on an empty part of the flow to insert higher order functions.
  • New command in context menu for inserting a subflow as a link.
  • Improved file dialogs in node configurations, by using native dialog when asking for an existing directory and starting file dialogs from currently selected file path.

API changes

  • Simpler APIs for writing nodes. See Node writing tutorial
  • New method in ADAF API: Group.number_of_rows
  • Configuration widgets can expose a method called save_parameters which is called before the gui is closed. See Custom GUIs
  • Added API (parameter helper): List parameter widgets emit valueChanged signal
  • Improved slicing of (sy)table with slice object with negative or undefined stride
  • Automatically update order, label, and description for parameters when the node’s definition changes
  • NodeContext is no longer a named tuple
  • Added new method: NodeContext.manage_input(). A managed input will have its lifetime decided outside of the node

Linked/locked subflows

  • Include subflows relative to path of parent flow, not relative to root flow. This affects where sympathy searches for linked subflows inside linked subflows and should hopefully feel more natural than the old system
  • Allow opening of flows with broken links
  • Import and export nodes can now be used inside locked subflows and lambdas
  • Made it impossible for flows below a locked flow to themselves be locked
  • Improved abort for locked subflows

Node changes

Other improvements

  • Added default workflow environment variables SY_FLOW_FILEPATH, SY_FLOW_DIR and SY_FLOW_AUTHOR. All flows have these and they can’t be set or deleted.
  • Subflows can define workflow variables. Each subflow specializes the variables of its parent flow, so that the parent flows vars are accessible in the subflow but not vice versa.
  • Improve performance by skipping validation of any nodes that don’t implement verify_parameters()
  • Improve performance by changing compression settings for sydata files, compression is faster but compresses slightly less
  • Pretty print workflow xml files, making diffs possible

New requirements:

  • Requiring pandas version 0.15 for the CSV import, for versions before 0.15 down to 0.13 it will still work but may behave slightly differently in edge cases with blank rows

News in 1.2 series

Sympathy for Data version 1.2 is a significant minor release for Sympathy for Data. It features several prominent new features, improved stability and more. It is however not redesigned and with only a few small modifications, all existing nodes and flows should work as well as in 1.1.

The bundled python installation has been upgraded with new versions of almost every package. Added to the packages is scikit-learn, used for machine learning. Our investigations suggest that the new package versions are reasonably compatible with old nodes and cause no significant differences for the standard library.

New features

  • Added support for using environment variables, and per installation/workflow variables. The variables which can have a default value are used in string fields of configuration widgets to enable parametrization. See Using environment variables.
  • Added support for profiling, with the ability to produce graphs if Graphviz is available. See Profiling nodes and workflows.
  • Added support for debugging single nodes with data available from Sympathy using spyder. See Debugging nodes.
  • Added new Node Wizard for generating new nodes. See The node wizard.
  • Added support for configuring subflows by aggregating selected node configurations. See Subflow configuration.
  • Improved support for plugins in third party libraries. It is no longer necessary to add the folder with the plugin to python path in preferences
  • Support for adding custom data types in third party libraries. See Creating a custom data type.
  • Significantly improved handling of unicode paths including the ability to install Sympathy and third party libraries in a path with unicode characters

Nodes and plugins

  • Added CarMaker type 2 ERG ADAF importer plugin called “CM-ERG”
  • Plugins can now export to non-ascii filenames
  • Fixed MDF export of boolean signals
  • Added generating nodes for empty Table, Tables ADAF and ADAFs.
  • Convert column nodes can convert to datetime
  • Calculator node can produce compact output for length matched output
  • Lookup nodes handles both event column and other columns with datetimes
  • Time Sync nodes “SynchronizeLSF” strategy should work as expected again. The Vjoin index option is now only used for the ”Sync parts” strategy

New command line options

See Sympathy Start options for more info.

  • Added new command line option, ‘–generate_documentation’ for generating documentation from CLI
  • Added ‘exit_after_exception’ argument which is activated by default in CLI. It makes Sympathy exit with error status if an unhandled exception occurs in a signal handler.
  • Added separate flag: –node_loglevel, for controlling the log output from nodes.
  • Made it possible to set the number of workers using –num_worker_processes n.

API changes

  • Libraries must now have only a single python package in their Common folders. See Node writing tutorial. In the Standard Library this package is called sylib
  • Removed has_parameter_view from node interface. See Custom GUIs.
  • Changed default unit for time series to empty string instead of 'unknown'.
  • Added has_column method in sytable and added corresponding method in table.File
  • Accessing an ADAF basis which does not exist will raise a KeyError
  • Improved node error handling, making it possible for nodes to issue user friendly error messages as well as warnings. See Errors and warnings.
  • Expanded and improved documentation, including API references for all default data types, and documentation on how to create your own data type
  • Improved error handling in many data type API functions

User interface

  • Improved selection and context menu handling
  • “Help” in node context menus will now also build documentation if necessary.
  • Allow connections to be made by dragging from an input to an output port
  • Added zoom with Ctrl/Cmd + scroll wheel
  • Added working stop button.
  • Improved the presentation of data in the viewer with a clearer font and better size handling as well as coloring of columns by data type
  • Improved undo/redo functionality, making more operations available in the undo history

Stability

  • Avoid hanging on Windows when too much output is produced during startup
  • Avoid infinite wait during node validation