Pipeline overview

The Data Pipeline is the set of steps that take your raw data and shape it into the results your Visualisation uses. Think of it as an assembly line: data comes in at one end, each step does a little work, and finished results come out the other end.

Nodes and connections

Each step in the pipeline is a node. You connect nodes so that one feeds the next, and the canvas draws those connections as lines, so you can see the flow of data at a glance. A node early in the flow brings data in; a later node takes that data and transforms it.

You build the pipeline in the editor, on the Data Pipeline tab, by adding nodes and drawing connections between them. A pipeline has three stages: inputs - a dataset and an indicator bridge - feed a consumer, a SQL transform (the only transform type for now), whose result is the output. That output is surfaced on the separate Visualisation tab as a table, chart, pivot, or SQL editor.

The four node types

There are four kinds of node, each with its own page.

• Datasets bring data in. A dataset is an input table you fill with your own data, usually by importing a file.
• SQL nodes transform data. They take the output of earlier nodes and reshape it.
• Indicator data nodes pull metrics from one of your indicator apps so you can combine those metrics with other data.
• Analytic app data nodes import a node's result from another of your analytic apps, so you can build on a shared, already-published result without copying it.

Run the pipeline and read the results

Adding and connecting nodes describes what should happen, but nothing is processed until you run it. Click Run to process every step in order. The run happens in the background, and the status updates on its own when it finishes, so you do not need to wait or refresh.

When the run finishes, click any node to view the data it produced. Each node shows whether it succeeded, and your Visualisation widgets show the fresh results. If a step fails, you can open it to read the error, fix it, and run again.

Run history and what is kept

Every run is listed in the app's history, under History (the clock button in the editor bar). Click View full history for the complete list, with the start time, status, duration, number of nodes, and the size of what the run produced. Click a row to see the pipeline exactly as that run left it.

The list keeps every run. The data a run produced does not stay forever, because it is the biggest thing an app stores. Ovidens keeps the results of the newest few runs (three, unless your workspace is set differently), always keeps the latest successful run, and quietly frees the rest. The older run still shows in the history, it just has no data to open any more.

Administrators can override that on any individual run:

• Protect this run's data pins a run so the clean-up never touches it. A protected run is marked Protected in the list. Stop protecting this run's data releases it again, and says plainly what that means: the data becomes "eligible for automatic deletion by the next retention sweep".
• Delete this run frees that run's data immediately and removes it from the history. It cannot be undone, and it is not offered for the latest successful run.

Your workspace's System page lists both limits under Limits: "Run data retained per app" and "Protected runs per app". See Administration.

When results are out of date

If you change data after a run, Ovidens says so instead of quietly showing stale numbers. In the editor you get a banner: "The data changed since the last run. Re-run to refresh the results.", with a Re-run button. Individual widgets carry a small Out of date badge with the same message. People using the published app see the equivalent note: someone who builds the app, such as a workspace administrator, needs to run it again and publish it.

Let Vera build it for you

You do not have to assemble a pipeline by hand. Describe the data and the result you want, and Vera can build the whole pipeline for you. See Build with Vera.