# Pipeline overview (/docs/analytical-apps/pipeline/overview)



The **Data Pipeline** is the set of steps that take your raw data and shape it into
the results your Visualisation uses. Think of it as an assembly line: data comes in at
one end, each step does a little work, and finished results come out the other end.

## Nodes and connections [#nodes-and-connections]

Each step in the pipeline is a **node**. You connect nodes so that one feeds the
next, and the canvas draws those connections as lines, so you can see the flow of
data at a glance. A node early in the flow brings data in; a later node takes that
data and transforms it.

You build the pipeline in the editor, on the **Data Pipeline** tab, by adding nodes
and drawing connections between them. A pipeline has three stages: **inputs** - a dataset and an indicator
bridge - feed a **consumer**, a **SQL** transform (the only transform type for
now), whose result is the **output**. That output is surfaced on the separate
**Visualisation** tab as a table, chart, pivot, or SQL editor.

<PipelineDiagram />

## The four node types [#the-four-node-types]

There are four kinds of node, each with its own page.

* **[Datasets](./datasets.mdx)** bring data in. A dataset is an input table you
  fill with your own data, usually by importing a file.
* **[SQL nodes](./sql-nodes.mdx)** transform data. They take the output of earlier
  nodes and reshape it.
* **[Indicator data](./indicator-data.mdx)** nodes pull metrics from one of your
  indicator apps so you can combine those metrics with other data.
* **[Analytic app data](./analytic-data.mdx)** nodes import a node's result from
  another of your analytic apps, so you can build on a shared, already-published
  result without copying it.

## Run the pipeline and read the results [#run-the-pipeline-and-read-the-results]

Adding and connecting nodes describes what should happen, but nothing is processed
until you run it. Click **Run** to process every step in order. The run happens in
the background, and the status updates on its own when it finishes, so you do not
need to wait or refresh.

When the run finishes, click any node to view the data it produced. Each node shows
whether it succeeded, and your Visualisation widgets show the fresh results. If a step
fails, you can open it to read the error, fix it, and run again.

## Run history and what is kept [#run-history-and-what-is-kept]

Every run is listed in the app's history, under **History** (the clock button in
the editor bar). Click **View full history** for the complete list, with the start
time, status, duration, number of nodes, and the size of what the run produced.
Click a row to see the pipeline exactly as that run left it.

The *list* keeps every run. The *data* a run produced does not stay forever,
because it is the biggest thing an app stores. Ovidens keeps the results of the
newest few runs (three, unless your workspace is set differently), always keeps the
latest successful run, and quietly frees the rest. The older run still shows in the
history, it just has no data to open any more.

Administrators can override that on any individual run:

* **Protect this run's data** pins a run so the clean-up never touches it. A
  protected run is marked **Protected** in the list. **Stop protecting this run's
  data** releases it again, and says plainly what that means: the data becomes
  "eligible for automatic deletion by the next retention sweep".
* **Delete this run** frees that run's data immediately and removes it from the
  history. It cannot be undone, and it is not offered for the latest successful
  run.

Your workspace's **System** page lists both limits under **Limits**: "Run data
retained per app" and "Protected runs per app". See [Administration](../../administration.mdx).

## When results are out of date [#when-results-are-out-of-date]

If you change data after a run, Ovidens says so instead of quietly showing stale
numbers. In the editor you get a banner: "The data changed since the last run.
Re-run to refresh the results.", with a **Re-run** button. Individual widgets carry
a small **Out of date** badge with the same message. People using the published app
see the equivalent note: someone who builds the app, such as a workspace administrator,
needs to run it again and publish it.

## Let Vera build it for you [#let-vera-build-it-for-you]

You do not have to assemble a pipeline by hand. Describe the data and the result
you want, and **Vera** can build the whole pipeline for you. See
[Build with Vera](../build-with-vera.mdx).