Run & Monitor Executions

Trigger runs, watch live logs and inspect history.

Every run is recorded with its inputs, timings, output size and a full log. That record is your audit trail when a number looks wrong.

Triggering a run

Open a pipeline and choose Run now. The run is queued and executed by the next available worker; the page follows it automatically.

POST/api/pipeline/pipelines/{id}/execute/Auth required

Queues an execution and returns its identifier and status.

Watching the live log

Running a pipeline opens the live view: a progress bar, the run number, and the log as the worker writes it. Log lines are pushed over a WebSocket channel while it is open, so nothing needs refreshing, and the console colours failures, completions and system notices separately.

Use Cancel run to stop a queued or running execution from inside the viewer. Close in background dismisses the window without touching the run - it keeps going, and the execution history records the outcome.

Connection states

The badge beside the pipeline name tells you how the view is being kept up to date. It changes on its own; nothing needs to be reopened.

Field	Type	Description
Live	string	Streaming from the pipeline engine. The view updates the moment a step emits a line.
Reconnecting	string	The connection dropped and is being re-established with a short, increasing delay. Nothing is lost - the run continues on the worker.
Polling	string	No socket could be established, so progress is being read from the API instead. You still see status, progress and the full log; updates arrive every few seconds.
Offline	string	The view has stopped updating, usually because the session ended or the pipeline was deleted. Reopen the execution from the history table to try again.

Execution history

Field	Type	Description
status	string	pending, running, success, failed or cancelled.
started_at	string	When the worker picked up the run.
finished_at	string	When it completed or aborted.
duration_seconds	string	Wall-clock runtime.
total_records	string	Rows written to the dataset.
success_rate	string	Share of rows processed without error.
file_size	string	Size of the materialized Parquet output.

Reading the metrics

Record count dropping sharply usually means an upstream format change — not a small data change.
Duration climbing run over run points to growing input volume or an expensive sort.
Success rate below 100% means some rows were rejected. Inspect the log for the offending values.

Handling failures

1
Open the log
Read from the top and identify the first failing step.
2
Reproduce with a small source
Fix the transformation on a small file first, then apply it to the real pipeline.
3
Re-run
Running again produces a new execution. The previous failed record is preserved.
4
Confirm the dataset
Check that record count and success rate returned to their usual range.

Last reviewed October 2026.