Connect a Data Source
Upload CSV or JSON files, or pull from a REST endpoint.
Data sources are the entry point of every workflow. A source describes where raw data lives; it does not transform anything by itself.
Before you start
- You need the
adminoreditorrole. - Files must have a header row with unique column names.
- API sources must return JSON and be reachable from the worker.
Add a data source
- 1
Open the Data Sources page
Choose Data Sources from the sidebar, then Add Data Source. - 2
Name it
Pick a stable, lowercase name — teammates and pipelines reference it by name. - 3
Select the type
CSV, JSON or API endpoint. The form adapts to the choice. - 4
Provide the payload
Upload the file, or enter the endpoint URL and optional bearer token. - 5
Review the parsed sample
The platform shows the detected columns and a preview of the first rows. Confirm the types look right. - 6
Save
The source appears in the table with a status of active.
Choosing a source type
Comparison
| Field | Type | Description |
|---|---|---|
| CSV | string | Best for exports from other systems. Fastest to process and the most predictable to parse. |
| JSON | string | Best for API dumps. Nested objects are flattened using dotted keys. |
| API | string | Best for live data. The endpoint is called on every pipeline run, so the dataset stays current. |
Validating the sample
The preview is generated by the extraction endpoint and is not stored as a dataset. Use it to answer three questions:
- Are all expected columns present?
- Did numeric columns arrive as numbers rather than quoted strings?
- Are dates in a single, consistent format?
/api/datasource/datasources/extract-sample/Auth requiredUploads a file, parses it and returns the detected columns plus a sample of rows. Nothing is persisted.
Managing sources
- Search filters by name and type as you type.
- Filters narrow the table by source type or status.
- Edit opens the row menu and reopens the source, pre-filled, so a name, status or endpoint can be corrected without re-uploading anything.
- Delete removes the source and its stored file after a confirmation. Pipelines that read from it lose their input, so repoint them first.
Editing an existing source
Choose Edit source from the row menu. The form opens with the stored values and saves through a partial update, which means only the fields you touch are sent: renaming a CSV source keeps its uploaded file, and leaving the file picker empty is not an error.
Changing the type is the exception. A source that becomes an API source still has to satisfy the API requirement, so the endpoint URL is asked for at that point, and a source that becomes a file source asks for the file.
| Field | Type | Description |
|---|---|---|
| File | string | Leave empty to keep the stored file. Pick a new one to replace it; the sample preview is refreshed at the same time. |
| Endpoint URL | string | Required for API sources and validated as a URL. Save is refused, with the field highlighted, if it is missing or malformed. |
| Auth token | string | Optional bearer token sent with each request. Leave it untouched to keep the stored value. |
| Status | string | Set a source to inactive to keep its definition and stop new runs from reading it. |
Common problems
| Field | Type | Description |
|---|---|---|
| Empty upload | string | The file has no header row or uses a delimiter other than a comma. Re-export as standard CSV. |
| Duplicate column names | string | Two columns share a name, so the second overwrites the first. Rename one in the source file. |
| API returns 401 | string | The token is missing, expired or lacks the required scope. Re-issue it and update the source. |
| Numbers arrive as text | string | Thousands separators or currency symbols force a string type. Strip them, or add a convert_type transformation. |
Last reviewed October 2026.