Engineering a multi-tenant analytics platform

Teams that need analytics usually assemble the same stack by hand: a script that pulls a report, a spreadsheet that cleans it, a BI tool that renders it, and a cron job that glues the three together. That stack has no ownership boundary, no audit trail and no way to answer the question a stakeholder actually asks — why does this number look wrong?

The product hypothesis was that a single declarative pipeline definition could replace the script, the spreadsheet and the scheduling glue, while keeping the audit trail that made the numbers trustworthy.

• Ingestion is manual and untracked, so nobody knows which numbers are current.
• Cleaning logic lives in a spreadsheet, so it cannot be reviewed or reused.
• Failures are silent, so a broken feed is discovered by a customer rather than a monitor.
• There is no isolation boundary, which makes a shared deployment impossible to sell.

• One deployment must serve many independent organizations without cross-tenant leakage.
• Data volumes are unbounded and unknown; the platform must stay responsive as inputs grow.
• Non-technical analysts must be able to configure transformations without writing code.
• Analytics is the primary service, so chart readability and speed are product requirements, not polish.
• The operational surface must be small enough for a lean team to run on a single orchestrated stack.

The system is a four-layer pipeline with a strict direction of dependency. Each layer has one responsibility and is replaceable without touching the others.

  Presentation        Next.js 15 · React 19
  ─────────────        │  server components, cookie session, chart engine
                       ▼
  Interface layer      REST (DRF) · WebSocket (Channels)
  ─────────────        │  serialization, validation, throttling, tenancy
                       ▼
  Domain layer         workspaces · sources · pipelines · executions
  ─────────────        │  declarative transformation vocabulary
                       ▼
  Execution layer      Celery workers · pandas · PyArrow
  ─────────────        │  extract → transform → materialize Parquet
                       ▼
  Persistence          PostgreSQL · Redis · object storage

Reads and writes never cross a layer downwards in the same request: the interface layer persists a definition, the execution layer processes it out of band, and the presentation layer only ever reads the materialized result.

Five objects describe the whole product. Every one of them belongs to a workspace, and every API path resolves the workspace before it touches data.

workspace ──┬── data source ──┬── pipeline ──┬── execution ──► parquet
            │                 │              └── log
            │                 └── transformations[]  (JSON, ordered)
            └── member (role)                        └── chart ──► dashboard

Six decisions account for most of the system’s behaviour. Each one was chosen against a concrete alternative.

Analytics is the service, so the chart layer is engineered rather than decorated. Charts are described as data — type, dimension, measure, aggregation, layout width — which makes them portable between the dashboard grid and the report builder.

• A single theme module owns the palette, axis style, grid style, tooltip style and default height, so Recharts (SVG) and ApexCharts (canvas) render identically.
• Every chart shares one frame handling loading, empty and populated states, which removed four divergent implementations.
• Categorical series are aggregated beyond eight slices into an Other bucket, keeping pie and donut charts readable without dropping data silently.
• Axes format large values with K/M/B/T suffixes and axis ticks stay on a fixed width so density is predictable across breakpoints.
• Charts read the latest successful execution, so a failed run degrades to the previous good dataset instead of an empty chart.
• Dashboards are a twelve-column grid with fractional widths; tiles reflow to full width on tablets and phones instead of scrolling horizontally.

• Workspace context is resolved once per request and validated against ownership or membership before any queryset runs.
• Object-level access is re-checked on execution and dataset reads, so a valid workspace cannot be used to reach another workspace's objects.
• Access and refresh tokens live in HttpOnly, SameSite cookies; the short-lived access token is refreshed transparently behind an interceptor.
• Authentication endpoints are rate limited, and production enforces secure cookies, HSTS, no-sniff, referrer policy and frame denial.
• Refresh tokens rotate and are blacklisted, so signing out or resetting a password revokes every outstanding token for the account.
• Password recovery uses a single-use, time-boxed token, and the request step answers identically for known and unknown addresses so it cannot be used to enumerate accounts.
• Secrets and host allow-lists are injected from the environment, and the process refuses to start in production without a secret key.

• Materialized Parquet is read in streamed batches, so windowed reads cost the same whether the dataset holds thousands or millions of rows.
• Transformations use nullable pandas dtypes, which keeps type coercion predictable and avoids silent object-column fallbacks.
• Derived chart series and summary aggregations are memoised, so resizing or re-filtering a dashboard does not recompute data.
• A pipeline cannot queue a second run while one is pending or running, which prevents duplicate work from repeated clicks.

The runtime is four containers: the ASGI application, a Celery worker, Redis for queue and channel layers, and the database. Both the application and the worker consume the same image.

daphne  ──►  ASGI application (HTTP + WebSocket)
celery  ──►  4 concurrent workers, graceful 60s shutdown
redis   ──►  broker, result backend, channel layer
prom    ──►  /metrics scraped by Prometheus, visualised in Grafana

• Health and throughput metrics are exported on the Prometheus endpoint and consumed by the bundled Grafana dashboard.
• Workers shut down gracefully, so an in-flight pipeline finishes instead of leaving a dataset half-written.
• Long-running runs can be cancelled; the worker is revoked and the execution is closed as cancelled with a log entry.
• Database, cache and object storage are all environment-selectable, so the same image runs locally on SQLite and in production on PostgreSQL.

Failure is where integrations usually lose trust, so the contract is explicit rather than incidental. Every failure the API can emit - validation, authentication, permission, throttling, not found, or an unexpected fault - is rendered by one exception handler into a single envelope, and URL-level failures produce the same shape.

{
  "error": {
    "code": "validation_error",
    "message": "Enter a valid email address.",
    "fields": { "email": ["Enter a valid email address."] },
    "request_id": "b1f0c4d5e6a7483192c3d4e5f6071829"
  }
}

• Codes are stable identifiers the client branches on; the prose is written for the person reading it and is safe to show verbatim.
• An X-Request-ID correlates the response with the log line, and the id appears in the body, the header and every structured log record.
• A dependency that is down is classified as an availability event and answered with 503 and retry_after rather than an opaque 500, so clients back off instead of failing permanently.
• The client never renders an exception. It renders a recovery panel with a retry action and the request id, and holds the session when the server never decided the session was invalid.
• The whole product - workspace, pipeline engine, chart dashboard and the platform control centre - is explorable without an account through a runtime demo session that swaps the HTTP transport in the browser.

• One definition replaces a script, a spreadsheet and a scheduler, and it is versioned, reviewable and re-runnable.
• Every number on a dashboard traces back to an execution record with a log, a row count and a file.
• A failed run is a first-class state with a retry path, not a silent gap in the data.
• The workspace boundary is enforced in one place, so the same deployment can serve many organizations.
• Analytics surfaces share one visual language, which makes a new chart type a configuration change rather than a design project.

• Incremental loads: watermark columns so a run only processes new rows.
• Direct database connectors alongside file and REST ingestion.
• Scheduled report delivery to email and Slack.
• A chart-level permission model so dashboards can be shared outside their workspace.