Skip to main content
OpenDot is a modular monolith plus a separate realtime voice runtime. The platform console owns configuration and device binding; the runtime owns live audio sessions, OpenAI Realtime client-secret exchange, and device Realtime bridging; PostgreSQL stores durable control-plane state; firmware connects over OTA/bootstrap and WebSocket paths.
OpenDot system architecture diagram

OpenDot separates configuration, durable state, runtime sessions, hosted providers, and firmware connections.

​
Current system

The control boundary is intentional: browser and device clients do not push raw identity configuration directly into the runtime. They ask the platform API for an authorized session or credential, and the runtime verifies that token with the platform API before loading the identity config. For contributor orientation, see Contribution areas. For sequencing across current, next, and later work, see the Roadmap.

​
Platform stack

Docs

Mintlify in docs/ publishes product, setup, architecture, and contributor guidance.

Console

React, TypeScript, Vite, TanStack Query, and lucide-react power the browser workflow.

Platform API

Fastify handles auth, agents, pipelines, device inventory, deployments, and runtime/session tokens.

Database

PostgreSQL stores OpenDot-owned product state through Drizzle schema and SQL migrations.

Voice runtime

Node.js orchestrates /voice and /ws WebSocket sessions, the /realtime/client-secret boundary for browser Speech-to-speech tests, and the server-side Realtime bridge for Speech-to-speech Dot turns.

Providers

Deepgram backs the starter VAD, STT, and TTS paths. OpenAI-compatible Responses or Chat Completions calls back the starter LLM stage. OpenAI Realtime backs the Speech-to-speech browser and device paths.

Device firmware

ESP-IDF firmware targets the Waveshare ESP32-S3-AUDIO-Board reference device today.

Packaging

Docker Compose runs local services, while the Render Blueprint maps the same containers to hosted preview services.

​
Local development topology

OpenDot local development topology diagram

Local development can run as a Compose stack or as separate pnpm processes.

Docker Compose runs PostgreSQL, migrations, the platform API, the voice runtime, and the web console. The pnpm workflow runs the same application surfaces as separate local processes. The docs site is separate and previews from docs/ with mint dev. Use Runtime configuration when changing environment variables or service URLs.

​
Voice turn flow

OpenDot voice turn flow diagram

Sandwich turns move through VAD, STT, LLM, and TTS. Speech-to-speech uses browser WebRTC or the runtime device bridge.

The default Sandwich pipeline remains explicit even when a provider combines lower-level details:
Current media paths are:
  • Sandwich Browser Test: the platform mints a short-lived voice-session token, then the runtime verifies it before accepting /voice.
  • Dot device: activated devices use stored device credentials, then the runtime verifies them before accepting /ws. Sandwich agents use the Deepgram + LLM + TTS path; Speech-to-speech agents use a server-side OpenAI Realtime WebSocket bridge while firmware keeps sending and receiving Opus.
  • Speech-to-speech Browser Test: the platform mints a realtime browser-session token at POST /api/runtime/realtime-browser-sessions, the runtime exchanges it at /realtime/client-secret for an OpenAI Realtime client secret, and the browser connects with native WebRTC.
OPENAI_API_KEY stays inside the runtime. OPENAI_BASE_URL only affects the Sandwich LLM stage; OpenAI Realtime browser secrets and device bridges call OpenAI’s Realtime API from the runtime. See Configuration for stage settings and Browser Test for the live test loop.

​
Device activation and binding

OpenDot device activation and binding diagram

Device activation starts at OTA bootstrap, then stores device credentials and applies the bound identity config on authenticated sessions.

Firmware reaches the runtime OTA endpoint first. The runtime proxies activation requests to the platform API, the platform creates or verifies a device activation request, and the user claims the spoken code in Dot Device. After claiming, the platform stores an active device credential. Authenticated /ws connections use that credential, and the runtime loads the currently bound agent configuration from the platform API before choosing Sandwich or Speech-to-speech. See Dot Device console for the product flow and the firmware README for flashing and LAN setup details. See Dot device for the firmware and hardware reference target.

​
Data ownership

OpenDot data model map diagram

OpenDot-owned PostgreSQL tables keep identity mapping, agents, pipelines, devices, sessions, and deployments durable.

PostgreSQL stores OpenDot-owned product data: app users, local auth credentials, preferences, SDK API keys, versioned agents and pipelines, devices, activation requests, device credentials, runtime session tokens, device state, deployments, and deployment device targets. Supabase Auth can provide identity, but OpenDot keeps durable product state in its own tables. When changing persisted platform data, update the Drizzle schema, SQL migrations, API types, environment examples, and docs together.