Architecture · Live cluster
Shown: net taker flow (buys minus sells) summed over each venue’s perpetuals, per minute, 18:00–01:00 UTC on 10 Oct 2025, buys above each venue’s line and sells below it on a square-root scale; the deepest sell minute, 21:13 UTC with $1.9B net sold across the three venues, is orange · built with flow 1m
Venues
Capture
Broker
Compute
Store
Deliver
Hover or select a component to trace what feeds it and what it feeds.
01 · Venues
Binance
USDⓈ-M futures
2 connectionsEvery public market-data channel we capture from Binance is streamed by two redundant producers, each on its own connection.
OKX
Perpetual swaps
2 connectionsEvery public market-data channel we capture from OKX is streamed by two redundant producers, each on its own connection.
Hyperliquid
Perpetuals, HIP-3 included
2 connectionsHyperliquid perpetuals, with the HIP-3 builder-deployed markets on the trade and quote feeds. Every channel is streamed by two redundant producers.
02 · Capture
Producers
Rust · 200+ instances
200+ streaming · 2 replicas per feedA fleet of over two hundred producers streams in parallel. Each feed is captured by its own process twice over, by two replicas on separate nodes, stamped with our capture time the moment it arrives and normalized to one schema across venues.
03 · Broker
Message broker (raw)
5 nodes · High Availability
5 nodesEvery raw record lands on a replicated, Kafka-compatible message broker that the stream engine and the archivers read independently. Both replicas of a feed publish here; the broker keeps the first copy of each event. A restarted consumer resumes from its own committed offset.
04 · Compute
Stream engine
Rust · streaming SQL
10 s checkpointsOur own engine, written in Rust, runs every metric as SQL generated from one registry: tumbling windows on capture time, ASOF joins, and order books rebuilt in-process for the L2 metrics.
Raw archivers
Rust · Kafka → Parquet
Replays on restartArchivers decode the raw records into Parquet and upload them in batches. The Kafka offset is committed only after the upload is acknowledged.
05 · Store
Message broker (metrics)
3 nodes · High Availability
3 nodesEvery metric, venue and interval has its own versioned topic on a second message broker, apart from the raw one, so customer reads never compete with ingestion.
Metric archivers
Rust · Kafka → Parquet
Replays on restartArchivers read every metric topic and write it to Parquet in batches. As on the raw side, the Kafka offset is committed only after the upload is acknowledged.
Object storage
Colocated + cloud · Parquet
Colocated + cloudRaw and metric archives, each written by its own archivers, land as partitioned Parquet on replicated object stores, both colocated and cloud-managed.
06 · Deliver
Kafka
Institutional · TLS + SASL
3 brokersOn Institutional plans, subscribe to the metric topics with any Kafka client. Credentials are per customer and scoped to the metrics you are entitled to.
WebSocket
Edge · JSON
Cloudflare edgeA bridge forwards the topics customers subscribe to into an edge Worker, so a script or a browser streams metrics with an API key and one subscribe message.
Pre-signed URL API
History · Parquet
Cloudflare edgeA history request resolves to short-lived pre-signed URLs, so files stream straight from object storage. The Python SDK and AI agents use the same API.
Every window keys on the moment our producer received the event, not the timestamp the exchange stamped on it. An exchange-time window cannot close until a late enough exchange clock reading arrives, which puts the whole network leg and every exchange batching delay inside the watermark. Keying on capture time also means a bar holds what you could actually have seen by its close: no look-ahead in a backtest.
The engine checkpoints its state and source positions every 10 seconds and remembers the last window it published to each topic. After a crash it resumes from the checkpoint and suppresses what it already wrote. A pipeline with no checkpoint withholds any window it cannot compute whole, so the worst case is a missing bar, never a wrong one.
200+ producers stream at once, and every feed runs on two of them, on separate nodes, so a dropped connection or a lost node never opens a hole in the data. Both publish the same events, and the message broker drops the second copy by each event’s ID, before any consumer reads it.
Archivers commit their Kafka offset only once object storage has acknowledged the Parquet file. A crash mid-batch replays the batch instead of skipping it, and every file gets a unique name, so a retry never overwrites another writer’s file.
A running pipeline’s SQL is never edited in place. A changed metric ships as a new version that reads the same raw topics and publishes next to the old one, on its own topics and Parquet prefix. Each released version’s SQL hash is pinned, and the build fails if it would ever compute differently.
We replaced a general-purpose streaming database with our own engine: one Rust binary per pipeline, threads rather than async on the data path, columnar batches, and an order book rebuilt in-process so the L2 metrics are plain SQL.
See every metric this cluster computes, or stream them into your own systems.
Shown: minute lows of the 465 Binance perpetuals, 20:40–22:00 UTC on 10 Oct 2025, with BTC’s path traced · built with ohlcv 1m