Distributed Cloud Telemetry & Real-Time Event Platform
Designing a high-throughput event processing platform capable of ingesting and aggregating 150,000 events/sec with sub-second dashboard visualization.
Architecture & Tech Stack:
System Architecture
To handle volatile telemetry bursts from maritime and fleet IoT sensors, our team separated the ingestion ingress path from analytical compute workloads.
[ IoT Telemetry Devices ]
│
▼
[ Go Ingress Gateways (gRPC) ]
│
▼
[ Apache Kafka Partitioned Cluster ]
│ │
▼ ▼
[ Go Stream Worker ] [ Anomaly AI Worker ]
│ │
▼ ▼
[ ClickHouse Columnar DB ] ──► [ Real-Time Analytics UI ]1. Zero-Allocation Go Ingestion Ingress
Using Protobuf over gRPC with connection pooling and memory buffer pooling (sync.Pool), we minimized garbage collector pauses, keeping p99 ingestion latency under 4ms.
2. High-Density Columnar Analytics
By deploying ClickHouse with structured time-series partition keys, 100-million row analytical aggregates (such as fuel efficiency heatmaps and geofence compliance) return in under 30 milliseconds.