Case study
← All workAI security operations platform
A 23-service pipeline that ingests security telemetry, enriches it in flight, and runs its own models over the stream.
- Client
- Undisclosed — under NDA
- Year
- 2026
- Discipline
- Security · Data platform
- Status
- Delivered
Context
Security operations drown in their own telemetry. A mid-sized estate emits more events per day than any team can read, and the existing tooling answers the wrong question — it tells you what happened, exhaustively, and leaves you to work out which fifty of the two million lines mattered. The brief was a platform that ingests that firehose from the tools already in place, decides what is worth a human's attention, and does so fast enough that the answer is still useful.
Constraints
Three constraints shaped everything. The platform could not ask anyone to replace their existing SIEM, so it had to speak that ecosystem's protocols and accept forwarded events rather than demand a migration. Enrichment could not block ingest — the moment lookups sit in the write path, a slow third-party API becomes dropped security events. And the models had to be retrained on the platform's own accumulated data without the training load ever competing with inference for the same workers, because a scoring pipeline that stalls during retraining is a blind spot on a schedule.
Architecture
Events arrive over an HEC-compatible ingress or a connector that forwards from the existing SIEM, and land on Kafka immediately — that first hop is deliberately the cheapest possible write, so ingest survives anything happening downstream. From the log, a normaliser puts every source into one shape and an enrichment engine adds context. Enriched events go to ClickHouse, which is a columnar store precisely because the queries are analytical scans over time ranges rather than row lookups. The model workers read the same Kafka topics independently of that path, so scoring runs beside the write rather than inside it. MongoDB holds configuration and case state, Redis is broker and cache, and an API gateway is the only thing the admin interface talks to. Five worker pools are separated by job — default, inference, training, monitoring, pipeline operations — so no class of work can starve another.
01
Kafka as the first write, before any processing.
Ingest availability is the one property a security platform cannot trade away. Writing to an append-only log first means the pipeline can be slow, restarting or broken without a single event being lost — everything downstream becomes a consumer that can catch up.
What it cost
A schema registry and the operational weight of a streaming backbone, which is real overhead for a system that could otherwise have written straight to a database.
02
ClickHouse for events, MongoDB for state.
These are two different workloads wearing the same word 'database'. Event analysis is a columnar scan over enormous time ranges; configuration and case state is small, mutable, document-shaped. One engine doing both does one of them badly.
What it cost
Two datastores to operate, back up and reason about, and a boundary that a careless query can cross expensively.
03
Training and inference on separate worker pools.
Retraining is bursty and enormous; inference is continuous and latency-sensitive. Sharing a pool means every retrain quietly degrades detection at exactly the moment the model is being improved.
What it cost
Idle capacity in the training pool between runs, and a more complex deployment topology than a single generic worker fleet.
- Services in the deployed topology
- 23
- read from deployment
- Dedicated worker pools
- 5
- read from deployment
- Datastores
- ClickHouse · MongoDB · Redis
- read from deployment
- Stream backbone
- Kafka with schema registry
- read from deployment
- SIEM ingest path
- Splunk HEC connector
- read from deployment
The full topology runs as 23 services — ingress, stream, normalisation, enrichment, five worker pools, three datastores, gateway and admin interface — deployed as a staging environment ahead of production. The architecture is the deliverable at this stage: the ingest path, the storage split and the worker separation are all proven under a real deployment rather than a diagram.
Stack
Python · Kafka · ClickHouse · MongoDB · Redis · Celery · Docker · Splunk HEC