RK
back to work
Case studyStreaming · Kafka + Spark2024

TransitFlow Realtime Event Stream

Five live IoT streams, one Spark job, analytics-ready in S3.

A vehicle travelling London → Birmingham emits vehicle, GPS, traffic-camera, weather and emergency events. The goal: move all five streams through Kafka, give each a schema in Spark Structured Streaming, and land them in S3 where Glue, Athena and Redshift can query them — without a manual step in between.

Impact

vehicle · GPS · traffic · weather · emergency
Live event streams
5
per stream, on event time
Late-data watermark
2 min
Glue ETL into Redshift
Faster reporting
~30%
Dockerised Kafka + Spark
Less setup time
50%

01/Architecture

How the pieces fit.

transitflow-architecture.html open full screen
TransitFlow Realtime Event Stream architecture diagram

02/By the numbers

What the data says.

Fields per event stream

Each Kafka topic gets its own explicit Spark schema.

  • vehicle_data10 fields
  • weather_data10 fields
  • emergency_data8 fields
  • gps_data6 fields
  • traffic_data6 fields

03/How it works

4 stages, each with a job.

stage 01

Simulate

  1. 01A Python producer moves a vehicle from London to Birmingham in 100 steps with realistic jitter.
  2. 02Each step emits five correlated events sharing device id, timestamp and location.
  3. 03Events are keyed by id and serialised to JSON with a delivery callback per message.

04/Engineering decisions

What I optimised for.

01

Topic per event type

Streams evolve independently — a new weather field never forces a change to the vehicle pipeline.

02

Schemas at the edge

Explicit Spark schemas turn loose JSON into typed columns before anything lands in the lake.

03

Checkpointed sinks

Each query has its own S3 checkpoint, so a restarted job resumes from its last committed offset.

04

Local first

Kafka, ZooKeeper and Spark come up with one docker-compose command — the whole stream is reproducible on a laptop.

05/Stack

Built with.

PythonApache KafkaZooKeeperSpark Structured StreamingPySparkAmazon S3AWS GlueAthenaRedshift ServerlessDocker Compose

Next case study

Reddit Sentiment ETL

How do data communities on Reddit actually feel? A daily pipeline that keeps score.