TransitFlow real-time event stream architecture

TransitFlow real-time event stream architecture An architecture diagram generated by Archify. .env secrets · AWS keys · S3 bucket · Architecture component .env secrets AWS keys · S3 bucket Spark master · spark://:7077 · UI 9090 · Docker Compose · transitflow network · spark.apache.org Spark master spark://:7077 · UI 9090 Amazon Athena · serverless SQL on S3 · AWS cloud Amazon Athena serverless SQL on S3 IoT simulator · London → Birmingham · Architecture component · Python IoT simulator London → Birmingham Kafka broker · 5 topics · cp-server 7.4 · Docker Compose · transitflow network · Apache Kafka Kafka broker 5 topics · cp-server 7.4 Spark workers ×2 · 2 cores · 1g each · Docker Compose · transitflow network · spark.apache.org Spark workers ×2 2 cores · 1g each Amazon S3 lake · parquet + checkpoints · AWS cloud Amazon S3 lake parquet + checkpoints Glue crawler · schema discovery · AWS cloud Glue crawler schema discovery Glue Data Catalog · 5 tables · AWS cloud Glue Data Catalog 5 tables ZooKeeper · cluster metadata :2181 · Docker Compose · transitflow network · zookeeper.apache.org ZooKeeper cluster metadata :2181 Redshift Serverless · warehouse · dashboards · AWS cloud Redshift Serverless warehouse · dashboards produce subscribe s3a parquet crawl tables SQL Spectrum schedule coordinate AWS keys Docker Compose · transitflow network AWS cloud Legend Frontend Backend Database Cloud Security Message bus External

Kafka topics

  • • vehicle_data · gps_data
  • • traffic_data · weather_data
  • • emergency_data

Streaming guarantees

  • • 2-minute event-time watermark
  • • Per-topic checkpoints on S3
  • • bootstrap broker:29092

Scale out

  • • Add spark-worker-N to grow executors
  • • Kafka partitions set consumer parallelism
  • • S3 · Athena · Redshift scale serverless