RK
back to work
Case studyConfig-driven ingestion2025

Data Feed Engine

A new data feed should be a config file, not a new pipeline.

Most ingestion code gets copy-pasted per source: new connection logic, new incremental logic, new logging, new bugs. I built an engine where every feed shares one run contract — validate, extract, process, check, load, emit metrics — and adding a feed means writing one JSON config and one small class.

Impact

one JSON config + one class
Files to add a feed
2
resolved through the registry
Runner changes per feed
0
validate → extract → process → DQ → load → metrics
Run phases, one contract
6
JDBC (MySQL) · S3 parquet
Datasources live
2

01/Architecture

How the pieces fit.

framework.html open full screen
Data Feed Engine architecture diagram

02/How it works

4 stages, each with a job.

stage 01

Bootstrap

  1. 01A Lambda event or CLI call (config, date, full_load) is parsed into argv and handed to the runner.
  2. 02The runner loads the feed's JSON config and asks the factory for its feed_type.
  3. 03The factory resolves classes through a registry filled by @register_feed and @register_datasource decorators.

04/Engineering decisions

What I optimised for.

01

Registry over if/else

New sources and feeds register themselves by decorator, so the runner and factory never change when the catalogue grows.

02

Always leave a trace

Metrics are emitted in a finally block — a failed run still records its status and error for the next one to read.

03

State lives with the data

The incremental watermark is read from the feed log in S3, so runs stay stateless and Lambda-friendly.

04

Idempotent output keys

A SHA-256 hash of the composite key lands on every row, so downstream merges can dedupe reliably.

05/Stack

Built with.

Python 3.12pandasPySparkSQLAlchemyPyMySQLboto3s3fsPyArrowPydanticAWS LambdaDockerSonarQube

Next case study

TransitFlow Realtime Event Stream

Five live IoT streams, one Spark job, analytics-ready in S3.