Data Feed Engine
A new data feed should be a config file, not a new pipeline.
Most ingestion code gets copy-pasted per source: new connection logic, new incremental logic, new logging, new bugs. I built an engine where every feed shares one run contract — validate, extract, process, check, load, emit metrics — and adding a feed means writing one JSON config and one small class.
Impact
- one JSON config + one class
- Files to add a feed
- 2
- resolved through the registry
- Runner changes per feed
- 0
- validate → extract → process → DQ → load → metrics
- Run phases, one contract
- 6
- JDBC (MySQL) · S3 parquet
- Datasources live
- 2
01/Architecture
How the pieces fit.
Entry point → runner → factory and registry → BaseFeed.run, with pluggable JDBC and S3 datasources.


02/How it works
4 stages, each with a job.
stage 01
Bootstrap
- 01A Lambda event or CLI call (config, date, full_load) is parsed into argv and handed to the runner.
- 02The runner loads the feed's JSON config and asks the factory for its feed_type.
- 03The factory resolves classes through a registry filled by @register_feed and @register_datasource decorators.
03/Evidence
Straight from the repo.
04/Engineering decisions
What I optimised for.
01
Registry over if/else
New sources and feeds register themselves by decorator, so the runner and factory never change when the catalogue grows.
02
Always leave a trace
Metrics are emitted in a finally block — a failed run still records its status and error for the next one to read.
03
State lives with the data
The incremental watermark is read from the feed log in S3, so runs stay stateless and Lambda-friendly.
04
Idempotent output keys
A SHA-256 hash of the composite key lands on every row, so downstream merges can dedupe reliably.
05/Stack
Built with.
Next case study
TransitFlow Realtime Event Stream
Five live IoT streams, one Spark job, analytics-ready in S3.

