data-feed-engine framework

data-feed-engine framework An architecture diagram generated by Archify. lambda_handler · event → argv · Architecture component lambda_handler event → argv cli.main · argparse -c -d -f · Architecture component cli.main argparse -c -d -f runner · run_feed_from_config · Architecture component runner run_feed_from_config factory · create_feed · create_datasource · Architecture component factory create_feed · create_datasource registry · FEED / DATASOURCE maps · Architecture component registry FEED / DATASOURCE maps Feed config JSON · artifacts/configs · Architecture component Feed config JSON artifacts/configs BaseFeed.run · extract · process · DQ · load · Architecture component BaseFeed.run extract · process · DQ · load Feed subclasses · reddit_posts · employees · Architecture component Feed subclasses reddit_posts · employees SQL files · artifacts/sql · Architecture component SQL files artifacts/sql JDBCDatasource · MySQL · SQLAlchemy · Architecture component JDBCDatasource MySQL · SQLAlchemy S3Datasource · parquet · sha256 hash_key · Architecture component S3Datasource parquet · sha256 hash_key ExecutionMetrics · → S3 feed_log CSV · Architecture component ExecutionMetrics → S3 feed_log CSV argv args create_feed lookup json.load instantiate @register_feed extends feed.run() get_data put_data emit read .sql Legend Frontend Backend Database Message bus External

BaseFeed.run lifecycle

  • • validate_connection on input + output
  • • get_data → process → DQ checks
  • • put_data · emit metrics in finally

Add a feed

  • • Subclass BaseFeed + @register_feed
  • • Implement process + DQ checks
  • • Add config JSON with feed_type

Add a datasource

  • • Subclass BaseDatasource
  • • @register_datasource(type)
  • • Implement get_data / put_data