data-feed-engine deployment architecture

data-feed-engine deployment architecture An architecture diagram generated by Archify. GitHub repo · push · pull request · Architecture component · GitHub Actions GitHub repo push · pull request SonarQube scan · quality gate · Architecture component SonarQube scan quality gate Container image · python:3.12-slim · awslambdaric · Architecture component · Docker Container image python:3.12-slim · awslambdaric Invoke event · config · date · full_load · Architecture component Invoke event config · date · full_load AWS Lambda · lambda_handler.handler · AWS · AWS_REGION AWS Lambda lambda_handler.handler data-feed-engine · cli → runner → feed · AWS · AWS_REGION · Python data-feed-engine cli → runner → feed S3 processed · parquet · load_type/date · AWS · AWS_REGION S3 processed parquet · load_type/date Env variables · MYSQL_* · S3_BUCKET_NAME · AWS · AWS_REGION Env variables MYSQL_* · S3_BUCKET_NAME MySQL source · reddit_db · test · Architecture component · MySQL MySQL source reddit_db · test S3 feed_log · CSV run metrics · AWS · AWS_REGION S3 feed_log CSV run metrics scan image event main(argv) put parquet SQL read metrics secrets AWS · AWS_REGION Legend Backend Database Cloud Security External

Invoke payload

  • • config · default reddit_posts.json
  • • date · YYYY-MM-DD, defaults to today
  • • full_load · overridden by config

Registered feeds

  • • reddit_posts_feed · MySQL r_posts → S3
  • • employee_data_feed · MySQL delta → S3
  • • Incremental watermark read from feed_log

Not defined in repo

  • • Scheduler / trigger for the Lambda
  • • Image build and deploy pipeline
  • • API datasource · Spark engine paths