AURUM as-built architecture

AURUM as-built architecture An architecture diagram generated by Archify. EventBridge Scheduler · 3 cron schedules (UTC) · AWS us-east-1 · managed by Terraform · a0.awsstatic.com EventBridge Scheduler 3 cron schedules (UTC) Step Functions · daily-market · edgar · train · AWS us-east-1 · managed by Terraform · a0.awsstatic.com Step Functions daily-market · edgar · train SNS alerts · on failure · AWS us-east-1 · managed by Terraform · a0.awsstatic.com SNS alerts on failure Yahoo Finance · daily OHLCV · Architecture component · finance.yahoo.com Yahoo Finance daily OHLCV SEC EDGAR · XBRL statements · Architecture component · www.sec.gov SEC EDGAR XBRL statements Ingest tasks · ECS Fargate · AWS us-east-1 · managed by Terraform · a0.awsstatic.com Ingest tasks ECS Fargate RDS Postgres · bronze · silver · gold · AWS us-east-1 · managed by Terraform · PostgreSQL RDS Postgres bronze · silver · gold dbt build · 237 tests · AWS us-east-1 · managed by Terraform · docs.getdbt.com dbt build 237 tests Train task · LightGBM + SHAP · AWS us-east-1 · managed by Terraform · lightgbm.readthedocs.io Train task LightGBM + SHAP Model registry · EFS models · S3 runs · AWS us-east-1 · managed by Terraform · a0.awsstatic.com Model registry EFS models · S3 runs GitHub Actions · CI · terraform apply · Architecture component · GitHub Actions GitHub Actions CI · terraform apply ECR image · one pinned image · AWS us-east-1 · managed by Terraform · a0.awsstatic.com ECR image one pinned image Terraform · infra/terraform · provisions all 46 AWS resources · S3 state + lockfile · Architecture component · Terraform Terraform · infra/terraform provisions all 46 AWS resources · S3 state + lockfile HTTPS HTTPS landing medallion SQL mart_features promote cron runTask.sync runTask.sync fail push pull apply provisions provisions provisions AWS us-east-1 · managed by Terraform

3 scheduled pipelines

  • • daily-market · 22:30 weekdays
  • • semimonthly-edgar · 1st + 15th
  • • monthly-train · 1st, 12:00

Guarantees

  • • dbt runs after every ingest
  • • dbt failure never reaches training
  • • Non-zero exit ends execution red

Terraform manages (46)

  • • 3 state machines · 3 schedules
  • • ECS + 4 task defs · ECR · EFS · S3
  • • RDS · SNS · CloudWatch · IAM · SSM