The ML Loop — Production ML Architecture

This document describes di-factory's reference architecture for machine learning systems in production. It is the design we deploy when a client asks the honest question: "how do we keep a model good after it ships — and prove to CNBV, NOM-024, or LFPDPPP that we did?" Seven stages, arranged as a closed loop, cut clockwise from ingested data to deployed model to monitored drift to retrain trigger — with Governance and Audit & Lineage as first-class stages, not a compliance appendix.

The audience is technical. If you are evaluating whether di-factory can put a production ML system on your cloud — with the artifact stores, the drift monitoring, the retrain automation, and the regulator-grade audit trail — this is the document.

Design principles

Four commitments shape every decision below.

1. Loops beat pipelines. Every ML model degrades in production — the world drifts, the data drifts, the model drifts, and the pipeline you shipped last quarter is not the pipeline that should be running this quarter. Competitors draw the ML workflow as a left-to-right waterfall. We draw it as a loop with a bold drift → retrain arrow because that is the technical truth and the commercial one: di-factory delivers ongoing operation, not a one-shot notebook. If the diagram doesn't show the arrow, the system doesn't survive.

2. Stack-neutral by construction. The same seven-stage loop is delivered on Open Source, AWS, GCP, or Azure — four tabs, one contract. Procurement, security, and finance pick their preferred vendor path without asking us to redesign. The core commitment is the shape of the loop; the tool cells inside each stage are configuration. Clients on hybrid setups (regulated data on-prem, elastic training in the cloud) get a mixed path — the boundary is a policy decision, not a rewrite.

3. Artifacts outlive engineers. Every stage generates persistent artifacts that survive runs, laptops, and staff turnover — Feature Store (Feast · SageMaker Feature Store · Vertex FS · Azure ML FS), ML Registry (MLflow · SageMaker Model Registry · Vertex Model Registry · Azure ML Model Registry), Lineage Repository (OpenLineage · Marquez · Dataplex Lineage). What the client buys is infrastructure, not notebooks that die when a laptop is lost. This is the difference between "we trained a model for you" and "you own a production ML capability now."

4. Compliance is a stage, not a bolt-on. Governance and Audit & Lineage are stages 6 and 7 — not an afterthought, not a policy PDF, not a checkbox added on the last sprint. LFPDPPP (personal data), CNBV Cir. 20/2021 (regulated financial entities), and NOM-024 (health data interchange) are wired into the loop from the first run. Every decision the model makes produces a lineage record that ties the score back to the exact feature vector, the exact model version, and the exact training data snapshot that produced it. When the regulator asks "why did the model reject this customer?", we answer with a trace ID, not adjectives.

The 7 stages

The system is a closed loop organized into five phases — INGEST & PREP (get the data honest), BUILD (train and evaluate the model), SHIP (deploy to production), OPERATE (monitor for drift and close the loop with retraining), and GOVERN (policy, audit, lineage). Data enters at stage 1, a working model reaches production at stage 4, drift closes the loop back to stage 1 via the retrain arc. Governance and Audit & Lineage are stages 6 and 7 — they have cross-loop reach (every stage that touches data reports up to them) but they are first-class stages in the architecture, not diffuse "planes."

Phase 1 — INGEST & PREP

Stage 1 — Data. Ingest, validate, version (Airflow · DVC · Great Expectations · PostgreSQL · MinIO/S3). Raw sources — data lakes, warehouses, event streams, third-party feeds — land in a validated, versioned dataset before any model touches them. Great Expectations enforces schema and value contracts at ingest; a violation stops the run before it poisons downstream stages. DVC pins the exact snapshot of data used for a training run, which is what lets us reproduce a model six months later when the regulator asks. This stage is where "we trained it on real data" becomes "we can prove which data we trained it on."

Stage 2 — Features. Engineer, store, serve (Feast · pandas · Polars · JupyterLab). Feature engineering is where domain knowledge enters the loop — bureau ratios for banking, seasonality for retail, patient history for healthcare. The Feature Store is not a nice-to-have — it is what makes offline training and online inference agree on the definition of a feature. Skew between "the feature we trained on" and "the feature we serve" is the single most common cause of models that pass eval and fail in production. We eliminate the class of bug, not the incident.

Phase 2 — BUILD

Stage 3 — Train & Eval. Fit, tune, explain, register (scikit-learn · XGBoost · PyTorch · MLflow · Optuna · SHAP). This is where the model gets made. MLflow tracks every experiment — hyperparameters, metrics, environment, code hash — so a run is reproducible and comparable. SHAP produces per-decision explanations that ride with the model into production; when a banking underwriter or a regulator asks "why this score?", the answer is a SHAP plot, not a hand-wave. Eval is not just "does it beat baseline" — it's disaggregated performance (by segment, by geography, by protected class where policy applies), calibration, and stability under the past N weeks of data. A model that beats baseline on average and fails on the segment you actually care about is not shipped.

Phase 3 — SHIP

Stage 4 — Deploy & Serve. Package, expose, scale (FastAPI · Docker · Kubernetes · MLflow Model Registry). The registered model is packaged into a serving container with a versioned API contract, deployed behind a stable endpoint, and scaled to the client's throughput profile — a credit scorer at 500 rps, a demand forecast in a nightly batch, an insurance triager in a webhook fan-out. The Model Registry, not the git repo, is the promotion authority: a model version moves from Staging to Production on an explicit promotion event with lineage attached. Rollback is a version pin, not a redeploy.

Phase 4 — OPERATE (the loop closer)

Stage 5 — Monitor. Drift, latency, quality, cost (Prometheus · Grafana · Evidently AI · Loki · Alertmanager). The deployed model is watched on four axes at once: (1) input drift — is the incoming feature distribution still what we trained on, (2) output drift — is the score distribution still stable, (3) label drift — when ground truth arrives (weeks or months later), is calibration holding, (4) system health — p95 latency, error rate, cost per prediction. When any axis crosses its threshold, Evidently emits a drift event; the retrain arc — this is the drift → retrain arrow on the diagram — enqueues a retraining run against stage 1's most recent validated snapshot. The loop closes automatically for the shapes where automation is safe (retail forecasts, no-show); it enqueues a human-approved retrain for the shapes where a regulator is watching (credit, claims). Either way, the model doesn't decay silently.

Phase 5 — GOVERN

Stage 6 — Governance. Identity, secrets, policy (Keycloak · HashiCorp Vault · Model Card Toolkit · OPA). Governance is the "who is allowed to do what to which model on whose data" plane, promoted to a first-class stage because in regulated MX verticals it is the plane the regulator actually reads. Keycloak/OIDC gates access to the pipeline. Vault holds credentials to data sources and cloud APIs — no secrets in code, no secrets in notebooks. OPA (Open Policy Agent) enforces model-promotion rules — a model cannot leave Staging for Production without passing eval thresholds, drift-baseline registration, and explainability attachment. The Model Card Toolkit produces a public-facing card per model version — intended use, training data, evaluation results, known limitations — which is what CNBV, NOM-024, and IFT inspectors want to see when they show up.

Stage 7 — Audit & Lineage. Row-to-decision traceability (OpenLineage · Marquez · PostgreSQL audit tables). Every prediction in production carries a trace ID that resolves to: the exact model version, the exact feature vector, the exact training data snapshot (via DVC hash), the exact code commit, the exact promotion approver, and the exact SHAP explanation attached to the score. When the CNBV inspector, the LFPDPPP officer, or an internal audit team asks "why was this customer rejected on 2026-08-14 at 14:23?" — we answer with a lineage query, not a war-room. This is the stage that turns "we have a model" into "we have a deployable model for regulated MX industries."

The drift → retrain feedback arc

The single arrow on the diagram that changes the story is the orange arc from stage 5 (Monitor) back to stage 1 (Data). Without it, "ML architecture" is a slide from 2020 — a left-to-right waterfall that produces a model, ships it, and moves on. With it, the architecture survives production. The arc carries three signals: (1) drift detected — retrain with a fresher data snapshot on the current model shape; (2) performance degraded — retrain with a fresher snapshot and re-tune hyperparameters; (3) concept shift — the target itself has moved (e.g. a new fraud pattern), retrain requires new features, escalate to a data scientist. The retrain cadence is a per-model policy: quarterly for banking (regulator wants a stable model), weekly for retail (SKU × store demand shifts fast), monthly for healthcare (patient behavior shifts seasonally). The arc is why the client keeps buying month N+1, not just month 1.

Where it plugs into your stack

Three integration surfaces.

Inbound (data). Stage 1's ingest layer connects to whatever the client already runs — data lake (S3, GCS, Blob), warehouse (BigQuery, Snowflake, Redshift), event stream (Kafka, Kinesis, PubSub), OLTP (Postgres, SQL Server, Oracle), or third-party feed (bureau, weather, geospatial). If the data lives somewhere, we consume from it. Great Expectations sits at the boundary — bad data doesn't enter the loop.

Outbound (predictions). Stage 4's serving layer exposes the model in the shape the client's application needs — synchronous REST (credit scoring at underwriting time), batch scores landed to a warehouse table (nightly demand forecasts), event-driven webhook (claims triage on submission), or a streaming inference lane on Kafka/Kinesis (fraud detection at transaction time). The API contract is versioned; clients pin to a version and receive breaking-change notice.

Outbound (alerts). Stage 5's drift detectors and quality monitors emit events into the client's existing on-call and analytics tooling — Slack, PagerDuty, Teams, Opsgenie, or an existing observability suite (Datadog, New Relic, Dynatrace). We do not replace the client's alerting infrastructure — we feed into it, with full context (model version, drift axis, threshold breached, suggested action) attached.

The four stack tabs (OSS · AWS · GCP · Azure) render the same seven-stage loop against the client's preferred vendor path. A tenant on AWS gets MWAA + SageMaker + CloudWatch + Glue Lineage; a tenant on OSS gets Airflow + MLflow + Grafana + OpenLineage; the architecture is identical.

Business cases we deploy this for

Same loop, different domain, different features, different retrain cadence. Six shapes we see repeatedly in the MX market:

• Credit scoring augmentation (banking · CNBV). Bureau + behavioral features → gradient boosting → scoring endpoint with SHAP explanations. Quarterly retrain governed by drift; per-decision explainability that stands up to a CNBV audit. +15–25 pts on approval rate at flat delinquency is the recurring outcome.
• SKU × store demand forecasting (retail). Seasonality + price + promotion features → XGBoost/DeepAR → daily forecasts landed to the ERP. Weekly retrainable; per-SKU error monitoring with alerts on drift. −12–18% inventory carried, −30–40% stockouts on A-class SKUs.
• Appointment no-show prediction (healthcare · NOM-024). Patient history + booking channel + distance + weather features → calibrated classifier → orchestrated reminders and calibrated overbooking. Monthly calibration monitoring per clinical unit. −20–30% idle capacity, +8–12% patients seen per physician per week, NOM-024 audit trail per decision.
• Automated claims triage (insurance · LFPDPPP). Text + structured features → multiclass classifier → severity + complexity + route (fast-track / junior / senior / fraud). Monthly drift-driven retrain; LFPDPPP-compliant handling of medical and personal data. −40% fast-track resolution time, +25% first-contact NPS, senior adjusters reserved for genuinely complex cases.
• Transaction fraud detection (banking / fintech · CNBV). Real-time transactional features → streaming classifier at Kafka/Kinesis edge → block/challenge/allow decision under 50 ms p95. Daily retrain against labeled feedback from the fraud ops team. Precision + recall trade-off is a per-tenant policy — chargeback cost vs. customer friction — and the retrain arc respects it.
• Customer churn prediction (retail / telecom / SaaS). Behavioral + tenure + engagement features → survival or classification model → risk scores fed into the CRM's retention workflow. Retrain monthly against arriving churn labels; per-segment calibration monitoring to catch drift in specific customer cohorts.

The loop is the same. The features, the retrain cadence, the serving pattern, and the compliance envelope are configuration.

What this architecture is not

Anticipating the questions we get in every CDO / VP-DS evaluation:

• This is not RAG. RAG answers open-domain questions against a corpus using retrieval + LLM. This is deterministic scoring — a numerical prediction from structured features, with a SHAP explanation, no generative model in the hot path. The two compose (an ML classifier can be a tool a RAG or agentic system calls), but they are different systems with different failure modes and different regulator conversations. When a client only needs "answer questions from our docs," we deploy the RAG architecture, not this one.
• This is not agentic. The agent decides what to do by reasoning over tools; the ML model here decides how to score by learning over features. Agents are non-deterministic and act in the world; this model is deterministic (given identical features, identical model version, identical seed) and does not act — it emits a score, and the client's application decides what to do with it. Same reason a scoring model is auditable in a way a chat agent isn't.
• This is not AutoML-in-a-box. We don't hand you a wizard that fits ten models on your data and picks one. AutoML picks the model; it doesn't operate the loop. The loop — drift monitoring, retrain automation, artifact stores, lineage, governance — is the product. The base algorithm inside stage 3 is a choice we make with you based on the shape of the problem, not a hidden default the tool picked.
• This is not a research notebook. Notebooks die when a laptop is lost. This architecture doesn't. Every stage produces persistent, versioned artifacts — the Feature Store, the Model Registry, the Lineage Repository, the audit tables. What you own after we finish is infrastructure and a running loop, not a Jupyter file.
• This is not one-shot delivery. The retrain arc means we — or your team, once we've handed off — keep operating the loop. "We built you a model" is a 2020 sentence. "We built you a loop that keeps the model good" is the 2026 one. Our ML delivery LOB is priced for the loop, not for the notebook.
• This is not fine-tuned foundation models. Fine-tuning an LLM is a different architecture with a different compliance envelope (the LLM sees the training data), a different serving profile (GPU inference), and a different failure mode. If a client needs classic ML — tabular prediction, forecasting, structured scoring — this is the loop. If they need a fine-tuned LLM for a specific NLP task, that's a scoped variant that shares stages 4–7 but replaces stage 3.

Latency and cost envelope

Realistic numbers, not marketing numbers.

• Online scoring (single request, tabular model, cached features): 10–80 ms p95 end-to-end. This is the common path for credit, fraud, and triage endpoints.
• Online scoring with feature enrichment (Feature Store lookup + model call): 50–200 ms p95. Adds a round trip to the Feature Store; still well inside a synchronous API budget.
• Batch scoring (millions of rows, distributed): minutes to a few hours. Nightly demand forecasts, monthly churn scores, weekly retention lists.
• Streaming inference (Kafka/Kinesis edge, sub-50 ms): requires a lightweight model shape and cached features. Fraud detection sits here. Not every model can live at this latency — we scope it up front, not after a POC ships.
• Retrain run: minutes (linear/tree models, <10M rows) to hours (deep models, 100M+ rows). Cadence: weekly (retail) to quarterly (banking). Cost per retrain is a line item in the monthly loop budget, not a surprise.

Monthly cloud cost lands in the low-thousands USD for a single-tenant retail or healthcare loop and scales with data volume, retrain frequency, and inference throughput. The observability plane at stage 5 lets you enforce cost ceilings per model, per stage, and per tenant — no runaway experiments.

Closing

This is the architecture we deploy for production ML. It is opinionated. Each opinion is a scar. If any of the design principles conflict with your environment — you can't touch a hosted service, your compliance office rejects shared infrastructure, your latency target is sub-10 ms, your retrain cadence is real-time — we adjust the specific stage, not the shape. The shape is load-bearing.

For a technical deep dive on any single stage, or a scoping conversation on your specific data sources and serving profile, the follow-up is a 30-minute call. Growgy (CGO) schedules it; Teky (CTO) runs it.