Reference architecture
The platform that makes it possible
How a regulated card org turns raw network files into governed, entitled data products - and runs ML and GenAI on top without ever loosening control. Built on a data-product mesh; the live Postgres RLS plane at the bottom is the one runnable proof.
The spine - four paradigms, layered
Each layer answers a question a card org has to answer for a regulator. Together they are the mesh.
The entitlements/RLS stage is the one runnable proof - click it to open the live simulator.
One ACID store for raw auth streams and curated products - no warehouse/lake split to reconcile at examiner time.
A disciplined refinement path from raw network files to certified products - replayable, quarantined, contract-validated.
Card domains (spend, rewards, risk, partner economics) own and publish their own products; no central team bottleneck.
Policy is computed and enforced uniformly - row scope, column masking, purpose, lineage, audit - so a regulated bank can prove control.
Decentralized ownership of analytical data by the domain that knows it best.
Each domain serves its data as a discoverable, addressable, trustworthy product.
A platform that lets domains build and run products without bespoke plumbing.
Global policies (security, interop, quality) enforced as code across all products.
Data-mesh four principles - Dehghani / Thoughtworks canonical formulation.
Medallion, mapped to card data
The Gold tables ARE the published data products - the same ones in the live catalog.
Network and ledger events land exactly as received. PAN tokenized at landing.
- ›Card auths + settlement/clearing (ISO 8583 / ISO 20022)
- ›Visa / Mastercard network files
- ›Disputes & chargebacks, rewards events
- ›PAN tokenized at landing; append-only; replayable
- ›Ingested via streaming / Auto Loader
Cleaned, deduped, joined to reference data, and validated; bad rows quarantined.
- ›Joined to merchant / MCC / account / cardholder reference
- ›Deduped & type-conformed
- ›Validated with declarative pipeline Expectations (DLT / Spark Declarative Pipelines)
- ›Non-conforming rows quarantined, not silently dropped
Certified aggregates and ML feature tables. These Gold tables ARE the data products in the catalog.
- ›Certified business aggregates (spend, approval, fraud, attrition)
- ›ML feature tables with online/offline parity
- ›Contract + SLA + entitlement attached per product
- ›Consumed by BI, ML, partner reporting, and GenAI - never the raw lake
Domain-owned data products + data contracts
A data product = data + ownership + SLA + contract + consumers. A product without a contract is just a table.
Bitol project, Linux Foundation AI & Data - originated at PayPal
Open Data Product Standard (ODPS) v1.0 - the product wrapper around the contract
This is what a contract looks like, enforcedGovernance & entitlements - the Chase-scrutinized part
Two enterprise stacks, because a reviewer should see you know both.
- ABAC policies
Row-filter and column-mask policies bound to governed tags - attribute-based, not per-table hand-coding.
- Governed tags + classification
Auto-tag PII / financial data, aligned to GDPR / PCI / HIPAA controls.
- Row filters & column masks
Enforced uniformly across SQL, ML, and BI consumers.
- Automatic lineage
Table- and column-level lineage captured automatically for impact analysis and audit.
- System tables
system.access.audit, lineage, and billing tables - examiner evidence by query.
- Masking + external tokenization
Dynamic data masking and external tokenization integrations at column level.
- Row access policies
Row-level filtering bound to roles / context.
- Tag-based masking
Object tags drive masking policies so classification propagates automatically.
- Trust Center
Posture monitoring across the account.
- Classification / dependencies / access history
Data Classification, Object Dependencies, Access History, Data Metric Functions.
Vaultless tokenization + format-preserving encryption (FPE) applied to PAN at Bronze landing. Detokenization is itself an entitlement - granted only to roles and purposes that need clear values. Representative vendor: Protegrity (representative vaultless-tokenization vendor).
Atlas enforces row scope + purpose + column masking + audit on a real Postgres RLS plane. Open the live simulator - the one runnable governance component.
The JPMC home-turf reference (AWS-native mesh)
JPMorgan has publicly described an AWS-native data mesh. Atlas's design is consistent with it; the firm-level mesh, JADE, and the move to Databricks lakehouse + Snowflake Cortex are the closest public proxy for an internal card stack the firm does not publish.
One per data product / producer - hard isolation, not shared buckets.
Per-product Glue catalogs roll into a master catalog that tracks every share.
Data shared in place (not copied) at column / record / individual-value granularity.
Consumers query entitled shares directly; no extract proliferation.
Producer/consumer accounts roll up to a master line-of-business Lake Formation account.
Federation and policy enforced centrally; JADE = JPMorgan Advanced Data Ecosystem.
ML pipelines + GenAI on curated data
GenAI consumes only Gold / certified products through the catalog - never the raw lakes.
Atlas's live /console is exactly this: a Genie-style governed text-to-SQL surface running under the same RLS plane.
Mirrored in the console's model allowlist + SELECT-only run_select guard - the same idea, enforced in this demo.
Model risk - SR 11-7 → 2026 guidance
On April 17, 2026 the Federal Reserve, FDIC, and OCC replaced SR 11-7 (and OCC 2011-12 / FIL-22-2017) with a risk-based, principles-driven model-risk framework. GenAI and agentic AI are formally OUT OF SCOPE of SR 26-2 (the agencies deemed them novel and rapidly evolving, with an RFI to follow) - but they inherit the same MRM principles by analogy.
| Shift | Old - SR 11-7 (2011) | New - SR 26-2 (Apr 2026) |
|---|---|---|
| Scope | One-size-fits-all validation for all models | Risk-based tiering - controls scale with materiality |
| Cadence | Validation as a point-in-time gate | Lifecycle thinking - risk managed end-to-end |
| Challenge | Independent validation function | Effective challenge remains the backbone |
| Monitoring | Periodic revalidation | Continuous drift & outcome monitoring in prod |
| AI / GenAI | Not addressed | Out of scope of SR 26-2 - inherits MRM principles by analogy; RFI to follow |
Controls scale with model materiality and risk - not one-size-fits-all.
Risk managed across the full model lifecycle, not at a single validation gate.
Independent, competent challenge remains the backbone of validation.
Ongoing performance, drift, and outcome monitoring in production.
GenAI / agentic systems are out of scope of SR 26-2, but inherit the same MRM principles by analogy until dedicated AI guidance follows.
| Requirement | Component |
|---|---|
| Model inventory | Unity Catalog tables / model registry |
| Model cards | MLflow + UC Model Registry, bound to the production version |
| Validation / effective challenge | MLflow Evaluate: AUC / KS / PSI; LLM: groundedness, toxicity, PII-leak, LLM-as-judge |
| Continuous monitoring | Lakehouse Monitoring + Gateway telemetry |
| Promotion controls | UC Registry aliases + ABAC; Tier-1 dual control |
| Examiner RFI | Answered via a governed Genie space over the inventory + validation log |
Component cheat-sheet
The full name-drop set, grouped by layer.
Sources & methodology
- 01
- 02
- 03
- 04
- 05
- 06
- 07