MDClarity · data platform

Data Foundation & RevFind v2diagram family · v2 · A1 concept · A2 architecture · B serving

shared handoff node: customer bucket (S3)
status: directional — grain & patterns in flux
Mon 2026-07-06
data movement control / orchestration IP-rich transform owned store current→future seam

Three diagrams, one family. A1 says what the layers mean; A2 says how the compute works; B says how the product reads. All three hand off at the same node — the customer's S3 bucket — so they stitch together.

A1 · Concept

Medallion progression — what the layers mean

No tools here — just meaning. Layers, not steps: each is a resting shape data can sit in. Aggregation is the line between silver and gold. Replayability lives in bronze: silver is always re-derivable to a point in time from bronze.

when gold pressure recurs, ask whether silver should evolve — don't just wire it through

Source systems

land into bronze

Bronzelayer, not a step

what comes in, made trustworthy — without leaving the raw behind.

raw

exactly as landed
nothing adjusted

frontier

cleaned, deduplicated, history-aware

"shining glory"

Replayability lives here

Silver can always be re-derived to any point in time by replaying bronze forward.

Fix a transform → replay history.

future: table formats + time-travel / SCD2

Silverour domain

how we model our domain entities.

"the API entities"

if the only way in were an API, these are what we'd have you send

NO aggregation

transaction tables, not roll-ups

defined by

grain · cardinality · relationships

canonical-ID / stitch layer → own diagram (Emerson)

aggregation
happens here
Goldthe product

shaped to the product's query patterns — designed for experiences.

standard tables

shared across customers where we can

sidecar tables

per-customer / per-vertical variation

accrete safely · formalize at rule-of-3

not single-source-of-truth

same data may appear in many shapes — as long as versions land atomically

no inconsistent view, ever

RevFind

reads gold

layers, not stepsEach tier is a shape data rests in, not a stage it passes through. Bronze→bronze movement is normal before anything reaches silver.
the aggregation lineSilver holds transaction-grain domain entities with no roll-ups. The moment you aggregate, you're producing gold.
replay from bronzeBronze builds up a replayable history of source data.
lineage & provenance plumbed throughTransforms annotate as they go — added columns track the provenance of data as it flows through bronze→silver→gold. Dagster gives table-level lineage across the asset graph, column-level when enabled.
A2 · Architecture

Compute architecture — once it's in the house, it's all the same

The architectural truth A1 hides: past bronze/raw, bronze-vs-silver stops mattering. It's uniformly DuckDB transforms over Parquet on S3. Ingest heterogeneity — many transports, many formats, sometimes VPNs — is quarantined to one funnel; the boundary is "getting to S3 is messy; past S3 it's just IAM + Parquet."

Getting to S3 — heterogeneousVPNs · per-source · sometimes nasty

Pushed to us

SFTP · customer-run scripts

scripts need updating ▸

Pulled by us

SQL · Snowflake · APIs · S3 · SFTP

reach out & fetch

formats (tags)

EDI · HL7v2 · FHIR · flat

full taxonomy → own diagram

Past S3 — uniform: IAM + Parquet, nothing else
Dagster+ (hybrid) — orchestratorasset graph · dependencies · retries · asset checks · freshness · launches compute

Ephemeral compute · ECS / Fargate · pay-by-the-second

one container

packed: many assets / job

DuckDBin-memory computeexternal-materialize only

…or a burst of containers

fan-out · all one customer

DuckDuckDuck

every asset: reads Parquet from S3 → writes Parquet to S3

lingua franca of data engineering: Python + SQL

Customer bucket (S3)

bronze/ rawsilver/gold/

shared handoff →

dlt — stateful ingest

  • incrementality: what we've already ingested vs. not
  • schema-change detection → alert + don't promote
    (hold downstream materialization)
  • config-driven sources

not mandatory — for stateful problems

dbt Core — reusable SQL

  • lingua franca for SQL transforms
  • go-to, not required
  • libraries / collections → reuse
  • column-level lineage (free)
  • catalog/schema → Athena

today: Glue catalog · future: dbt-authored ▸

dlt + dbt both run via Dagster plugins

Pricing Service

the one IP-rich transform · CalcEngine

  • rich enrichment on the silver→gold path
  • .NET on ECS · fed by pipeline
  • exposes an interface; owns its storage privately

app-layer contract, not a lake asset

not called by the app in RevFind

the contained boundaryGetting to S3 can need VPNs and per-source glue and is genuinely messy. Past S3, every asset is just IAM + Parquet. That contrast is the point of the diagram.
DuckDB = pure computeIn-memory over ephemeral compute, with assets read from and materialized to Parquet files on S3 (not duckdb files).
data layer vs. app layerA lake asset's interface IS its schema + data. The pricing service instead exposes an interface and owns its storage privately — the app-layer contract.
schema-change → haltdlt detects schema drift and holds promotion — new data doesn't materialize downstream until it's dealt with. The irreversible×silent failure class, wired shut.
Deferred to their own diagramsFull ingest taxonomy (transport × format, push/pull, VPNs) · customer-bucket structure (medallion + non-medallion top-level folders, accretion governance, "delete-the-bucket-safely") · security & isolation (CMK, tag-fencing, cross-account trust) · canonical-ID / stitch layer (Emerson) · SFTP→Transfer Family move into bronze/landing (handoff-able work item).
B · Serving

Serving — two read paths over one bucket

The product reads gold two ways, both in place, both from the same bucket A2 handed off. Omni reads analytically via cross-account Athena; the RevFind API reads record-level via in-process DuckDB. Nothing writes back here.

RevFind v2 — the app

React SPA · serves end users

Cognito OIDC

tenant from claim

Omni embed (iframe in product)

server-signed URL · embed data path bypasses backend

API layer

.NET minimal API · ECS · Api→Infra→Core

in-process DuckDB reader

record-level reads of gold, per request

deliberate v1 · latency floor to evolve later

Customer bucket

shared handoff from A2

gold/Parquet · read in place

Omni analytics

embedded in product · cross-account, read-only

Omni SaaS → assume role (cross-account)

dedicated analytics account = only trust surface

Athena

+ Glue catalog

semantic model

governed · MCP/agentic folds in here

analytical scans — reads the same gold/, in place

two readers, one goldOmni (analytical, via Athena) and the API (record-level, via in-process DuckDB) read the same gold Parquet in place. Different engines, same source, no copies.
app owns no lake dataRevFind reads precomputed gold; it doesn't own or mutate it. Auth is real (Cognito, tenant from claim) because it serves end users.
DuckDB is the deliberate v1In-process DuckDB wins on simplicity now and carries a latency floor that will eventually force an evolution. Start-small everywhere; the high bar is reserved for security.
Pulled out of this diagram (their own sketches)Annotation round-trip (Aurora → event → worker → annotations/ → Athena view) — directional, unreviewed · isolation & compliance layer (CMK, tag-fencing, per-Omni-customer connection, cross-account trust) — different audience · referential-stability seam — rides with the annotation sketch.