Medallion Architecture · Migration Campaign

Medallion architecture progress — laying our new data foundation

The road is largely built. First, the platform foundations already laid; then where each customer stands. The customer counts look early because the hard 0→1 — the foundations — is done: with the patterns and per-customer isolation in place, new customers now roll fast (three onboarded in ~1 PR each via shared libraries).
9
platform foundations laid
36
customers tracked
11
in flight in medallion
9
parked (started, stalled)
6
need EDI / X12 (parser shipped, in customer testing)
Last winEDI / X12 parser shipped — the gate that blocked every 835/837 customer is open; SuncoastSkin + PrecisionHealth are the first two through it
TrackingNew customers now onboard directly onto medallion — OCP, HillsboroEye, SuncoastSkin, PrecisionHealth, WomensCare are being built with no legacy pipeline at all
Ingest denominators & product mix are real (from the pipeline export). Medallion state is curated in medallion-status.yaml — as of 2026-08-12.

Foundations laid9 of 12 platform wins — the 0→1 the customer view hides

  1. Nov 2025S3 landing buckets
  2. Dec 2025ECS platform
  3. Jan 2026Dagster hybrid cloud
  4. Mar 2026OneGI: full medallion + write-back + checks
  5. May 20263 customers onboarded in ~1 PR (shared libs)
  6. Jun 202623 customer data stacks — fleet isolation
  7. Jun 2026filesystem-common: 9 customers' bronze in one PR
  8. Jun 2026ARSA: first FHIR-API customer (bronze + silver, in testing)
  9. Jul 2026EDI / X12 parser ships — the EDI gate opens
  10. Aug 2026ARSA live in prod — silver powers the legacy system
  11. Aug 2026New customers onboard straight onto medallion — no legacy pipeline built
✓ DoneDec 2025
ECS container platform
The Fargate platform every Dagster agent, pipeline run, and API runs on.
Infra #243
✓ DoneJan 2026
Dagster hybrid-cloud orchestration
Dagster Cloud control plane + our-AWS agents; ingest agent isolated in the SFTP VPC.
Infra #286 / #301
✓ EstablishedMar 2026
Silver → SQL Server write-back
Medallion silver powers the legacy gold / RevFind — atomic rebuild + rename-swap. Reused by 4 customers.
MdC #5282 / #5389
⚠ Needs repetitionMar 2026
Data validation (asset checks)
Catches silent bronze truncation + schema drift. Proven on OneGI; not yet on the athena pipelines.
MdC #5282
✓ DoneMar 2026
Managed SFTP (AWS Transfer Family)
Cognito-auth SFTP, jailed per customer/source into S3; replaces Bitvise.
Infra #330
✓ DoneMar 2026
SFTP → S3 retention shim
Inbound files replicated to S3 with retention — the legacy system deletes on read; we keep them.
Manual op · Mar 25 (no PR)
✓ DoneJun 2026
Customer-isolated HIPAA data lake
Per-customer S3 + own CMK + scoped task role + Object-Lock + cross-region replica. 23 customers bootstrapped.
Infra #418 / #437
✓ DoneJun 2026
Reusable FHIR connector (dlt)
Vendor-agnostic FHIR ingest — type-incremental + reference-following, pluggable OAuth2, rate-limited, count-only logging. Proven in prod by ARSA/NexTech; any FHIR-API customer reuses it.
MdC #5564
◷ In progressJun 2026
filesystem-common bronze core
Unit-tested CSV ingestion core + Dagster FilesystemSourceComponent — stands up bronze for 9 S3-landing customers in one PR. The PR stalled: those 9 customers are parked, not in flight.
MdC #5581
✓ Done
Gold available in Omni
Gold data queryable in Omni — Athena + Glue catalogs & connectors, with cross-account access into each customer's bucket wired for Omni.
✓ DoneJul 2026
EDI / X12 parser
First-class 835/837 parsing in medallion — the gate that held every EDI customer. Shipped; SuncoastSkin and PrecisionHealth are the first two customers building on it.
⚠ Needs repetitionAug 2026
On-prem NextGen extract
Extract scripts we run inside the customer's network, output pushed back over inbound SFTP — reaches EHRs with no share, no API, and no path to their SQL Server. Live for HillsboroEye; one customer so far.

The ingest surface — format · protocol · source

File formatwhat the payload is
CSV29
HL7 v27
Pipe-delimited7
EDI / X126
Multi-char delimited4
No file · share / API / direct SQL2
Tab-delimited2
Protocol / transporthow the data arrives
SFTP (push to us)26
Vendor API4
Vendor-host pull4
Snowflake share3
Direct SQL Server1
Source systemwhich EHR / vendor feed — exact per-source bronze state
NextGen8
HL7 interface7
EDI (835/837)4
eClinicalWorks3
athenahealth2
EDI 8352
FHIR2
ModMed2
Centricity1
Experity1
gGastro1
HL7 v21
MSN1
NexTech · FHIR1
NextTech1
Phygen1
QuickSight1
SFTP CSV1
Snowflake1
livecutoverbuilding / paralleluntouchedBar length = customers with that feed (real, from the export). Fill = bronze migration progress. The three axes are independent — one feed reads on all three.
Source × deliverywhere each feed actually lands — a source in two columns arrives two ways
Source systemSnowflake shareSFTP filesDirect SQL ServerVendor APIAll
NextGen·8··8
HL7 interface·7··7
EDI (835/837)·4··4
eClinicalWorks·3··3
athenahealth2···2
FHIR···22
ModMed·2··2
Centricity·1··1
Experity·1··1
gGastro··1·1
MSN·1··1
NexTech · FHIR···11
NextTech··1·1
Phygen·1··1
QuickSight···11
Snowflake1···1

Customers per source × delivery channel, tinted by bronze progress. eClinicalWorks spans SFTP files and direct SQL Server (Partnercare) — the same feed, two transports.

Not startedBuildingTestingFirst liveRollingAll done

Bronze — ingest capabilities

SFTP delimited files
Rolling

Delimited files — CSV / pipe / tab, plus the rare ^|^ multichar — parsed via ProcessDataFile. Usually pushed to our SFTP, sometimes pulled from a vendor host (a separate axis — see vendor pull). The workhorse and widest blast radius: it fires for any customer with a delimited feed, even a secondary one.

Not started
Building
Testing
First live
Rolling
▲
All done
3 of 28 customers live
🎉 First live: OneGI
AWS QuickSight export · Delimited extract · Experity (CCP) · FHIR API…ClarityFlowRevFind
HL7 v2 files
Building

File-based HL7 (ADT / SIU / DFT / ORM) — no MLLP listeners. Needs a parser + message-type mapping before first live.

Not started
Building
▲
Testing
First live
Rolling
All done
0 of 8 customers live
HL7 interface · HL7 v2 messages · MSN LLC (radiology service bureau) · ModMed…ClarityFlowRevFind
Frequent HL7 v2 processing (Flow)
Not started

ClarityFlow consumes HL7 continuously through the day — not the daily batch cadence the rest of medallion runs at. Bronze needs a frequent small-batch HL7 mode before Flow's HL7 customers can move. ClarityFlow-only: RevFind's daily cadence is unaffected.

Not started
▲
Building
Testing
First live
Rolling
All done
8 customers waiting · not yet built
⚠ Risk: every ClarityFlow HL7 customer is held until this exists — not yet started
HL7 interface · HL7 v2 messages · MSN LLC (radiology service bureau) · ModMed…ClarityFlowRevFind
EDI / X12 (835 / 837)
Testing

X12 claims & remits. Some pipelines use the in-repo EDIParser task; iSpine routes parsing through an off-repo PowerShell script (insert_edi.ps1). Medallion needs a first-class X12 parser either way.

Not started
Building
Testing
▲
First live
Rolling
All done
6 customers in testing · not yet live
⛔ Blocks ECP, ISpine, PrecisionHealth, SuncoastSkin, TheOregonClinic, USDH
EDI (835/837) · Snowflake (varies) · X12 EDI claims / remitsRevFind
Vendor-host SFTP pull
Not started

We reach out to *their* server (e.g. MSNFTP.MSNLLC.NET) instead of receiving a push — a transport-direction signal, independent of file format. Needs outbound creds + scheduling.

Not started
▲
Building
Testing
First live
Rolling
All done
4 customers waiting · not yet built
HL7 v2 messages · MSN LLC (radiology service bureau) · ModMed · X12 EDI claims / remitsClarityFlowRevFind
Snowflake data share
Rolling

athenahealth (and other Snowflake-share sources) read straight from a Snowflake share — no inbound files to receive or parse, though bronze still writes parquet like every source. Cloud-native, the smoothest path. NB: 'Athena' here is athenahealth, NOT AWS Athena.

Not started
Building
Testing
First live
Rolling
▲
All done
3 of 3 customers live
🎉 First live: Commonwealth
HL7 v2 messages · Snowflake (varies) · X12 EDI claims / remits · athenahealthClarityFlowRevFind
Vendor API (FHIR connectors)
First live

First-class FHIR API connectors — API client & token handling rather than file ingest. First customer is ARSA (medallion-native, no legacy export). ModMed's off-repo Flow-MM API (OneGI) is invisible to the export and not counted here.

Not started
Building
Testing
First live
▲
Rolling
All done
1 of 3 customers live
🎉 First live: ARSA
FHIR APIClarityFlow
Direct SQL Server queries
First live

Live OLTP reads against the EHR's SQL Server — no file drop at all. Spans NexTech (EyeSouth) and eClinicalWorks-over-SQL-Server (Partnercare); new-stack feeds the export can't see (OneGI's gGastro) are added via a curated override. Needs a network path + read credentials.

Not started
Building
Testing
First live
▲
Rolling
All done
1 of 2 customers live
🎉 First live: OneGI
ModMed · NexTech · NextGen · eClinicalWorks (ECW)ClarityFlowRevFind
On-prem NextGen extract
Complete

Scripts we run inside the customer's network pull data out of their NextGen SQL Server and push the output back to us over inbound SFTP. Distinct from direct SQL Server (we never reach their DB) and from plain SFTP delimited (the burden is the on-prem extract agent we install and maintain). HillsboroEye is the first — medallion-native, invisible to the legacy export.

Not started
Building
Testing
First live
Rolling
All done
▲
1 of 1 migrated — capability complete
🎉 First live: HillsboroEye
NextGenRevFind
QuickSight → S3 export
First live

SGP's daily source: exported by QuickSight into S3 and picked up there. Narrow — SGP's bronze pickup is live; the full RevFind-Source replacement (#5533) is still in flight.

Not started
Building
Testing
First live
▲
Rolling
All done
1 of 1 customers live
🎉 First live: SGP
AWS QuickSight exportRevFind

The matrix — bronze → silver → gold, grouped by what gates each customer

CustomerBronzeSilverGold
In flight in medallionat least one layer building, cutting over, or live11 customers · 18 cells moving
ARSAClarityFlow⇢ legacy
NexTech · FHIR✦
live ✓✦
CommonwealthRevFind
athena
cutover
internal✦
ECPRevFind
EDISnow
parallel
HillsboroEyeRevFind⇢ legacy
NextGen✦
live ✓✦
OCPRevFind
SFTP CSV✦
building✦
OneGIClarityFlowRevFind
ModMedECWgGastro✦
PrecisionHealthRevFind
EDI 835✦
SGPRevFind
QuickSight
StrideCareClarityFlowRevFind
HL7athena
parallel
SuncoastSkinRevFind
EDI 835✦
WomensCareClarityFlowgated · HL7 v2 fileslegacy HL7 pipeline built in parallel so the customer isn't blocked
HL7 v2✦
Gated on EDI / X12 ingestblocked until the X12 parser exists in medallion3 customers
ISpineRevFind
EDI
TheOregonClinicRevFind
EDI
USDHRevFind
EDI
Gated on vendor-host pullwe must reach out to their SFTP3 customers
EugeneGIClarityFlow
HL7ModMed
RIAClarityFlowRevFind
HL7MSN
SWOrofacialClarityFlow
HL7
Gated on direct SQL Serverlive OLTP reads, no file drop1 customer
EyeSouthRevFind
NextGenNexTech
Gated on HL7 v2 ingestfile-based HL7 parser needed3 customers
OrthoAllianceClarityFlowRevFind
HL7NextGen
PPRMClarityFlow
HL7NextGen
SeaRadClarityFlow
HL7
Gated on vendor APIFHIR connector needed2 customers
MannEyeClarityFlow
FHIR
NWDermClarityFlow
FHIR
Gated on SFTP delimited ingestthe workhorse path12 customers
CCPRevFind
Experity
FemwellRevFind
—
HeatonEyeClarityFlowRevFind
NextGen
parked
JewishBoardRevFind
—
MNEyeClarityFlowRevFind
NextGen
parked
MiddletownMedicalRevFind
ECW
Monarch
—
PAARevFind
Phygen
SouthSoundRadiologyClarityFlow
—
SunLifeClarityFlow
NextGenECW
USHPRevFind
NextGen
WildflowerClarityFlow
—
No detected ingest shapeexport shows no parseable format (e.g. TFOC/Centricity)1 customer
TFOCClarityFlowRevFind
Centricity
Bronze / Silver untouchedbuildingin progress / parallelcutovernew is source, old on standbylive ✓live / old retiredparkedstarted, then stalled — not in flightGoldinternalavailable internallyto customeravailable to the customer⌐ dashed = net-new (no legacy to retire)⇢ legacy = the new silver powers the legacy systemBronze shows one chip per source system (hover for source · state) — a customer can be live on one feed and untouched on another.