Medallion Architecture · Migration Campaign

Medallion architecture progress — laying our new data foundation

The road is largely built. First, the platform foundations already laid; then where each customer stands. The customer counts look early because the hard 0→1 — the foundations — is done: with the patterns and per-customer isolation in place, new customers now roll fast (three onboarded in ~1 PR each via shared libraries).
7
platform foundations laid
31
customers tracked
15
in flight in medallion
4
need EDI / X12 (parser in progress)
Last winARSA's NexTech FHIR pipeline wired bronze → silver on the per-customer data plane · built on a new reusable fhir-common connector
Tracking9 S3-landing customers onboarding at once on the rebuilt filesystem-common core
Ingest denominators & product mix are real (from the pipeline export). Medallion state is curated in medallion-status.yaml — as of 2026-07-16.

Foundations laid7 of 11 platform wins — the 0→1 the customer view hides

  1. Nov 2025S3 landing buckets
  2. Dec 2025ECS platform
  3. Jan 2026Dagster hybrid cloud
  4. Mar 2026OneGI: full medallion + write-back + checks
  5. May 20263 customers onboarded in ~1 PR (shared libs)
  6. Jun 202623 customer data stacks — fleet isolation
  7. Jun 2026filesystem-common: 9 customers' bronze in one PR
  8. Jun 2026ARSA: first FHIR-API customer (bronze + silver, in testing)
✓ DoneDec 2025
ECS container platform
The Fargate platform every Dagster agent, pipeline run, and API runs on.
Infra #243
✓ DoneJan 2026
Dagster hybrid-cloud orchestration
Dagster Cloud control plane + our-AWS agents; ingest agent isolated in the SFTP VPC.
Infra #286 / #301
✓ EstablishedMar 2026
Silver → SQL Server write-back
Medallion silver powers the legacy gold / RevFind — atomic rebuild + rename-swap. Reused by 4 customers.
MdC #5282 / #5389
⚠ Needs repetitionMar 2026
Data validation (asset checks)
Catches silent bronze truncation + schema drift. Proven on OneGI; not yet on the athena pipelines.
MdC #5282
✓ DoneMar 2026
Managed SFTP (AWS Transfer Family)
Cognito-auth SFTP, jailed per customer/source into S3; replaces Bitvise.
Infra #330
✓ DoneMar 2026
SFTP → S3 retention shim
Inbound files replicated to S3 with retention — the legacy system deletes on read; we keep them.
Manual op · Mar 25 (no PR)
✓ DoneJun 2026
Customer-isolated HIPAA data lake
Per-customer S3 + own CMK + scoped task role + Object-Lock + cross-region replica. 23 customers bootstrapped.
Infra #418 / #437
◷ In progressJun 2026
Reusable FHIR connector (dlt)
Vendor-agnostic FHIR ingest — type-incremental + reference-following, pluggable OAuth2, rate-limited, count-only logging. ARSA/NexTech is the first consumer; any FHIR-API customer reuses it.
MdC #5564
◷ In progressJun 2026
filesystem-common bronze core
Unit-tested CSV ingestion core + Dagster FilesystemSourceComponent — stands up bronze for 9 S3-landing customers in one PR.
MdC #5581
✓ Done
Gold available in Omni
Gold data queryable in Omni — Athena + Glue catalogs & connectors, with cross-account access into each customer's bucket wired for Omni.
◷ In progress
EDI / X12 parser
First-class 835/837 parsing in medallion — the gate for the EDI-blocked customers. Preparing for an internal security review.

The ingest surface — format · protocol · source

File formatwhat the payload is
CSV27
Pipe-delimited7
HL7 v26
EDI / X124
Multi-char delimited4
No file · share / API / direct SQL2
Tab-delimited2
Protocol / transporthow the data arrives
SFTP (push to us)21
Vendor API4
Vendor-host pull4
Snowflake share3
Direct SQL Server1
Source systemwhich EHR / vendor feed — exact per-source bronze state
HL7 interface7
NextGen7
EDI (835/837)4
eClinicalWorks3
athenahealth2
FHIR2
ModMed2
Centricity1
Experity1
gGastro1
MSN1
NexTech · FHIR1
NextTech1
Phygen1
QuickSight1
Snowflake1
livecutoverbuilding / paralleluntouchedBar length = customers with that feed (real, from the export). Fill = bronze migration progress. The three axes are independent — one feed reads on all three.
Source × deliverywhere each feed actually lands — a source in two columns arrives two ways
Source systemSnowflake shareSFTP filesDirect SQL ServerVendor APIAll
HL7 interface·7··7
NextGen·7··7
EDI (835/837)·4··4
eClinicalWorks·3··3
athenahealth2···2
FHIR···22
ModMed·2··2
Centricity·1··1
Experity·1··1
gGastro··1·1
MSN·1··1
NexTech · FHIR···11
NextTech··1·1
Phygen·1··1
QuickSight···11
Snowflake1···1

Customers per source × delivery channel, tinted by bronze progress. eClinicalWorks spans SFTP files and direct SQL Server (Partnercare) — the same feed, two transports.

Not startedBuildingTestingFirst liveRollingAll done

Bronze — ingest capabilities

SFTP delimited files
Rolling

Delimited files — CSV / pipe / tab, plus the rare ^|^ multichar — parsed via ProcessDataFile. Usually pushed to our SFTP, sometimes pulled from a vendor host (a separate axis — see vendor pull). The workhorse and widest blast radius: it fires for any customer with a delimited feed, even a secondary one.

Not started
Building
Testing
First live
Rolling
All done
3 of 27 customers live
🎉 First live: OneGI
AWS QuickSight export · Experity (CCP) · FHIR API · HL7 v2 messages…ClarityFlowRevFind
HL7 v2 files
Building

File-based HL7 (ADT / SIU / DFT / ORM) — no MLLP listeners. Needs a parser + message-type mapping before first live.

Not started
Building
Testing
First live
Rolling
All done
0 of 7 customers live
HL7 v2 messages · MSN LLC (radiology service bureau) · ModMed · NextGen…ClarityFlowRevFind
Frequent HL7 v2 processing (Flow)
Not started

ClarityFlow consumes HL7 continuously through the day — not the daily batch cadence the rest of medallion runs at. Bronze needs a frequent small-batch HL7 mode before Flow's HL7 customers can move. ClarityFlow-only: RevFind's daily cadence is unaffected.

Not started
Building
Testing
First live
Rolling
All done
7 customers waiting · not yet built
⚠ Risk: every ClarityFlow HL7 customer is held until this exists — not yet started
HL7 v2 messages · MSN LLC (radiology service bureau) · ModMed · NextGen…ClarityFlowRevFind
EDI / X12 (835 / 837)
Building

X12 claims & remits. Some pipelines use the in-repo EDIParser task; iSpine routes parsing through an off-repo PowerShell script (insert_edi.ps1). Medallion needs a first-class X12 parser either way.

Not started
Building
Testing
First live
Rolling
All done
0 of 4 customers live
⛔ Blocks ECP, ISpine, TheOregonClinic, USDH
Snowflake (varies) · X12 EDI claims / remitsRevFind
Vendor-host SFTP pull
Not started

We reach out to *their* server (e.g. MSNFTP.MSNLLC.NET) instead of receiving a push — a transport-direction signal, independent of file format. Needs outbound creds + scheduling.

Not started
Building
Testing
First live
Rolling
All done
4 customers waiting · not yet built
HL7 v2 messages · MSN LLC (radiology service bureau) · ModMed · X12 EDI claims / remitsClarityFlowRevFind
Snowflake data share
Rolling

athenahealth (and other Snowflake-share sources) read straight from a Snowflake share — no inbound files to receive or parse, though bronze still writes parquet like every source. Cloud-native, the smoothest path. NB: 'Athena' here is athenahealth, NOT AWS Athena.

Not started
Building
Testing
First live
Rolling
All done
3 of 3 customers live
🎉 First live: Commonwealth
HL7 v2 messages · Snowflake (varies) · X12 EDI claims / remits · athenahealthClarityFlowRevFind
Vendor API (FHIR connectors)
Testing

First-class FHIR API connectors — API client & token handling rather than file ingest. First customer is ARSA (medallion-native, no legacy export). ModMed's off-repo Flow-MM API (OneGI) is invisible to the export and not counted here.

Not started
Building
Testing
First live
Rolling
All done
3 customers in testing · not yet live
FHIR APIClarityFlow
Direct SQL Server queries
First live

Live OLTP reads against the EHR's SQL Server — no file drop at all. Spans NexTech (EyeSouth) and eClinicalWorks-over-SQL-Server (Partnercare); new-stack feeds the export can't see (OneGI's gGastro) are added via a curated override. Needs a network path + read credentials.

Not started
Building
Testing
First live
Rolling
All done
1 of 2 customers live
🎉 First live: OneGI
ModMed · NexTech · NextGen · eClinicalWorks (ECW)ClarityFlowRevFind
QuickSight → S3 export
First live

SGP's daily source: exported by QuickSight into S3 and picked up there. Narrow — SGP's bronze pickup is live; the full RevFind-Source replacement (#5533) is still in flight.

Not started
Building
Testing
First live
Rolling
All done
1 of 1 customers live
🎉 First live: SGP
AWS QuickSight exportRevFind

The matrix — bronze → silver → gold, grouped by what gates each customer

CustomerBronzeSilverGold
In flight in medallionat least one layer building, cutting over, or live15 customers · 22 cells moving
ARSAClarityFlow
NexTech · FHIR
building
CCPRevFind
Experity
CommonwealthRevFind
athena
cutover
internal
ECPRevFind
EDISnow
parallel
HeatonEyeClarityFlowRevFind
NextGen
parallel
JewishBoardRevFind
MNEyeClarityFlowRevFind
NextGen
parallel
MiddletownMedicalRevFind
ECW
OneGIClarityFlowRevFind
ModMedECWgGastro
PAARevFind
Phygen
SGPRevFind
QuickSight
SouthSoundRadiologyClarityFlow
StrideCareClarityFlowRevFind
HL7athena
parallel
SunLifeClarityFlow
NextGenECW
USHPRevFind
NextGen
Gated on EDI / X12 ingestblocked until the X12 parser exists in medallion3 customers
ISpineRevFind
EDI
TheOregonClinicRevFind
EDI
USDHRevFind
EDI
Gated on vendor-host pullwe must reach out to their SFTP3 customers
EugeneGIClarityFlow
HL7ModMed
RIAClarityFlowRevFind
HL7MSN
SWOrofacialClarityFlow
HL7
Gated on direct SQL Serverlive OLTP reads, no file drop1 customer
EyeSouthRevFind
NextGenNexTech
Gated on HL7 v2 ingestfile-based HL7 parser needed3 customers
OrthoAllianceClarityFlowRevFind
HL7NextGen
PPRMClarityFlow
HL7NextGen
SeaRadClarityFlow
HL7
Gated on vendor APIFHIR connector needed2 customers
MannEyeClarityFlow
FHIR
NWDermClarityFlow
FHIR
Gated on SFTP delimited ingestthe workhorse path3 customers
FemwellRevFind
Monarch
WildflowerClarityFlow
No detected ingest shapeexport shows no parseable format (e.g. TFOC/Centricity)1 customer
TFOCClarityFlowRevFind
Centricity
Bronze / Silver untouchedbuildingin progress / parallelcutovernew is source, old on standbylive ✓live / old retiredGoldinternalavailable internallyto customeravailable to the customer⌐ dashed = net-new (no legacy to retire)Bronze shows one chip per source system (hover for source · state) — a customer can be live on one feed and untouched on another.