Medallion Architecture · Migration Campaign

Where the campaign stands

The road is largely built. First, the platform foundations already laid; then where each customer stands. The customer counts look early because the hard 0→1 — the foundations — is done: with the patterns and per-customer isolation in place, new customers now roll fast (three onboarded in ~1 PR each via shared libraries).
9
platform foundations laid
30
customers tracked
15
in flight in medallion
3
need EDI / X12 (not started)
Last winARSA's NexTech FHIR pipeline wired bronze → silver on the per-customer data plane · built on a new reusable fhir-common connector
Tracking9 S3-landing customers onboarding at once on the rebuilt filesystem-common core
Ingest denominators & product mix are real (from the pipeline export). Medallion state is curated in medallion-status.yaml — as of 2026-06-13.

Foundations laid 6 of 9 platform wins — the 0→1 the customer view hides

  1. Nov 2025 S3 landing buckets
  2. Dec 2025 ECS platform
  3. Jan 2026 Dagster hybrid cloud
  4. Mar 2026 OneGI: full medallion + write-back + checks
  5. May 2026 3 customers onboarded in ~1 PR (shared libs)
  6. Jun 2026 23 customer data stacks — fleet isolation
  7. Jun 2026 filesystem-common: 9 customers' bronze in one PR
  8. Jun 2026 ARSA: first FHIR-API customer (bronze + silver, in testing)
✓ Done Dec 2025
ECS container platform
The Fargate platform every Dagster agent, pipeline run, and API runs on.
Infra #243
✓ Done Jan 2026
Dagster hybrid-cloud orchestration
Dagster Cloud control plane + our-AWS agents; ingest agent isolated in the SFTP VPC.
Infra #286 / #301
✓ Established Mar 2026
Silver → SQL Server write-back
Medallion silver powers the legacy gold / RevFind — atomic rebuild + rename-swap. Reused by 4 customers.
MdC #5282 / #5389
⚠ Needs repetition Mar 2026
Data validation (asset checks)
Catches silent bronze truncation + schema drift. Proven on OneGI; not yet on the athena pipelines.
MdC #5282
✓ Done Mar 2026
Managed SFTP (AWS Transfer Family)
Cognito-auth SFTP, jailed per customer/source into S3; replaces Bitvise.
Infra #330
✓ Done Mar 2026
SFTP → S3 retention shim
Inbound files replicated to S3 with retention — the legacy system deletes on read; we keep them.
Manual op · Mar 25 (no PR)
✓ Done Jun 2026
Customer-isolated HIPAA data lake
Per-customer S3 + own CMK + scoped task role + Object-Lock + cross-region replica. 23 customers bootstrapped.
Infra #418 / #437
◷ In progress Jun 2026
Reusable FHIR R4 connector (dlt)
Vendor-agnostic FHIR ingest — type-incremental + reference-following, pluggable OAuth2, rate-limited, count-only logging. ARSA/NexTech is the first consumer; any FHIR-API customer reuses it.
MdC #5564
◷ In progress Jun 2026
filesystem-common bronze core
Unit-tested CSV ingestion core + Dagster FilesystemSourceComponent — stands up bronze for 9 S3-landing customers in one PR.
MdC #5581
Not started Building Testing First live Rolling All done

Bronze — ingest capabilities

SFTP delimited files
Rolling

Delimited files — CSV / pipe / tab, plus the rare ^|^ multichar — parsed via ProcessDataFile. Usually pushed to our SFTP, sometimes pulled from a vendor host (a separate axis — see vendor pull). The workhorse and widest blast radius: it fires for any customer with a delimited feed, even a secondary one.

Not started
Building
Testing
First live
Rolling
All done
3 of 26 customers live
🎉 First live: OneGI
AWS QuickSight export · Experity (CCP) · FHIR API · HL7 v2 messages… CLRE
HL7 v2 files
Not started

File-based HL7 (ADT / SIU / DFT / ORM) — no MLLP listeners. Needs a parser + message-type mapping before first live.

Not started
Building
Testing
First live
Rolling
All done
7 customers waiting · not yet built
HL7 v2 messages · MSN LLC (radiology service bureau) · ModMed · NextGen… CLRE
Snowflake data share
Rolling

athenahealth (and other Snowflake-share sources) read straight from a Snowflake share — no inbound files to receive or parse, though bronze still writes parquet like every source. Cloud-native, the smoothest path. NB: 'Athena' here is athenahealth, NOT AWS Athena.

Not started
Building
Testing
First live
Rolling
All done
3 of 3 customers live
🎉 First live: Commonwealth
HL7 v2 messages · Snowflake (varies) · X12 EDI claims / remits · athenahealth CLRE
EDI / X12 (835 / 837)
Not started

X12 claims & remits. Some pipelines use the in-repo EDIParser task; iSpine routes parsing through an off-repo PowerShell script (insert_edi.ps1). Medallion needs a first-class X12 parser either way.

Not started
Building
Testing
First live
Rolling
All done
3 customers waiting · not yet built
⛔ Blocks ECP, ISpine, USDH
Snowflake (varies) · X12 EDI claims / remits RE
Vendor-host SFTP pull
Not started

We reach out to *their* server (e.g. MSNFTP.MSNLLC.NET) instead of receiving a push — a transport-direction signal, independent of file format. Needs outbound creds + scheduling.

Not started
Building
Testing
First live
Rolling
All done
3 customers waiting · not yet built
HL7 v2 messages · MSN LLC (radiology service bureau) · ModMed CLRE
Direct SQL Server queries
First live

Live OLTP reads against the EHR's SQL Server — no file drop at all. Spans NexTech (EyeSouth) and eClinicalWorks-over-SQL-Server (Partnercare); new-stack feeds the export can't see (OneGI's gGastro) are added via a curated override. Needs a network path + read credentials.

Not started
Building
Testing
First live
Rolling
All done
1 of 3 customers live
🎉 First live: OneGI
ModMed · NexTech · NextGen · eClinicalWorks (ECW) CLRE
Vendor API (FHIR connectors)
Testing

First-class FHIR API connectors — API client & token handling rather than file ingest. First customer is ARSA (medallion-native, no legacy export). ModMed's off-repo Flow-MM API (OneGI) is invisible to the export and not counted here.

Not started
Building
Testing
First live
Rolling
All done
3 customers in testing · not yet live
FHIR API CL
QuickSight → S3 export
First live

SGP's daily source: exported by QuickSight into S3 and picked up there. Narrow — SGP's bronze pickup is live; the full RevFind-Source replacement (#5533) is still in flight.

Not started
Building
Testing
First live
Rolling
All done
1 of 1 customers live
🎉 First live: SGP
AWS QuickSight export RE

The matrix — bronze → silver → gold, grouped by what gates each customer

Customer BronzeSilverGold
In flight in medallion at least one layer building, cutting over, or live 15 customers · 22 cells moving
ARSA CL
NexTech
building
CCP RE
Experity
Commonwealth RE
athena
cutover
building
ECP RE
EDI Snow
parallel
HeatonEye CLRE
NextGen
parallel
JewishBoard RE
MNEye CLRE
NextGen
parallel
MiddletownMedical RE
ECW
OneGI CLRE
ModMed ECW gGastro
PAA RE
Phygen
SGP RE
QuickSight
SouthSoundRadiology CL
StrideCare CLRE
HL7 athena
parallel
SunLife CL
NextGen ECW
USHP RE
NextGen
Gated on EDI / X12 ingest blocked until the X12 parser exists in medallion 2 customers
ISpine RE
EDI
USDH RE
EDI
Gated on vendor-host pull we must reach out to their SFTP 3 customers
EugeneGI CL
HL7 ModMed
RIA CLRE
HL7 MSN
SWOrofacial CL
HL7
Gated on direct SQL Server live OLTP reads, no file drop 2 customers
EyeSouth RE
NextGen NexTech
Partnercare RE
ECW
Gated on HL7 v2 ingest file-based HL7 parser needed 3 customers
OrthoAlliance CLRE
HL7 NextGen
PPRM CL
HL7 NextGen
SeaRad CL
HL7
Gated on vendor API FHIR connector needed 2 customers
MannEye CL
FHIR
NWDerm CL
FHIR
Gated on SFTP delimited ingest the workhorse path 2 customers
Monarch
Wildflower CL
No detected ingest shape export shows no parseable format (e.g. TFOC/Centricity) 1 customer
TFOC CLRE
Centricity
Bronze / Silver  untouched buildingin progress / parallel cutovernew is source, old on standby live ✓live / old retired Gold internalavailable internally to customeravailable to the customer ⌐ dashed = net-new (no legacy to retire) Bronze shows one chip per source system (hover for source · state) — a customer can be live on one feed and untouched on another.