Medallion Data-Quality Stack

Adapter refactor

Silver used to read athena bronze from a dozen models, with every athena assumption implicit. Now ten src_* tables are the only thing core silver reads from the source, and a test fails if that changes. Stewardship, masters, labels and appeal stages moved into the shared medallion package, so Commonwealth, mvp and mvpdev share one copy instead of three drifting ones.

-- athena columns, read directly
from {{ ref('bronze_claim') }}
coalesce(nullif(cast(originalclaimid as varchar), ''),
         cast(claimid as varchar)) as root_claim_id

-- the adapter decides the family; core just reads it
from {{ medallion.staging_parquet('src_claim',
                                  'source_claim_id') }}
select context_id, source_claim_id, claim_sequence, root_claim_id

The Silver Adapter Refactor page walks through this PR with seven before/after examples.

Claim families

An encounter is a family of claims: an original and its rebills. athena's originalclaimid names the claim a rebill replaced, and that can itself be a rebill. Silver took it as the root, so a rebill of a rebill opened a second family and split one encounter in two. src_claim now walks the chain back to the first claim. A new test, claim_families_are_flat, fails if any claim's root is itself a rebill.

NPPES reader

The CMS NPI registry (NPPES) and the NUCC taxonomy code set are read where lapidary publishes them, under one shared root. Nothing is copied. Two package macros return the newest month: each NPI's primary taxonomy and deactivation status, read with the same rules pricing uses, and the newest NUCC release. When a project has no NPPES root set, both return an empty, typed result, so mvp still builds.

from {{ medallion.nppes_npi_current() }}
-- npi, entity_type, registry_name, primary_taxonomy_code,
-- is_deactivated, deactivation_date, practice address, …
from {{ medallion.nucc_taxonomy_current() }}
-- code, grouping, classification, specialization, display_name

-- the slot NPPES marks primary, else slot 1
coalesce(
  case when switch_1 = 'Y' then taxonomy_code_1 end,
  …
  case when switch_15 = 'Y' then taxonomy_code_15 end,
  taxonomy_code_1)

Providers

The stewardship ledger always wins. Where it's silent, NPPES fills the taxonomy and NUCC names the specialty, and taxonomy_source and specialty_source say which supplied each value. Mid-level means the taxonomy falls in one of the eight nonphysician classes pricing applies its rate by (MdClarity/MdClarity#6233). dim_provider gains nonphysician_classification and npi_status.

A provider whose ledger entry has an NPI but no taxonomy. The test fixtures use this case.

Commonwealth's ledger already has a taxonomy for every provider, so the NPPES fallback rarely fires there. What changes is specialty, mid-level, the class and NPI status, which were empty.

Observed providers

observed_provider is the queue of providers as athena records them, before stewardship. It gains four columns that say what the registry knows about the NPI athena recorded: npi_status, nppes_entity_type, nppes_name and nppes_taxonomy. A name that doesn't match nppes_name points to an NPI keyed against the wrong person.

npi_status is one shared macro, so provider and observed_provider classify an NPI the same way. It's blank when a build has no NPPES, so a project without it never calls every NPI unregistered.

Billing NPIs

The same four columns on observed_billing_entity, for the billing NPI each TIN bills under most often. NPPES publishes no EINs (every one reads <UNAVAIL>), so nothing here can confirm an NPI bills under its TIN. The registered legal name beside the medical group's name is the check a reviewer can make.

Master resolution

One TIN, whatever its spelling

A TIN observation was keyed on the raw string, so each spelling needed its own row in the hand-edited crosswalk, and a new spelling left encounters unresolved. medallion.tin_key keys a nine-digit TIN, with or without its dash, on its digits. Any other shape keeps its own spelling: an SSN-style value or free text is never coerced into a TIN.

Claims and lines sit with their encounter

An encounter's location decides whether it's shown and who can see it. Lines used to resolve location and provider from their own claim, so a line could carry a different location from its encounter, or none at all. Reports by location then didn't tie out, and lines from a hidden location showed up unlabeled inside visible encounters. Claims and lines now carry the encounter's location, rendering provider and billing entity, and a new test enforces it.

In production

No production line moves. Most likely athena moves a rebill's charges onto the new claim, so every line already sits on the claim that speaks for its encounter, while a replaced submission's claim positions keep their own department. In production this PR corrects about 2,100 to 3,600 claims; the line-level fix guards against data like the synthetic lake's.

Emerson's MdClarity/MdClarity#6254 maps each department to its own location. It touches different crosswalk files, so the two don't conflict, but it makes a line in a different location from its encounter more likely, which this PR prevents.