MDClarity · Pipeline explainer

How RIA's scheduled pipelines work

RIA runs two MDClarity products on two very different rhythms against its Abbadox radiology system: a continuous, every-minute ClarityFlow patient-access loop (benefits, estimates, GFE letters, bidirectional HL7) and a monthly RevFind underpayment-recovery pass over billing files from MSN LLC.

Runner: WORKER-1 Windows Task Scheduler → RunDriverJob.ps1 → jobs on prod-db-3
Export: dev-1 snapshot 2026-07-09 · Scheduled-task XML: MDC set · Structure & names only, no patient data

EHR / RIS

Abbadox

Bidirectional HL7 over SFTP

Scheduled tasks

continuous · nightly · monthly

HL7 poll interval

1 min

every minute, around the clock

Pipeline files in export

109

only ~6 scheduled; rest ad-hoc

external I/O (SFTP)

sub-job (pipeline)

SQL task

external script

task_type native pipeline task

▶ click sub-jobs to expand

Cadence · 01

When each pipeline runs

Three Windows Scheduled Tasks under \MDC\ on WORKER-1, at three wildly different frequencies. The HL7 loop is the surprise: its 00:01 is only a daily anchor — a PT1M repetition runs it every minute for a full day, so it effectively polls Abbadox around the clock.

00:00 06:00 12:00 18:00 24:00

HL7 Processing

continuous

every minute · IgnoreNew

Nightly letters

daily · 1×

19:01

RevFind

monthly · 15th

07:30

Pipeline 1 · continuous · every minute · ClarityFlow

The HL7 processing loop

The main patient-access loop, and RIA's near-real-time interface to Abbadox. The scheduled task HL7 - Processing - RIA fires every minute, all day (<Repetition><Interval>PT1M</Interval><Duration>P1D</Duration>), self-throttled by MultipleInstancesPolicy=IgnoreNew so a slow run never overlaps the next tick. Each tick runs three sequential actions: the pipeline, then two PowerShell scripts that finish and ship the outbound letters.

Action 1

Flow-HL7-Processing

RunDriverJob.ps1 · prod-db-3 · yields to nightly

the pipeline
Flow-HL7-Processing.yaml

downloadDownload inbound HL7

Abbadox SFTP /Outbound/ → {HL7Directory}

Pulls ADT / SIU / ORM / DFT / ORU messages; deletes remote on success. This is the poll that fires every minute.
load_hl7_filesLoadHL7Files

HL7 files → HL7Archive + HL7* staging tables

Allowed types: ADT (A01/03/04/08/09/10/28/31), SIU (S12–S26), DFT^P03, ORM^O01. Archives files to {HL7Archive}.
SQLHL7ProcessLog + HL7Delta

Opens a process-log row & computes the delta of changed messages to process
▶Flow-HL7-Transform— shape raw HL7 into Transform* tables · 15 SQL steps
- TruncateVisitDelta → reset delta
- TransformVisitIDMap · OrderSourceIDDelta → map source IDs / orders
- ImportHL7ADTCombined → consolidate ADT
- TransformPatient → TransformPatient
- ImportTransformPatientInsurance + merge into VisitDelta
- ImportTransformVisit + merge VisitDelta
- ImportTransformVisitInsurance + merge
- Recalculate Changed Bundles
- ImportTransformVisitCharge + merge
Output: the Transform* tables (patient / visit / insurance / charge) that the rest of the pipeline reads.
LoadPatientsLoadPatients

TransformPatient → Patient (upsert by PatientID)

RIA-specific: forces ElectronicCommunicationConsent = 1 ("Per RIA") and derives PreferredPhone from cell → home.
▶Flow-HL7-Processing-Load-Entities— Facility, Provider, InsuranceMap
- Load Facility
- Load Provider (SQL)
- Load InsuranceMap → maps source payer names to MDClarity insurances
▶Flow-HL7-Processing-LoadDimensionMembers— refresh 15 reporting dimensions
- LoadInsuranceDimensionMembers → Insurance / QuoteInsurance / EligibilityInsurance
- Provider · Facility · VisitType · VisitStatus · Time · State
- Diagnosis Code · Modifier · Place of Service · SourceInsurance
- RadioIsotopes Code Dimension ← radiology-specific
Metadata-only delete-and-replace into DimensionMember.
▶Flow-HL7-Processing-LoadVisits— the estimation core: bundle · benefits · price · letters · sync · 20+ steps
- Load TempBatchVisit · Augment Visit Delta (SQL) → stage prospective visits
- ReprocessChangedConfigurations → queue visits hit by config changes
- LoadVisitBatch → VisitBatch / QuoteBatch
- AutoApplyBundlesToVisitBatch → swap in bundle charges, make quotes
- CheckBenefits → external eligibility API (billable)
- Re-Check Benefits · AutoApplyBenefits → apply copay/deductible/coinsurance
- CalculateVisits → compute patient responsibility
- LogVisitChangeEventsBatch → diff batch vs prod
- CheckVisitGFEComplianceBatch → GFE due-date / resend status
- Update Patient Email for Commercial Insurance · Load-CustomProperty-CoverageStatus
- AutoSendLetters → estimate/GFE letters for batch visits
- AutoSendLettersForFutureVisits → upcoming visits
- SyncBatch → promote *Batch tables to production Visit/Quote/…
- ExportBatchVisitEligibility · SearchForInsurance
- Generate-Report · Flow-HL7-Processing-LoadVisitFact · Update-Dashboard-Tables
The heart of ClarityFlow: turns staged visits into priced, benefit-applied quotes, sends the patient letters, and promotes everything from the *Batch staging tables into production.
SQLUpdate HL7ProcessLog

Closes out the process-log row for this run
workqueue_assignmentWorkqueueAssignment

Populate user workqueues from WorkqueueAssignmentCriteria
send_hl7SendHL7Letters

Unsent letters → ORU^R01 .txt + PDF written to D:\ClientData\RIA\ORU

Receiving app Abbadox / facility RIA. These files are then finished & shipped by Actions 2–3 below.
execute_processGenerate and send ORMs

D:\ClientData\RIA\OutboundORM\py.bat → generateORMs.py

Independent outbound path (see Outbound ORM generator): coverage-verified ORM^O01 orders → Abbadox /Inbound.

Action 2

Convert-Hl7PdfLinkToEmbeddedWithOrderID.ps1

post-process the ORU letters

PowerShell · D:\ClientData\RIA\

scriptEmbed PDF + rewrite OrderID

ORU\ → ORU\Processed\ (archives originals → Processed/Archive/Error)

For each ORU: reads the PDF linked in OBX.5, base64-embeds it as encapsulated data, and replaces the internal SourceID in PV1.19 with the real Abbadox OrderID list (looked up in TransformOrderSourceIDMap).

Action 3

winscpExport.ps1

ship the finished letters to Abbadox

PowerShell (WinSCP) · D:\ClientData\RIA\

scriptSFTP upload ORU letters

ORU\Processed\* → Abbadox SFTP /Inbound

Successful uploads move to ORU\Success\; failures logged to sftp_upload_errors.txt.

Pipeline 2 · daily @ 19:01 · ClarityFlow

Nightly future-visit letters

A single-task job that runs once a day at 19:01 to send estimate/GFE letters for upcoming appointments — the forward-looking counterpart to the continuous loop, which explicitly yields to it. (One trigger, no repetition; the PT8H on it is a max-runtime cap, not an interval.)

Nightly

Flow-HL7-Processing-AutoSendLettersForFutureVisits

HL7 - Processing - RIA - Nightly · daily 19:01 · 8h max runtime

single task

auto_send_letters_for_future_visitsAutoSendLettersForFutureVisits

Queries live future Visits (ScheduledTime > now) → PatientLetter + send events

Sends only when the visit is calculated, benefits are applied, and the patient has consent + valid contact. Shares the AutoSendLetters core, but reads live visits instead of the batch table.

Pipeline 3 · monthly, 15th @ 07:30 · RevFind

Monthly RevFind processing

The underpayment-recovery side, entirely separate from the HL7 flow. Despite the job name (Daily-Processing-RevFind), the trigger is monthly on the 15th — one calendar trigger, no repetition. Ingests billing/remit files from MSN LLC (a radiology service bureau), rebuilds the account substrate, and scores accounts against ~40 payer contracts to flag underpayments.

Monthly

Daily-Processing-RevFind

Monthly - Processing - RIA · day 15 @ 07:30

4 sub-jobs

▶Process-MSN-Files-RevFind— download & import MSN billing files
- Download ← mdcssh SFTP /MSN Files/ → {SourceDataFolderRevFind}
- Download ← MSN LLC vendor SFTP (msnftp) /
- ExecuteJobForFiles → Load-MSN-ProcessDataFile-RevFind (per *.csv)
- process_file → scan structure, load Import*, merge into Source* tables
▶Load-Account-Transform-Tables— 8 SQL transforms of source billing data
- TransformData · AccountDelta · TransformAccountMap
- TransformCharge · TransformAccount · TransformTransaction
- TransformChargeTransactionMeasure (×2)
▶Load-Entities-RevFind— Facility, Provider
- Load Facility
- Load Provider (SQL)
▶Load-Account-Data— build accounts & score them
- LoadPatients
- Load ChargeMeasures (SQL)
- LoadAccountData → merge Account / AccountCharge / AccountRemit / AccountInsurance
- Load-Dimension-Members · Load-Custom-Properties (sub-jobs)
- Score-Accounts → ~40 per-payer scoring rules (Aetna, BCBS, Humana, Cigna, Medicare + MA variants ECFOI/MICTIC/RAOC/IMI)
- Load AccountDetail · Load AccountChargeDetail · Load Recovery Report (SQL)
Output: scored accounts + a recovery report highlighting underpaid claims by payer/contract.

Reference · Interfaces

Who RIA talks to

Every external hop is SFTP. Abbadox is bidirectional; the RevFind side is inbound-only from MSN LLC.

Endpoint	Direction	Connection	Carries	Used by
Abbadox `/Outbound`	inbound ⬇	`RIA_ProdMirror-abbadox-sftp`	ADT / SIU / ORM / DFT / ORU HL7	HL7 loop · Download
Abbadox `/Inbound`	outbound ⬆	script-managed SFTP	ORU letters (PDF-embedded) + ORM^O01 orders	winscpExport.ps1 · generateORMs.py
MSN LLC `/MSN Files/`	inbound ⬇	`RIA_ProdMirror-mdcssh-sftp`	Billing / remit CSVs	Monthly · Process-MSN
MSN LLC (vendor host)	inbound ⬇	`RIA_ProdMirror-msnftp-sftp`	Billing / remit CSVs	Monthly · Process-MSN

Reference · Scripts

Code outside the pipeline framework

The scheduled tasks and one pipeline step shell out to four scripts on WORKER-1. Three are generic/known; the fourth is a recent, self-contained outbound generator.

Script	Invoked by	What it does
`RunDriverJob.ps1`	all 3 tasks	Generic job runner — starts a named pipeline job on a server, logs, emails on failure, supports `yieldToJobs`.
`Convert-Hl7PdfLinkToEmbeddedWithOrderID.ps1`	HL7 loop · Action 2	Embeds linked PDFs into ORU `OBX.5` as base64 and rewrites `PV1.19` SourceID → OrderID.
`winscpExport.ps1`	HL7 loop · Action 3	SFTPs finished ORU letters to Abbadox `/Inbound`; moves successes to `Success\`.
`OutboundORM\generateORMs.py` (+ py.bat)	HL7 loop · Flow-HL7-Processing final step	Generates coverage-verified ORM^O01 orders from the DB and SFTPs them to Abbadox `/Inbound`.

generateORMs.py Outbound ORM generator

The final step of the HL7 loop — a standalone uv-run Python script (its own bundled .venv), sitting outside the normal task framework. It closes the loop back to Abbadox after coverage is verified:

Query RIA on prod-db-3 for future visits where PrimaryCoverageStatus = Active but ActiveCoverageUpdate ≠ Sent — i.e. coverage just verified.
Pull the original inbound HL7 from HL7Archive, extract the ORC segment, set ORC-25 (order status) = "Verified", wrap in a fresh MSH … ORM^O01.
Write one .hl7 per order to Generated\; mark the visit Sent in CustomPropertyMap (transactional).
SFTP the files to Abbadox /Inbound; move sent files to Sent\.

Reference · Glossary

Task-type glossary

The native pipeline tasks used above, per docs/tasks. Fir = HL7 / ClarityFlow, coral = RevFind.

download

Parallel SFTP/FTP pull to a local dir; optional archive/delete on the remote. No HTTP/S3.

load_hl7_files

Parse a folder of HL7 files into HL7Archive + type-specific staging tables; per-file, batch, import-only.

LoadPatients

Upsert Patient from a query over TransformPatient.

LoadDimensionMembers

Delete-and-replace the DimensionMember metadata cache from core entity tables.

LoadInsuranceDimensionMembers

Insurance-only subset: Insurance / QuoteInsurance (has ContractMap) / EligibilityInsurance (has ExternalAPI).

ReprocessChangedConfigurations

Queue future visits & accounts affected by changed configs (bundles, benefit rules, letter configs) for recalc.

AutoApplyBundlesToVisitBatch

Match each batch visit to a charge Bundle and swap in its charges; generate quotes per service type.

CheckBenefits

Batch eligibility — call payer APIs (e.g. Availity) for coverage/benefits; writes EligibilityTxn. Billable.

AutoBenefits

Apply copay/deductible/coinsurance/max-OOP to batch visits from eligibility txns via AutoBenefitsRule.

LogVisitChangeEventsBatch

Diff each batch visit against its production counterpart; log per-visit change reasons (truncated each run).

CheckVisitGFEComplianceBatch

Compute Good Faith Estimate due dates & whether a compliant letter was sent / needs resend.

AutoSendLetters

Generate & send estimate/GFE letters for batch visits per AutoSendLetterConfig (consent + contact required).

AutoSendLettersForFutureVisits

Same letter engine, but over live future visits instead of the batch table.

SyncBatch

Promote *Batch staging tables to production (delete-then-insert per entity, each in its own txn).

SendHL7Letters

Export unsent letters as HL7 ORU^R01 .txt + PDF to disk (file-based integration).

WorkqueueAssignment

Fill user workqueues from WorkqueueAssignmentCriteria; optional auto-unassign of non-matches.

UpdateDashboardTables

Rebuild pre-aggregated Dashboard_* rollup tables so dashboards read fast.

ProcessDataFile

Scan a delimited file's structure, load into Import{Base}, then merge into Source{Base}.

LoadAccountData

Import accounts/charges/remits/insurance via four SQL queries; merge into the four core account tables.

Built from the dev-1 export snapshot (2026-07-09), the MDC scheduled-task XML set, the docs/tasks docs, and the WORKER-1 client scripts (pulled via SSM). Pipeline structure & names only — no patient data. Job DAGs reflect the pipeline definitions as exported; ad-hoc / -Copy / -Test jobs (103 of 109 files) are not scheduled and are omitted.