Legacy Pipeline Atlas worker-1snapshot 2026-07-15

HL7 real-time feed · ClarityFlow pipeline

EugeneGI · Flow-MM-HL7-Processing

Overall Processing job for HL7 pipeline

pipeline EugeneGI/Flow-MM-HL7-Processing · db EugeneGI_ProdMirror · updated 2025-12-01
classification: flow · 10 Flow markers

trigger

How it's scheduled

Real-time loop — every 30 min (~48×/day)

New appointments land continuously on the HL7 real-time feed. Front-of-house work is time-boxed — the estimate and letter have to reach the patient before the visit — so the pipeline polls every 30 minutes to check eligibility, price the good-faith estimate, and send the patient letter within minutes of an appointment being booked, rather than waiting for an overnight batch. It stands down while HL7 - Processing - EugeneGI - Nightly runs (the heavier nightly pass).

Scheduled taskCadenceServerStateCoordination
\MDC\HL7 - Processing - EugeneGIevery 30 min (from 00:01)prod-db-1enabledyields to HL7 - Processing - EugeneGI - Nightly

shape

Against the canonical Flow spine

Every Flow pipeline is a variation on one shape. Lit stages are the ones this pipeline runs; the labels match the capability lens.

Ingest
HL7 real-time feed
→
Eligibility & benefits
EligibilityBenefitsDiscovery
→
Estimate
BundlesEstimate
→
Letters
HL7
→
Work queue & sync
Work queueDashboardsSync

computed concepts · from the SQL

What it computes

Business concepts mined from this pipeline's SQL text — the meaning behind the typed steps. Each links to the concept lens (who else computes it).

Coverage statusBundlesCPT / procedure logic

task tree

What the pipeline runs

Tasks in order, each with what it does — typed stepsfrom the MDClarity task docs (click to open the reference), sub-jobsyou can expand inline (▶) to reveal their own tasks, and SQL steps summarized by Gemini (✨). Every step shows the tables it writes/reads and which earlier step it feeds from(sqlglot lineage).

flow

Flow-MM-HL7-Processing

12 tasks · EugeneGI
sub-job / Flow action SQL / native step▶ expand a sub-job inline · ↗ open it as its own page · a typed step opens its doc
  1. DownloadDownload
    Downloads files from remote FTP or SFTP servers to local directories with support for archiving and parallel transfers.
    server=EugeneGI_ProdMirror-intva01-sftpremote directory=/local directory={HL7Download}
  2. Clean and Move HL7ExecuteProcess
    command="D:\ClientData\EugeneGI\py.bat"
  3. LoadHL7FilesLoadHL7Files
    Parses and imports patient demographics, visits, and charges from HL7 message files located in a specified directory.
    message folder path={HL7Directory}message processed folder path={HL7Archive}add timestamp to file name
  4. HL7ProcessLog + HL7Deltasqlflow_mm_hl7_processing__hl7processlog_hl7delta.sql
    ✨ Identifies and stages new HL7 messages from the archive for processing while logging the start of the integration task.
    writes HL7ADTBatchHL7DeltaHL7ProcessLogHL7ProcessLogArchiveKeysHL7SIUBatch · reads HL7ADTBatchHL7ArchiveHL7SIUBatchTransformPatientTransformVisit
  5. ▶Flow-MM-HL7-Processing-Load-Entitiessub-job · 4↗Load Facility, Provider, InsuranceMap
    1. Load FacilityLoadEntities
      writes Facility · reads FacilityHL7VisitMMFacilityMap
      entity type=Facilitycustom querykey prefix=F
    2. Load Providersqlflow_mm_hl7_processing_load_entities__load_provider.sqldisabled
      ✨ Loads provider information from HL7 order messages into the master provider table using the most recent record available.
      writes Provider · reads HL7Order
    3. Load ProviderMergeSql
      writes Provider · reads HL7VisitMMProviderMapProvider
      merge data source type=ThisDatabasetable name=Providermerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
    4. Load InsuranceMapLoadEntities
      writes InsuranceMap · reads HL7PatientHL7VisitInsuranceMap
      entity type=InsuranceMapcustom querykey prefix=IM
  6. ▶Flow-MM-HL7-Transformsub-job · 9↗HL7 Transformation Layer
    1. TruncateVisitDeltasqlflow_mm_hl7_transform__truncatevisitdelta.sql
      ✨ Clears the visit delta table to prepare for a fresh set of changed records in the current pipeline run.
      writes VisitDelta
    2. ImportHL7ADTCombinedsqlflow_mm_hl7_transform__importhl7adtcombined.sql
      ✨ Consolidates patient and visit information from various HL7 message types into a unified staging table for further transformation.
      writes ImportHL7ADTCombined
    3. TransformPatientsqlflow_mm_hl7_transform__transformpatient.sql
      ✨ Synchronizes patient demographic information by updating existing records or inserting new ones based on the latest HL7 data.
      writes TransformPatient · reads ImportHL7ADTCombined
      ↳ feeds from ImportHL7ADTCombined
    4. ImportTransformPatientInsurancesqlflow_mm_hl7_transform__importtransformpatientinsurance.sql
      ✨ Extracts and stages the most recent insurance details for patients from combined HL7 message data.
      writes ImportTransformPatientInsurance · reads ImportHL7ADTCombinedInsuranceMap
      ↳ feeds from ImportHL7ADTCombined
    5. TransformPatientInsurance+VisitDeltasqlflow_mm_hl7_transform__transformpatientinsurance_visitdelta.sql
      ✨ Updates patient insurance records and flags associated future visits in the delta table for reprocessing.
      writes TransformPatientInsuranceVisitDelta · reads InsuranceChangesTransformVisitimporttransformpatientinsurance
      ↳ feeds from ImportTransformPatientInsurance, MergeTransformVisit+VisitDelta
    6. ImportTransformVisitsqlflow_mm_hl7_transform__importtransformvisit.sql
      ✨ Builds the visit import table by extracting the latest appointment data from HL7 messages while mapping facilities and providers.
      writes ImportTransformVisit · reads FacilityHL7DeltaHL7VisitMMFacilityMapMMProviderMap
    7. MergeTransformVisit+VisitDeltasqlflow_mm_hl7_transform__mergetransformvisit_visitdelta.sql
      ✨ Updates the main visit transformation table with imported data and flags new visits in the delta tracking table.
      writes TransformVisitVisitDelta · reads ImportTransformVisitVisitDelta
      ↳ feeds from ImportTransformVisit, TransformPatientInsurance+VisitDelta, TruncateVisitDelta
    8. ImportTransformVisitInsurancesqlflow_mm_hl7_transform__importtransformvisitinsurance.sql
      ✨ Builds the insurance import table by linking the latest patient insurance details to visits identified in the delta tracking table.
      writes ImportTransformVisitInsurance · reads HL7DeltaTransformPatientInsuranceVisitDelta
      ↳ feeds from MergeTransformVisit+VisitDelta, TransformPatientInsurance+VisitDelta, TruncateVisitDelta
    9. MergeTransformVisitInsurancesqlflow_mm_hl7_transform__mergetransformvisitinsurance.sql
      ✨ Updates the visit insurance transformation table by merging the latest imported insurance records for each visit.
      writes TransformVisitInsurance · reads ImportTransformVisitInsurance
      ↳ feeds from ImportTransformVisitInsurance
  7. LoadPatientsLoadPatients
    Upsert patient records into the patient dimension.
    · reads PatientTransformPatient
    custom patient query
  8. ▶Flow-MM-HL7-Processing-LoadDimensionMemberssub-job · 15↗
    1. Load Insurance Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_insurance_dimension.sqldisabled
      ✨ Populates the insurance dimension with unique carrier keys to enable filtering and analysis by payer.
      writes DimensionMember · reads Insurance
    2. Load EligibilityInsurance Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_eligibilityinsurance_dimension.sqldisabled
      ✨ Loads insurance members that support electronic eligibility and benefits verification via API integration.
      writes DimensionMember · reads DimensionMemberInsurance
    3. Load QuoteInsurance Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_quoteinsurance_dimension.sqldisabled
      ✨ Builds a dimension of insurance providers associated with active contracts to facilitate accurate patient cost estimates.
      writes DimensionMember · reads ContractMapInsurance
    4. LoadInsuranceDimensionMembersLoadInsuranceDimensionMembers
      Synchronize insurance-specific dimension members from the insurance table to update contract and eligibility filtering options across the application.
      writes DimensionMember
      load insuranceload quote insuranceload eligibility insurance
    5. Load Time Dimensionssqlflow_mm_hl7_processing_loaddimensionmembers__load_time_dimensions.sql
      ✨ Builds and updates time-based dimension members, including days, months, and years, for reporting and analytical hierarchies.
      writes DimensionMember · reads TransformVisit
    6. Load VisitStatus Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_visitstatus_dimension.sql
      ✨ Loads unique visit status values into the dimension table to support standardized reporting on appointment states.
      writes DimensionMember · reads TransformVisit
    7. Load Provider Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_provider_dimension.sql
      ✨ Synchronizes provider details into the dimension member table to enable provider-based filtering and grouping in reports.
      writes DimensionMember · reads Provider
    8. Load Facility Dimensionssqlflow_mm_hl7_processing_loaddimensionmembers__load_facility_dimensions.sql
      ✨ Loads facility and facility group hierarchies into the dimension table to organize location-based data for analysis.
      writes DimensionMember · reads DimensionMemberFacility
      ↳ feeds from Load Provider Dimension, Load State Dimensions, Load Time Dimensions, Load VisitStatus Dimension, Load VisitType Dimensions, LoadInsuranceDimensionMembers
    9. Load VisitType Dimensionssqlflow_mm_hl7_processing_loaddimensionmembers__load_visittype_dimensions.sql
      ✨ Builds and updates the VisitType dimension using the most recent appointment type descriptions from HL7 visit data.
      writes DimensionMember · reads HL7Visit
    10. Load Diagnosis Code Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_diagnosis_code_dimension.sqldisabled
      ✨ Loads and updates the DiagnosisCode dimension by extracting unique primary, secondary, and tertiary diagnosis codes from charge data.
      writes DimensionMember · reads TransformCharge
    11. Load RadioIsotopes Code Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_radioisotopes_code_dimension.sqldisabled
      ✨ Flags and updates the RadioIsotopes dimension to indicate the presence of radioisotope codes within the source risk data.
      writes DimensionMember · reads SourceCareRiskRadioIsotopeCodes
    12. Load Modifier Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_modifier_dimension.sqldisabled
      ✨ Loads and updates the Modifier dimension by collecting unique modifier codes from multiple charge record fields.
      writes DimensionMember · reads TransformCharge
    13. Load Place of Service Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_place_of_service_dimension.sqldisabled
      ✨ Builds and updates the PlaceOfService dimension using unique location codes extracted from charge transaction records.
      writes DimensionMember · reads TransformCharge
    14. Load SourceInsurance Dimensionsqlflow_mm_hl7_processing_loaddimensionmembers__load_sourceinsurance_dimension.sqldisabled
      ✨ Loads and updates the SourceInsurance dimension with mapped insurance provider details for specific data sources and active accounts.
      writes DimensionMember · reads AccountInsuranceInsuranceMap
    15. Load State Dimensionssqlflow_mm_hl7_processing_loaddimensionmembers__load_state_dimensions.sql
      ✨ Updates state dimensions using facility location data to allow for geographic segmentation of healthcare services and revenue.
      writes DimensionMember · reads Facility
  9. ▶Flow-MM-HL7-Processing-LoadVisitssub-job · 22↗
    1. Load TempBatchVisitsqlflow_mm_hl7_processing_loadvisits__load_tempbatchvisit.sqldisabled
      ✨ Builds a temporary batch table to identify upcoming visits requiring initial or follow-up insurance eligibility and benefit verification.
      writes TempBatchVisitVisitEligServiceTypesVisitServiceTypes · reads DimensionMemberTempBatchVisitTransformVisitTransformVisitInsuranceTransformVisitInsuranceCoverageStatusVisitEligServiceTypesVisitServiceTypes
    2. Augment Visit Deltasqlflow_mm_hl7_processing_loadvisits__augment_visit_delta.sqldisabled
      ✨ Updates the visit delta table by adding records flagged as requiring new insurance eligibility lookups from the batch process.
      writes VisitDelta · reads TempBatchVisitVisitDelta
    3. ReprocessChangedConfigurationsReprocessChangedConfigurations
      Identify and queue visits or accounts affected by configuration changes to ensure historical data reflects updated pricing or insurance rules.
      reprocess visits
    4. Augment VisitDelta - ReprocessChangedConfigssqlflow_mm_hl7_processing_loadvisits__augment_visitdelta_reprocesschangedconfigs.sql
      ✨ Updates insurance mappings for queued visits and flags them in the delta table for downstream reprocessing.
      writes VisitDeltaVisitKeyLookup · reads TransformVisitVisitDeltaVisitKeyLookup
    5. LoadVisitBatchLoadVisitBatch
      Load scheduled visits and associated charges into the processing pipeline using custom SQL queries for pricing estimation and quote generation.
      writes QuoteBatchVisitBatch · reads DimensionMemberInsuranceMapPatientTransformVisitTransformVisitInsuranceVisitVisitDelta
      ↳ feeds from Augment VisitDelta - ReprocessChangedConfigs, Sync Visits and Quotes
      custom visit queryauto charge queryauto service type querycustom coverage queryadd zero charge
    6. Flow-MM-HL7-Processing-LoadCustomPropertiessub-jobdisabled
    7. AutoApplyBundlesToVisitBatchAutoApplyBundlesToVisitBatch
      Applies matching charge bundles to batch visits based on facility and insurance to ensure consistent pricing across service types.
    8. CheckBenefitsflow
      Verifies insurance eligibility and retrieves detailed benefits information for visits by making real-time API calls to insurance providers.
      auto query
    9. Re-Check Benefitsflowdisabled
      Verifies insurance eligibility and retrieves detailed benefits information for visits by making real-time API calls to insurance providers.
      · reads EligibilityStatusMapEligibilityTxnTempBatchVisitVisitBatch
      custom queryDISABLED
    10. AutoApplyBenefitsAutoApplyBenefits
      Populates insurance benefits like copays and deductibles on batch visits using data from eligibility verification transactions.
    11. CalculateVisitsflow
      Compute the good-faith estimate (CalculateVisits).
      · reads DimensionMemberVisitBatch
      ↳ feeds from LoadVisitBatch
      custom querycalculate chargesbatch mode
    12. LogVisitChangeEventsBatchLogVisitChangeEventsBatch
      Compare batch visits against production records to detect and log specific property changes for auditing and compliance tracking.
    13. ExportBatchVisitEligibilityflow
      Push the batch to the app / export results.
      · reads EligibilityTxnvisitbatch
      ↳ feeds from LoadVisitBatch
      custom querytable name=BatchVisitEligibility
    14. Load TransformVisitInsuranceCoverageStatussqlflow_mm_hl7_processing_loadvisits__load_transformvisitinsurancecoveragestatus.sql
      ✨ Calculates and loads the latest insurance eligibility status and coverage details for visits in the current batch.
      writes TempEligibilityStatsTransformVisitInsuranceCoverageStatus · reads TempEligibilityStatsTransformVisitInsuranceVisitBatch
      ↳ feeds from LoadVisitBatch
    15. ▶Flow-MM-HL7-Processing-CustomPropertysub-job · 8↗
      1. Load Primary Coverage StatusUpdateCustomPropertyFromQuery
        · reads TransformVisitInsuranceCoverageStatusVisitBatch
        entity=Visitcustom property key=PrimaryCoverageStatuscustom querybatch mode
      2. Load Secondary Coverage StatusUpdateCustomPropertyFromQuery
        · reads TransformVisitInsuranceCoverageStatusVisitBatch
        entity=Visitcustom property key=SecondaryCoverageStatuscustom querybatch mode
      3. Load Tertiary Coverage StatusUpdateCustomPropertyFromQuery
        · reads TransformVisitInsuranceCoverageStatusVisitBatch
        entity=Visitcustom property key=TertiaryCoverageStatuscustom querybatch mode
      4. LoadCustomNotesCustomPropertyLoadCustomNotesCustomProperty
        · reads BundleNotesMappingHL7VisitVisitBatch
        entity=Visitcustom property key=CustomNotescustom querybatch mode
      5. LoadPriorAuthCustomPropertyLoadPriorAuthCustomProperty
        · reads CustomPropertyMapTransformVisitInsuranceVisitbatchsourcepriorauthdetails
        entity=Visitcustom property key=PriorAuthcustom querybatch mode
      6. LoadMedicaidRolloverCustomPropertyLoadMedicaidRolloverCustomProperty
        · reads CustomPropertyMapTransformVisitInsuranceVisitBatch
        entity=Visitcustom property key=MedicaidRollovercustom querybatch mode
      7. Load Has Active Medicaid/Medicare InsuranceUpdateCustomPropertyFromQuery
        · reads DimensionMemberTransformVisitInsuranceCoverageStatusVisitBatch
        entity=Visitcustom property key=HasActiveMedicaid/MedicareInsurancecustom querybatch mode
      8. Update HasMisMatchedMemberID Custom Property after recheckUpdateCustomPropertyFromQuery
        · reads BatchVisitEligibilityEligibilityTxnVisitBatch
        entity=Visitcustom property key=HasMismatchedMemberIDcustom querybatch mode
    16. Sync Visits and QuotesSyncBatch
      Synchronizes processed data from batch staging tables to production tables, updating visits, quotes, and custom property mappings.
      writes CustomPropertyMapQuoteVisitVisitChangeEvent · reads CustomPropertyMapBatchQuoteBatchVisitBatchVisitChangeEventBatch
      ↳ feeds from LoadVisitBatch
      sync quotesync visitsync custom property mapsync visit change event
    17. ExportBatchVisitEligibilityflowdisabled
      Push the batch to the app / export results.
      · reads EligibilityTxnVisitBatch
      custom querytable name=BatchEligibility_Resultsarchive table name=BatchEligibility_ResultsArchiveDISABLED
    18. SearchForInsuranceflowdisabled
      Find coverage for visits missing insurance.
      · reads EligibilityTxnBatch
      custom querytable name=InsuranceSearch_Resultbatch modeDISABLED
    19. Generate-Reportsub-jobdisabled
    20. ▶Load-Visit-Factsub-job · 1↗
      1. Load VisitFactsqlload_visit_fact__load_visitfact.sql
        ✨ Refreshes the visit fact table by recalculating statuses, payment flags, and letter activity for visits with recent updates or transactions.
        writes VisitFact · reads EligibilityStatusMapEligibilityTxnPatientLetterPaymentTxnVisit
    21. ▶Update-Dashboard-Tablessub-job · 1↗
      1. UpdateDashboardTablesUpdateDashboardTables
        Regenerates optimized aggregate tables for all dashboard reports and updates their definitions to reflect the latest source data.
    22. LoadVisitBatchAutoSendsqlflow_mm_hl7_processing_loadvisits__loadvisitbatchautosend.sql
      ✨ Populates the auto-send queue with visit data to facilitate automated end-of-day processing and communication.
      writes VisitBatchAutoSend · reads VisitBatch
      ↳ feeds from LoadVisitBatch
  10. Update HL7ProcessLogsqlflow_mm_hl7_processing__update_hl7processlog.sql
    ✨ Finalizes the HL7 processing log and updates the tracking pointer to the last successfully processed message.
    writes HL7ProcessLogLastArchiveKey · reads HL7DeltaHL7ProcessLogHL7ProcessLogLastArchiveKey
    ↳ feeds from HL7ProcessLog + HL7Delta
  11. WorkqueueAssignmentWorkqueueAssignment
    Assigns accounts or visits to workqueue stacks by evaluating matching criteria and updating item lists based on search terms.
  12. SendHL7Lettersflowdisabled
    Exports pending patient letters as HL7 messages with PDF attachments to a local directory for transmission to external systems.
    output file path=D:\ClientData\RIA\ORUpublish directory=D:\ClientData\RIA\ORUsending application=MDClaritysending facility=MDClarityreceiving application=Abbadox

markers

Why it classified as flow

Front-of-house actions found anywhere in the tree (including nested sub-jobs).

CalculateVisits
Compute the good-faith estimate
CheckBenefits
Real-time eligibility (270/271) benefit lookup
ExportBatchVisitEligibility
Export eligibility results
Load TransformVisitInsuranceCoverageStatus
ClarityFlow front-of-house step
LoadVisitBatch
ClarityFlow front-of-house step
LoadVisitBatchAutoSend
ClarityFlow front-of-house step
SearchForInsurance
Insurance discovery for visits missing coverage
SendHL7Letters
Emit letters back over the HL7 interface
auto_apply_bundles
Apply procedure bundles for the estimate
auto_benefits
Auto-apply parsed benefits to visits