Batch / file feed · ClarityFlow pipeline
OneGI · Flow-ECW-Processing
Overall Processing job for HL7 pipeline
trigger
How it's scheduled
The Batch / file feed arrives as a scheduled drop, so one overnight pass is enough: it ingests the day's file and works the whole batch to check eligibility, price the good-faith estimate, and send the patient letter. A tighter loop would just re-scan the same file.
| Scheduled task | Cadence | Server | State | Coordination |
|---|---|---|---|---|
\MDC\Daily - Processing - ONEGI | daily at 23:00 | prod-db-3 | enabled | standalone |
shape
Against the canonical Flow spine
Every Flow pipeline is a variation on one shape. Lit stages are the ones this pipeline runs; the labels match the capability lens.
computed concepts · from the SQL
What it computes
Business concepts mined from this pipeline's SQL text — the meaning behind the typed steps. Each links to the concept lens (who else computes it).
task tree
What the pipeline runs
Tasks in order, each with what it does — typed stepsfrom the MDClarity task docs (click to open the reference), sub-jobsyou can expand inline (▶) to reveal their own tasks, and SQL steps summarized by Gemini (✨). Every step shows the tables it writes/reads and which earlier step it feeds from(sqlglot lineage).
Flow-ECW-Processing
▶Flow-ECW-Source-LoadDatasub-job · 16↗
▶Flow-Source-Load-ECW-AlertDatasub-job · 1↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWAlertDataSourceECWAlertDatafile path={FilePath}archive directory=D:\ClientData\OneGI\ECW\RevFind\Archivebase table name=ECWAlertDatarow terminator=Newline
▶Flow-Source-Load-ECW-ClinicalAlertssub-job · 1↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWClinicalAlertsSourceECWClinicalAlertsfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWClinicalAlertsrow terminator=Newline
▶Flow-Source-Load-ECW-CurrentOfficeVisitNotesub-job · 3↗
- Import delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWCurrentOfficeVisitNoteSourceECWCurrentOfficeVisitNotefile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWCurrentOfficeVisitNoterow terminator=NewlineDISABLED - LoadDataFileLoadDataFileImports delimited data files into database tables using predefined specifications for validation, loading, and optional file archiving.writes
ImportECWCurrentOfficeVisitNotedata file spec=ImportECWCurrentOfficeVisitNotefile path={FilePath}import mode=Replacearchive file path={SourceDataArchiveECW} - ExecuteMergeSqlExecuteMergeSqlSynchronizes data from external sources into target tables by performing batched insert, update, and delete operations based on keys.writes
SourceECWCurrentOfficeVisitNote· readsImportECWCurrentOfficeVisitNote↳ feeds from LoadDataFilemerge data source type=ThisDatabasetable name=SourceECWCurrentOfficeVisitNotemerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
▶Flow-Source-Load-ECW-Encountersub-job · 1↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWEncounterSourceECWEncounterfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWEncounterrow terminator=Newline
▶Flow-Source-Load-ECW-FlowPatientInsurancesub-job · 1↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWFlowPatientInsuranceSourceECWFlowPatientInsurancefile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWFlowPatientInsurancerow terminator=Newline
▶Flow-Source-Load-ECW-FlowPatientssub-job · 2↗
- ExecuteProcessExecuteProcesscommand=powershell.exe
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWFlowPatientsSourceECWFlowPatientsfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWFlowPatientsrow terminator=Newline
▶Flow-Source-Load-ECW-HistoryPresentIllnesssub-job · 3↗
- Import delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWHistoryPresentIllnessSourceECWHistoryPresentIllnessfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWHistoryPresentIllnessrow terminator=NewlineDISABLED - LoadDataFileLoadDataFileImports delimited data files into database tables using predefined specifications for validation, loading, and optional file archiving.writes
ImportECWHistoryPresentIllnessdata file spec=ImportECWHistoryPresentIllnessfile path={FilePath}import mode=Replacearchive file path={SourceDataArchiveECW} - ExecuteMergeSqlExecuteMergeSqlSynchronizes data from external sources into target tables by performing batched insert, update, and delete operations based on keys.writes
SourceECWHistoryPresentIllness· readsImportECWHistoryPresentIllness↳ feeds from LoadDataFilemerge data source type=ThisDatabasetable name=SourceECWHistoryPresentIllnessmerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
▶Flow-Source-Load-ECW-LastOfficeVisitNotesub-job · 1↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWLastOfficeVisitNoteSourceECWLastOfficeVisitNotefile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWLastOfficeVisitNoterow terminator=Newline
▶Flow-Source-Load-ECW-Orderssub-job · 1↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWOrdersSourceECWOrdersfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWOrdersrow terminator=Newline
▶Flow-Source-Load-ECW-PriorDiagnosissub-job · 4↗
- Import delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWPriorDiagnosisSourceECWPriorDiagnosisfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWPriorDiagnosisrow terminator=NewlineDISABLED - ExecuteProcessExecuteProcess
- LoadDataFileLoadDataFileImports delimited data files into database tables using predefined specifications for validation, loading, and optional file archiving.writes
ImportECWPriorDiagnosisdata file spec=ImportECWPriorDiagnosisfile path={FilePath}import mode=Replacearchive file path={SourceDataArchiveECW} - ExecuteMergeSqlExecuteMergeSqlSynchronizes data from external sources into target tables by performing batched insert, update, and delete operations based on keys.writes
SourceECWPriorDiagnosis· readsImportECWPriorDiagnosis↳ feeds from LoadDataFilemerge data source type=ThisDatabasetable name=SourceECWPriorDiagnosismerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
▶Flow-Source-Load-ECW-ProblemListsub-job · 4↗
- Import delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWProblemListSourceECWProblemListfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWProblemListrow terminator=NewlineDISABLED - ExecuteProcessExecuteProcess
- LoadDataFileLoadDataFileImports delimited data files into database tables using predefined specifications for validation, loading, and optional file archiving.writes
ImportECWProblemListdata file spec=ImportECWProblemListfile path={FilePath}import mode=Replacearchive file path={SourceDataArchiveECW} - ExecuteMergeSqlExecuteMergeSqlSynchronizes data from external sources into target tables by performing batched insert, update, and delete operations based on keys.writes
SourceECWProblemList· readsImportECWProblemList↳ feeds from LoadDataFilemerge data source type=ThisDatabasetable name=SourceECWProblemListmerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
▶Flow-Source-Load-ECW-Referralsub-job · 4↗
- Import delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWReferralSourceECWReferralfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWReferralrow terminator=NewlineDISABLED - ExecuteProcessExecuteProcess
- LoadDataFileLoadDataFileImports delimited data files into database tables using predefined specifications for validation, loading, and optional file archiving.writes
ImportECWReferraldata file spec=ImportECWReferralfile path={FilePath}import mode=Replacearchive file path={SourceDataArchiveECW} - ExecuteMergeSqlExecuteMergeSqlSynchronizes data from external sources into target tables by performing batched insert, update, and delete operations based on keys.writes
SourceECWReferral· readsImportECWReferral↳ feeds from LoadDataFilemerge data source type=ThisDatabasetable name=SourceECWReferralmerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
▶Flow-Source-Load-ECW-VisitCodessub-job · 1↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWVisitCodesSourceECWVisitCodesfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWVisitCodesrow terminator=Newline
▶Flow-Source-Load-ECW-VisitStatussub-job · 2↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWVisitStatusSourceECWVisitStatusfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWVisitStatusrow terminator=Newline - Remove NUL - ExecuteSqlsql
flow_source_load_ecw_visitstatus__remove_nul_executesql.sql✨ Cleanses visit status data by removing null characters to ensure data integrity and prevent processing errors.· readsSourceECWVisitStatus↳ feeds from ProcessDataFile
▶Flow-Source-Load-ECW-AscProcEncsub-job · 1↗
- ProcessDataFileProcessDataFileImport delimited files by automatically detecting their structure and merging the data into source tables.writes
ImportECWAscProcEncSourceECWAscProcEncfile path={FilePath}archive directory={SourceDataArchiveECW}base table name=ECWAscProcEncrow terminator=Newline
- SourceECWColonoscopyType - ExecuteMergeSqlMergeSqlwrites
CombinedDiagnosisCodesConsolidatedNotesEncounterFlagsPatientProblemsSourceECWColonoscopyType· readsSourceECWCurrentOfficeVisitNoteSourceECWEncounterSourceECWHistoryPresentIllnessSourceECWLastOfficeVisitNoteSourceECWProblemListSourceECWReferralmerge data source type=ThisDatabasetable name=SourceECWColonoscopyTypemerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
▶Flow-ECW-LoadEntitiessub-job · 3↗Load Facility, Provider, Patients
- Load FacilityLoadEntitieswrites
Facility· readsFacilitySourceECWFacilitiesentity type=Facilitycustom querykey prefix=F - Merge ProvidersMergeSqlwrites
Provider· readsSourceECWDoctorsTransformAccountmerge data source type=ThisDatabasetable name=Providermerge mode=InsertUpdateSoftDeletebulk copy timeout seconds=300bulk copy use table lock - Load PatientsLoadPatientsdisabledUpsert patient records into the patient dimension.· reads
PatientTransformPatientcustom patient queryDISABLED
▶Flow-ECW-Transformsub-job · 8↗EWC Transformation Layer
- TruncateVisitDeltasql
flow_ecw_transform__truncatevisitdelta.sql✨ Clears the visit delta table to prepare for a fresh set of changed records in the current pipeline run.writesVisitDelta - TransformPatientTransformPatientwrites
TransformPatient· readsSourceECWFlowPatientsmerge data source type=ThisDatabasetable name=TransformPatientmerge mode=InsertUpdateSoftDeletebulk copy timeout seconds=300bulk copy use table lock - TransformPatientInsurance PatientDeltasql
flow_ecw_transform__transformpatientinsurance_patientdelta.sql✨ Identifies patients with updated insurance information since the last processing time and loads them into the patient delta table.writesFlowPatientDelta· readsFlowLogLastProcessingTimeSourceECWFlowPatientInsurance - Merge TransformPatientInsuranceMergeSqlwrites
TransformPatientInsurancemerge data source type=ThisDatabasetable name=TransformPatientInsurancemerge mode=InsertUpdateHardDeletebulk copy timeout seconds=300bulk copy use table lock - TransformPatientInsurance VisitDeltasql
flow_ecw_transform__transformpatientinsurance_visitdelta.sql✨ Adds future visits to the visit delta table for patients who have had recent insurance updates.writesVisitDelta· readsFlowPatientDeltaTransformVisit↳ feeds from MergeTransformVisit, TransformPatientInsurance PatientDelta - VisitDeltasql
flow_ecw_transform__visitdelta.sql✨ Identifies new or updated encounters from source data and adds them to the visit delta table for processing.writesVisitDelta· readsFlowLogLastProcessingTimeSourceECWEncounterTransformVisitVisitDelta↳ feeds from MergeTransformVisit, TransformPatientInsurance VisitDelta, TruncateVisitDelta - MergeTransformVisitsql
flow_ecw_transform__mergetransformvisit.sql✨ Transforms and loads encounter details like status, time, and provider into the transformed visit table for identified delta records.writesTransformVisit· readsFacilitySourceECWAscProcEncSourceECWColonoscopyTypeSourceECWEncounterSourceECWPatientsSourceECWReferralSourceECWVisitCodesSourceECWVisitStatus↳ feeds from TransformPatientInsurance VisitDelta, TruncateVisitDelta, VisitDelta - MergeTransformVisitInsurancesql
flow_ecw_transform__mergetransformvisitinsurance.sql✨ Maps and loads patient insurance information into the transformed visit insurance table for visits identified in the delta set.writesTransformVisitInsurance· readsTransformPatientInsuranceTransformVisitVisitDelta↳ feeds from Merge TransformPatientInsurance, MergeTransformVisit, TransformPatientInsurance VisitDelta, TruncateVisitDelta, VisitDelta
- LoadPatientsLoadPatientsdisabledUpsert patient records into the patient dimension.· reads
FlowPatientDeltaPatientTransformPatientcustom patient queryDISABLED ▶Flow-ECW-Processing-LoadDimensionMemberssub-job · 16↗
- LoadDimensionMembersLoadDimensionMembersSynchronize facility, provider, and insurance metadata into the dimension member table to refresh dashboard filters and reporting categories.writes
DimensionMemberload insuranceload quote insuranceload eligibility insuranceload facilityload provider - Load Insurance Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_insurance_dimension.sqldisabled✨ Populates the insurance dimension with unique carrier keys to enable filtering and analysis by payer.writesDimensionMember· readsInsurance - Load EligibilityInsurance Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_eligibilityinsurance_dimension.sqldisabled✨ Loads insurance members that support electronic eligibility and benefits verification via API integration.writesDimensionMember· readsDimensionMemberInsurance - Load QuoteInsurance Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_quoteinsurance_dimension.sqldisabled✨ Builds a dimension of insurance providers associated with active contracts to facilitate accurate patient cost estimates.writesDimensionMember· readsContractMapInsurance - Synchronize insurance-specific dimension members from the insurance table to update contract and eligibility filtering options across the application.writes
DimensionMemberload insuranceload quote insuranceload eligibility insuranceDISABLED - Load Time Dimensionssql
flow_ecw_processing_loaddimensionmembers__load_time_dimensions.sql✨ Populates time dimension hierarchies for days, months, and years using appointment timestamps to support temporal data analysis.writesDimensionMember· readsTransformVisit - Load VisitStatus Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_visitstatus_dimension.sql✨ Builds the visit status dimension by extracting unique appointment statuses from transformed visit data.writesDimensionMember· readsTransformVisit - Load Provider Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_provider_dimension.sqldisabled✨ Synchronizes provider details into the dimension member table to enable provider-based filtering and grouping in reports.writesDimensionMember· readsProvider - Load Facility Dimensionssql
flow_ecw_processing_loaddimensionmembers__load_facility_dimensions.sqldisabled✨ Builds facility and facility group hierarchies, linking clinics to their parent groups and tax identification numbers for financial reporting.writesDimensionMember· readsDimensionMemberFacilitysourceFacilityGroupToTIN - Load VisitType Dimensionssql
flow_ecw_processing_loaddimensionmembers__load_visittype_dimensions.sql✨ Builds and updates visit type dimension members using appointment types from transformed visit data.writesDimensionMember· readsTransformVisit - Load Diagnosis Code Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_diagnosis_code_dimension.sqldisabled✨ Loads and updates the DiagnosisCode dimension by extracting unique primary, secondary, and tertiary diagnosis codes from charge data.writesDimensionMember· readsTransformCharge - Load RadioIsotopes Code Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_radioisotopes_code_dimension.sqldisabled✨ Flags and updates the RadioIsotopes dimension to indicate the presence of radioisotope codes within the source risk data.writesDimensionMember· readsSourceCareRiskRadioIsotopeCodes - Load Modifier Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_modifier_dimension.sqldisabled✨ Loads and updates the Modifier dimension by collecting unique modifier codes from multiple charge record fields.writesDimensionMember· readsTransformCharge - Load Place of Service Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_place_of_service_dimension.sqldisabled✨ Builds and updates the PlaceOfService dimension using unique location codes extracted from charge transaction records.writesDimensionMember· readsTransformCharge - Load SourceInsurance Dimensionsql
flow_ecw_processing_loaddimensionmembers__load_sourceinsurance_dimension.sqldisabled✨ Loads and updates the SourceInsurance dimension with mapped insurance provider details for specific data sources and active accounts.writesDimensionMember· readsAccountInsuranceInsuranceMap - Load State Dimensionssql
flow_ecw_processing_loaddimensionmembers__load_state_dimensions.sql✨ Updates state dimensions using facility location data to allow for geographic segmentation of healthcare services and revenue.writesDimensionMember· readsFacility
▶Flow-ECW-Processing-LoadVisitssub-job · 21↗
- Load TempBatchVisitsql
flow_ecw_processing_loadvisits__load_tempbatchvisit.sqldisabled✨ Builds a temporary batch table to identify upcoming visits requiring initial or follow-up insurance eligibility and benefit verification.writesTempBatchVisitVisitEligServiceTypesVisitServiceTypes· readsDimensionMemberTempBatchVisitTransformVisitTransformVisitInsuranceTransformVisitInsuranceCoverageStatusVisitEligServiceTypesVisitServiceTypes - Augment Visit Deltasql
flow_ecw_processing_loadvisits__augment_visit_delta.sqldisabled✨ Updates the visit delta table by adding records flagged as requiring new insurance eligibility lookups from the batch process.writesVisitDelta· readsTempBatchVisitVisitDelta - Identify and queue visits or accounts affected by configuration changes to ensure historical data reflects updated pricing or insurance rules.reprocess visitsDISABLED
- Augment VisitDelta - ReprocessChangedConfigssql
flow_ecw_processing_loadvisits__augment_visitdelta_reprocesschangedconfigs.sqldisabled✨ Updates insurance mappings for visits in the reprocess queue and adds new records to the visit delta table.writesVisitDeltaVisitKeyLookup· readsSourceOrganizationMMTransformVisitVisitDeltaVisitKeyLookup - LoadVisitBatchLoadVisitBatchLoad scheduled visits and associated charges into the processing pipeline using custom SQL queries for pricing estimation and quote generation.writes
QuoteBatchVisitBatch· readsInsuranceMapPatientTransformVisitTransformVisitInsuranceVisitVisitDelta↳ feeds from Sync Visits and Quotescustom visit queryauto charge queryauto service type querycustom coverage queryadd zero charge - AutoApplyBundlesToVisitBatchAutoApplyBundlesToVisitBatchApplies matching charge bundles to batch visits based on facility and insurance to ensure consistent pricing across service types.
- CheckBenefitsflowVerifies insurance eligibility and retrieves detailed benefits information for visits by making real-time API calls to insurance providers.auto query
- Verifies insurance eligibility and retrieves detailed benefits information for visits by making real-time API calls to insurance providers.· reads
EligibilityStatusMapEligibilityTxnTempBatchVisitVisitBatchcustom queryDISABLED - AutoApplyBenefitsAutoApplyBenefitsPopulates insurance benefits like copays and deductibles on batch visits using data from eligibility verification transactions.
- CalculateVisitsflowCompute the good-faith estimate (CalculateVisits).· reads
DimensionMemberVisitBatch↳ feeds from LoadVisitBatchcustom querycalculate chargesbatch mode - Generates and sends patient cost estimate letters for visits in the batch table based on configured qualification and communication rules.DISABLED
- LogVisitChangeEventsBatchLogVisitChangeEventsBatchCompare batch visits against production records to detect and log specific property changes for auditing and compliance tracking.
- Load TransformVisitInsuranceCoverageStatussql
flow_ecw_processing_loadvisits__load_transformvisitinsurancecoveragestatus.sql✨ Loads insurance coverage status by calculating the latest eligibility transaction results for visits in the current batch.writesTempEligibilityStatsTransformVisitInsuranceCoverageStatus· readsTempEligibilityStatsTransformVisitInsuranceVisitBatch↳ feeds from LoadVisitBatch ▶Flow-MM-Processing-CustomPropertysub-job · 4↗
- Load Primary Coverage StatusUpdateCustomPropertyFromQuery· reads
TransformVisitInsuranceCoverageStatusVisitBatchentity=Visitcustom property key=PrimaryCoverageStatuscustom querybatch mode - Load Secondary Coverage StatusUpdateCustomPropertyFromQuery· reads
TransformVisitInsuranceCoverageStatusVisitBatchentity=Visitcustom property key=SecondaryCoverageStatuscustom querybatch mode - Load Tertiary Coverage StatusUpdateCustomPropertyFromQuery· reads
TransformVisitInsuranceCoverageStatusVisitBatchentity=Visitcustom property key=TertiaryCoverageStatuscustom querybatch mode - Load Colonoscopy TypeUpdateCustomPropertyFromQuery· reads
TransformVisitVisitBatchentity=Visitcustom property key=ColonoscopyTypecustom querybatch mode
- Sync Visits and QuotesSyncBatchSynchronizes processed data from batch staging tables to production tables, updating visits, quotes, and custom property mappings.writes
CustomPropertyMapQuoteVisitVisitChangeEvent· readsCustomPropertyMapBatchQuoteBatchVisitBatchVisitChangeEventBatch↳ feeds from LoadVisitBatchsync quotesync visitsync custom property mapsync visit change event - ExportBatchVisitEligibilityflowdisabledPush the batch to the app / export results.· reads
EligibilityTxnVisitBatchcustom querytable name=BatchEligibility_Resultsarchive table name=BatchEligibility_ResultsArchiveDISABLED - SearchForInsuranceflowdisabledFind coverage for visits missing insurance.· reads
EligibilityTxnBatchcustom querytable name=InsuranceSearch_Resultbatch modeDISABLED
- Update HL7ProcessLogsql
flow_ecw_processing__update_hl7processlog.sql✨ Updates the processing log with the current timestamp to track the last successful execution of the data flow.writesFlowLogLastProcessingTime - WorkqueueAssignmentWorkqueueAssignmentAssigns accounts or visits to workqueue stacks by evaluating matching criteria and updating item lists based on search terms.
- Exports pending patient letters as HL7 messages with PDF attachments to a local directory for transmission to external systems.output file path=D:\ClientData\RIA\ORUpublish directory=D:\ClientData\RIA\ORUsending application=MDClaritysending facility=MDClarityreceiving application=Abbadox
markers
Why it classified as flow
Front-of-house actions found anywhere in the tree (including nested sub-jobs).