Batch / file feed · ClarityFlow pipeline
PPRM · Daily-Visit-Processing
End to end daily visit processing job
trigger
How it's scheduled
The Batch / file feed arrives as a scheduled drop, so one overnight pass is enough: it ingests the day's file and works the whole batch to check eligibility, price the good-faith estimate, and send the patient letter. A tighter loop would just re-scan the same file.
| Scheduled task | Cadence | Server | State | Coordination |
|---|---|---|---|---|
\MDC\Daily - Processing - PPRM | daily at 05:00 | prod-db-3 | enabled | standalone |
shape
Against the canonical Flow spine
Every Flow pipeline is a variation on one shape. Lit stages are the ones this pipeline runs; the labels match the capability lens.
computed concepts · from the SQL
What it computes
Business concepts mined from this pipeline's SQL text — the meaning behind the typed steps. Each links to the concept lens (who else computes it).
Payers in SQLTricare
task tree
What the pipeline runs
Tasks in order, each with what it does — typed stepsfrom the MDClarity task docs (click to open the reference), sub-jobsyou can expand inline (▶) to reveal their own tasks, and SQL steps summarized by Gemini (✨). Every step shows the tables it writes/reads and which earlier step it feeds from(sqlglot lineage).
Daily-Visit-Processing
▶Load-NextGen-Filessub-job · 30↗*TROUBLESHOOTING NOTE: The password contains an escape character \ before the ". This WILL NOT WORK if attempting to connect manually via FileZilla. Remove it from the password first. Job Description: Downloads and imports NextGen data
- DownloadDownloadDownloads files from remote FTP or SFTP servers to local directories with support for archiving and parallel transfers.server=PPRM_ProdMirror-mdcssh-sftpremote directory=/local directory={SourceDataFolder}
- Load AppointmentsLoadFilewrites
ImportAppointmentdata file spec=ImportAppointmentimport mode=Replace - Load SourceAppointmentsql
load_nextgen_files__load_sourceappointment.sql✨ Loads appointment details from the import table into the source repository, replacing existing records to ensure data freshness.writesSourceAppointment· readsImportAppointment↳ feeds from Load Appointments - Load PatientsLoadFilewrites
ImportPatientdata file spec=ImportPatientfile path={SourceDataFolder}\ExportPatient.txtimport mode=Replace - Load SourcePatientsql
load_nextgen_files__load_sourcepatient.sql✨ Updates the source patient table by replacing existing records with the latest demographic and contact information from the import layer.writesSourcePatient· readsImportPatient↳ feeds from Load Patients - Load Patient InsuranceLoadFilewrites
ImportPatientInsurancedata file spec=ImportPatientInsuranceimport mode=Replace - Load SourcePatientInsurancesql
load_nextgen_files__load_sourcepatientinsurance.sql✨ Loads patient insurance coverage and eligibility details into the source table, overwriting existing records for the same individuals.writesSourcePatientInsurance· readsImportPatientInsurance↳ feeds from Load Patient Insurance - Load PayerLoadFilewrites
ImportPayerdata file spec=ImportPayerfile path={SourceDataFolder}\ExportPayer.txtimport mode=Replace - Load SourcePayersql
load_nextgen_files__load_sourcepayer.sql✨ Refreshes the source payer table with updated insurance company information, financial classes, and subgroup classifications from the import data.writesSourcePayer· readsImportPayerSourcePayer↳ feeds from Load Payer - Load ProvidersLoadFilewrites
ImportProviderdata file spec=ImportProviderfile path={SourceDataFolder}\ExportProviders.txtimport mode=Replace - Load SourceProvidersql
load_nextgen_files__load_sourceprovider.sql✨ Updates the source provider registry with current credentials, specialties, and NPI data by replacing existing records with imported values.writesSourceProvider· readsImportProviderSourceProvider↳ feeds from Load Providers - Load ChargeMasterLoadFilewrites
ImportChargeMasterdata file spec=ImportChargeMasterimport mode=Replace - Load SourceChargeMastersql
load_nextgen_files__load_sourcechargemaster.sql✨ Synchronizes the charge master by updating existing service prices and descriptions or inserting new charge codes and library definitions.writesSourceChargeMaster· readsImportChargeMaster↳ feeds from Load ChargeMaster - Load EncountersLoadFiledisabledwrites
ImportEncounterdata file spec=ImportEncounterfile path={SourceDataFolder}\ExportEncounters.txtimport mode=ReplaceDISABLED - Load SourceEncountersql
load_nextgen_files__load_sourceencounter.sqldisabled✨ Loads clinical encounter and billing balance information into the source table, replacing existing records to maintain current visit status.writesSourceEncounter· readsImportEncounterSourceEncounter - Load ChargesLoadFiledisabledwrites
ImportChargedata file spec=ImportChargefile path={SourceDataFolder}\ExportCharges.txtimport mode=ReplaceDISABLED - Load SourceChargesql
load_nextgen_files__load_sourcecharge.sqldisabled✨ Loads and refreshes encounter charge details from imported files into the source charge table.writesSourceCharge· readsImportChargeSourceCharge - Load AdjustmentsLoadFiledisabledwrites
ImportAdjustmentdata file spec=ImportAdjustmentimport mode=ReplaceDISABLED - Load SourceAdjustmentsql
load_nextgen_files__load_sourceadjustment.sqldisabled✨ Loads and refreshes financial adjustment records from imported files into the source adjustment table.writesSourceAdjustment· readsImportAdjustmentSourceAdjustment - Load TransactionDetailLoadFiledisabledwrites
ImportTransactionDetaildata file spec=ImportTransactionDetailimport mode=ReplaceDISABLED - Load SourceTransactionDetailsql
load_nextgen_files__load_sourcetransactiondetail.sqldisabled✨ Loads and refreshes detailed transaction data for encounters and patient accounts from imported files into the source table.writesSourceTransactionDetail· readsImportTransactionDetailSourceTransactionDetail - Load Encounter InsuranceLoadFiledisabledwrites
ImportEncounterInsurancedata file spec=ImportEncounterInsuranceimport mode=ReplaceDISABLED - Load SourceEncounterInsurancesql
load_nextgen_files__load_sourceencounterinsurance.sqldisabled✨ Loads and refreshes insurance coverage and policy details for specific encounters from imported files into the source table.writesSourceEncounterInsurance· readsImportEncounterInsuranceSourceEncounterInsurance - Load ImportEventLoadFilewrites
ImportEventdata file spec=ImportEventfile path={SourceDataFolder}\ExportEvent.txtimport mode=Replace - Load SourceEventsql
load_nextgen_files__load_sourceevent.sql✨ Synchronizes event definitions and metadata from imported files into the source event table to maintain current records.writesSourceEvent· readsImportEvent↳ feeds from Load ImportEvent - Load LocationsLoadFilewrites
ImportLocationdata file spec=ImportLocationfile path={SourceDataFolder}\ExportLocation.txtimport mode=Replace - Load SourceLocationsql
load_nextgen_files__load_sourcelocation.sql✨ Loads and refreshes facility and service location details from imported files into the source location table.writesSourceLocation· readsImportLocationSourceLocation↳ feeds from Load Locations - Load ImportPatientRelationLoadFilewrites
ImportPatientRelationdata file spec=ImportPatientRelationimport mode=Replace - Load SourcePatientRelationsql
load_nextgen_files__load_sourcepatientrelation.sql✨ Loads and refreshes patient relationship and guarantor data from imported files into the source patient relation table.writesSourcePatientRelation· readsImportPatientRelationSourcePatientRelation↳ feeds from Load ImportPatientRelation - Archive FilesExecuteProcesscommand=cmd.exe
▶Load-Chargessub-job · 3↗Loads ChargeType and Chargemaster
- Load TransformChargeMastersql
load_charges__load_transformchargemaster.sql✨ Loads and synchronizes charge master data, updating prices, descriptions, and billing codes from the source system into the transformation layer.writestransformchargemaster· readsSourceChargeMaster - Load ChargeTypesql
load_charges__load_chargetype.sql✨ Builds the charge type dimension by extracting unique procedure codes and billing categories from the transformed charge master data.writesChargeType· readsTransformChargeMasterchargetype↳ feeds from Load TransformChargeMaster - Load ContractMapsql
load_charges__load_contractmap.sqldisabled✨ Updates the contract mapping table to associate charge-based fee logic with all combinations of facilities and billing types.writesContractMap· readsBillTypeFacilityTransformChargeMaster
▶Load-Visit-Transform-Tablessub-job · 7↗Loads transform tables for Visits
- Load TempVisitDeltasql
load_visit_transform_tables__load_tempvisitdelta.sql✨ Identifies visits and insurance records updated since the last process run to populate a delta table for incremental processing.writesVisitDelta· readsProcessLogVisitDelta ▶Load-SourceAppointmentChainsub-job · 1↗
- Load SourceAppointmentChainsql
load_sourceappointmentchain__load_sourceappointmentchain.sql✨ Builds a chain of rescheduled appointments to identify and retain the most recent record for each visit.writesSourceAppointmentChain· readsSourceAppointmentVisitDelta
- Load TransformPatientsql
load_visit_transform_tables__load_transformpatient.sql✨ Synchronizes patient demographic and contact information from source records into the transformed patient table using a merge operation.writesTransformPatient· readsSourcePatient - Load TransformVisitsql
load_visit_transform_tables__load_transformvisit.sql✨ Updates the transformed visit table with current appointment details, statuses, and facility information for records identified in the delta set.writesTransformVisit· readsSourceAppointmentSourceAppointmentChainSourceEventVisitDelta↳ feeds from Load TempVisitDelta - Load TransformCustomPropertiessql
load_visit_transform_tables__load_transformcustomproperties.sql✨ Extracts and loads custom appointment properties and key-value pairs into the transformed properties table for updated visits.writesTransformCustomProperties· readsSourceAppointmentSourceAppointmentChainVisitDelta↳ feeds from Load TempVisitDelta - Load TransformVisitInsurancesql
load_visit_transform_tables__load_transformvisitinsurance.sql✨ Updates transformed insurance coverage and coordination of benefits data for visits, including logic to identify self-pay and unmapped payers.writesTransformVisitInsurance· readsSourceAppointmentSourceAppointmentChainSourceInsuranceMapSourcePatientInsuranceSourcePatientRelationTransformCustomPropertiesTransformPatientTransformVisit↳ feeds from Load TempVisitDelta, Load TransformCustomProperties, Load TransformPatient, Load TransformVisit - Load TransformVisitChargesql
load_visit_transform_tables__load_transformvisitcharge.sql✨ Builds visit charges by mapping appointment types and patient demographics to procedure codes and quantities for cost estimation.writesTransformVisitCharge· readsChargeTypeImportGFEMappingSourceEventTransformPatientTransformVisitVisitDelta↳ feeds from Load TempVisitDelta, Load TransformPatient, Load TransformVisit
▶Load-Relationssub-job · 1↗Loads relational tables like Provider and ChargeType
- Load Provider Relationsql
load_relations__load_provider_relation.sql✨ Updates the master provider table with current names, types, and identifiers for providers associated with active visits.writesProvider· readsSourceProviderTransformVisit
▶Load-Entitiessub-job · 3↗Loads Facility, Patient, BillType Entities
- Load FacilityLoadEntitieswrites
Facility· readsSourceAppointmentSourceLocationentity type=Facilitycustom query - Load PatientsLoadPatientsUpsert patient records into the patient dimension.· reads
PatientTransformPatientcustom patient query - Load BillTypeLoadEntitieswrites
BillType· readsChargeTypeentity type=BillTypeauto query
▶Load-Dimension-Memberssub-job · 18↗Loads Dimension Members
- Load Insurance Categorysql
load_dimension_members__load_insurance_category.sqldisabled✨ Loads insurance categories into the dimension member table to support reporting by commercial and non-commercial insurance types.writesDimensionMember· readsInsuranceCategory - Load Insurancesql
load_dimension_members__load_insurance.sqldisabled✨ Loads individual insurance plans into the dimension member table and maps them to their respective insurance categories.writesDimensionMember· readsInsuranceInsuranceCategory - Load EligibilityInsurancesql
load_dimension_members__load_eligibilityinsurance.sqldisabled✨ Updates the eligibility insurance dimension by filtering insurance records for those containing valid electronic benefit API references.writesDimensionMember· readsInsurance - Load Facilitysql
load_dimension_members__load_facility.sql✨ Loads facility and location information into the dimension member table to enable data filtering and reporting by service site.writesDimensionMember· readsFacilitySourceLocation - Load VisitStatussql
load_dimension_members__load_visitstatus.sql✨ Loads visit status members into the dimension table using appointment status values extracted from transformed visit data.writesDimensionMember· readsTransformVisit - Load Provider Dimensionsql
load_dimension_members__load_provider_dimension.sql✨ Builds and updates the Provider dimension by mapping source provider IDs to their respective specialty codes and display names.writesDimensionMember· readsProviderSourceProvider - Load ProviderType Dimensionsql
load_dimension_members__load_providertype_dimension.sql✨ Builds the provider type dimension by extracting unique parent classifications from the existing provider dimension members.writesDimensionMember· readsDimensionMember↳ feeds from Load ChargeType, Load EligibilityServiceType, Load Facility, Load Modifier, Load PayerType Dimension, Load Procedure SubCategory, Load ProcedureCode, Load Provider Dimension, Load State, Load VisitStatus, Load VisitType DimensionMember - Load ChargeTypesql
load_dimension_members__load_chargetype.sql✨ Loads the ChargeType dimension, organizing charge IDs under parent CPT subcategories for reporting and analysis.writesDimensionMember· readsChargeType - Load Procedure SubCategorysql
load_dimension_members__load_procedure_subcategory.sql✨ Builds the procedure subcategory dimension by extracting parent categories from existing charge type records.writesDimensionMember· readsDimensionMember↳ feeds from Load ChargeType, Load EligibilityServiceType, Load Facility, Load Modifier, Load PayerType Dimension, Load ProcedureCode, Load Provider Dimension, Load ProviderType Dimension, Load State, Load VisitStatus, Load VisitType DimensionMember - Load ProcedureCodesql
load_dimension_members__load_procedurecode.sql✨ Builds the ProcedureCode dimension by extracting unique procedure codes from the charge type source data.writesDimensionMember· readsChargeType - Load Modifiersql
load_dimension_members__load_modifier.sql✨ Updates the modifier dimension with codes and descriptions sourced from common procedure modifier reference data.writesDimensionMember· readsProcedureModifier - Load Time - OLDsql
load_dimension_members__load_time_old.sqldisabled✨ Populates time dimensions for days, months, and years based on the range of dates found in appointment records.writesDimensionMember· readsSourceAppointment - Load Timesql
load_dimension_members__load_time.sql✨ Executes a stored procedure to initialize and load time-based dimension members starting from a defined historical date.· readsload_TimeDimensions - Load EligibilityServiceTypesql
load_dimension_members__load_eligibilityservicetype.sql✨ Loads the EligibilityServiceType dimension with standardized medical service categories used for insurance benefit verification.writesDimensionMember - Load PayerType Dimensionsql
load_dimension_members__load_payertype_dimension.sql✨ Updates the PayerType dimension to include specific members for self-pay classifications within the reporting system.writesDimensionMember - Load VisitType DimensionMembersql
load_dimension_members__load_visittype_dimensionmember.sql✨ Builds the VisitType dimension from source event data while filtering out specific excluded event identifiers.writesDimensionMember· readsSourceEvent - Load OAS Reasonsql
load_dimension_members__load_oas_reason.sqldisabled✨ Builds a list of unique OAS reason codes and display names for use as custom property dimensions.writesCustomPropertyMember· readsTransformCustomProperties - Load Statesql
load_dimension_members__load_state.sql✨ Updates state dimensions using facility location data to allow for geographic segmentation of healthcare services and revenue.writesDimensionMember· readsFacility
▶Load-Visitssub-job · 18↗Runs visit and eligibility processing
- Insert TempVisitBatchInsertTimesql
load_visits__insert_tempvisitbatchinserttime.sql✨ Creates a temporary table to record the current timestamp for tracking when a batch of visits was loaded.writesTempVisitBatchInsertTime - LoadVisitBatchLoadVisitBatchLoad scheduled visits and associated charges into the processing pipeline using custom SQL queries for pricing estimation and quote generation.writes
QuoteBatchVisitBatch· readsDimensionMemberPatientTransformVisitTransformVisitInsuranceVisitVisitDeltatransformvisitchargevisitdelta↳ feeds from Sync QuoteBatch,VisitBatch,CustomPropertyMapBatchcustom visit querycustom charge queryauto service type queryauto coverage queryadd zero charge - Applies matching charge bundles to batch visits based on facility and insurance to ensure consistent pricing across service types.DISABLED
- Load TempBatchVisitsql
load_visits__load_tempbatchvisit.sqldisabled✨ Populates a temporary table with visit and insurance details to identify which appointments require insurance eligibility verification.writesTempBatchVisit - Verifies insurance eligibility and retrieves detailed benefits information for visits by making real-time API calls to insurance providers.· reads
InsuranceServiceTypeMapSourceEventTypeServiceTypeMapTempBatchVisitcustom queryDISABLED - Verifies insurance eligibility and retrieves detailed benefits information for visits by making real-time API calls to insurance providers.· reads
CommonDataEligibilityTxnInsuranceServiceTypeMapSourceEventTypeServiceTypeMapTempBatchVisitTempVisitBatchInsertTimecustom queryDISABLED - Populates insurance benefits like copays and deductibles on batch visits using data from eligibility verification transactions.DISABLED
- CalculateVisitsflowCompute the good-faith estimate (CalculateVisits).· reads
DimensionMemberVisitBatch↳ feeds from LoadVisitBatchcustom querycalculate chargesbatch mode ▶Load-Custom-Propertiessub-job · 10↗Load custom properties from Load-Visits
- UpdateVisitCustomPropertyFromQueryUpdateVisitCustomPropertyFromQuerydisabled· reads
InsuranceSearch_ResultTransformVisitInsuranceCoverageStatusVisitBatchcustom visit update custom property queryDISABLED - Custom Properties: OtherActiveCoverageUpdateVisitCustomPropertyFromQuerydisabled· reads
TransformVisitInsuranceCoverageStatusVisitBatchcustom visit update custom property queryDISABLED - Custom Properties: OtherInactiveCoverageUpdateVisitCustomPropertyFromQuerydisabled· reads
TransformVisitInsuranceCoverageStatusVisitBatchcustom visit update custom property queryDISABLED - Custom Properties: Update MostRecentBatch to clear the previous recordsUpdateVisitCustomPropertyFromQuerydisabled· reads
CustomPropertyMapVisitcustom visit update custom property queryDISABLED - Custom Properties: MostRecentBatchUpdateVisitCustomPropertyFromQuerydisabledauto visit update custom property queryDISABLED
- Custom Properties: OAS ReasonUpdateCustomPropertyFromQuerydisabled· reads
CustomPropertyMapsourceappointmentchaintransformcustompropertiesvisitbatchvisitdeltaentity=Visitcustom property key=OASReasonauto querybatch modeDISABLED - Custom Properties: Has BundleUpdateCustomPropertyFromQuerydisabled· reads
CustomPropertyMapTransformVisitChargeVisitBatchvisitdeltaentity=Visitcustom property key=HasBundleauto querybatch modeDISABLED - Custom Properties: GFE ReasonUpdateCustomPropertyFromQuerydisabled· reads
CustomPropertyMaptransformcustompropertiesvisitbatchvisitdeltaentity=Visitcustom property key=GFEReasoncustom querybatch modeDISABLED - Custom Properties: Set LetterPrinted to NoUpdateCustomPropertyFromQuery· reads
CustomPropertyMapVisitBatchentity=Visitcustom property key=LetterPrintedcustom querybatch mode - UpdateCustomPropertyFromQuery (GFEDueDate)UpdateCustomPropertyFromQuery· reads
SourceAppointmentSourceAppointmentChainTransformVisitVisitBatchentity=Visitcustom property key=GFEDueDatecustom querybatch mode
- AutoSendLettersAutoSendLettersGenerates and sends patient cost estimate letters for visits in the batch table based on configured qualification and communication rules.
- Synchronizes processed data from batch staging tables to production tables, updating visits, quotes, and custom property mappings.writes
CustomPropertyMapQuoteVisitVisitChangeEvent· readsCustomPropertyMapBatchQuoteBatchVisitBatchVisitChangeEventBatch↳ feeds from LoadVisitBatchsync quotesync visitsync custom property mapsync visit change event - SearchForInsuranceflowdisabledFind coverage for visits missing insurance.· reads
EligibilityStatusMapEligibilityTxnBatchcustom querytable name=InsuranceSearch_Resultbatch modeDISABLED - ExportBatchVisitEligibilityflowdisabledPush the batch to the app / export results.· reads
EligibilityTxnTempVisitBatchInsertTimecustom querytable name=BatchEligibility_Resultsarchive table name=BatchEligibility_ResultsArchiveDISABLED ▶Generate-Reportsub-job · 5↗Generates the Daily Export report and pushed data into the Archive report for Load-Visits and Load-Visits-Reprocess
- Load TempEligibilityReportsql
generate_report__load_tempeligibilityreport.sql✨ Builds a temporary reporting table consolidating the latest insurance eligibility transaction details, member IDs, and coverage results for each visit.writesTempEligibilityReport· readsTempEligibilityReport - Load TempInsSearchResultssql
generate_report__load_tempinssearchresults.sql✨ Loads a temporary table with successful insurance search results to identify and rank new coverage found for patients.writesTempInsSearchResults - Generate Reportsql
generate_report__generate_report.sql✨ Generates the final eligibility report by consolidating primary, secondary, and tertiary insurance results with any newly discovered coverage information.writesBatchEligibility_ResultsReport - Create BatchEligibility_ResultsReportArchive if doesn't existsql
generate_report__create_batcheligibility_resultsreportarchive_if_doesn_t_exist.sql✨ Builds the archive table for storing historical insurance eligibility and benefit results if it does not already exist. - Archive Reportsql
generate_report__archive_report.sql✨ Updates the eligibility results archive by replacing existing records with the latest report data for specific visits and dates.writesBatchEligibility_ResultsReportArchive· readsBatchEligibility_ResultsReport↳ feeds from Generate Report
- Load TransformVisitInsuranceCoverageStatussql
load_visits__load_transformvisitinsurancecoveragestatus.sqldisabled✨ Updates insurance coverage status for visits by linking them to the most recent successful or failed eligibility verification transactions. - Load VisitDetailsql
load_visits__load_visitdetail.sqldisabled✨ Loads comprehensive visit details including patient demographics, provider information, and insurance coverage status into a reporting table.writesVisitDetail· readsBatchEligibility_ResultsArchiveCustomPropertyMapEligibilityTxnPatientTransformVisitVisitVisitDeltaVisitDetail ▶Load-Visit-Factsub-job · 2↗
- Create Visit Factsql
load_visit_fact__create_visit_fact.sql✨ Creates the VisitFact table to store aggregated visit data, including scheduling, insurance, and payment status, if it does not exist. - Load Visit Factsql
load_visit_fact__load_visit_fact.sql✨ Refreshes the VisitFact table by deleting and re-inserting records for visits with recent updates, payments, eligibility checks, or letters.writesVisitFact· readsEligibilityStatusMapEligibilityTxnPatientLetterPaymentTxnVisit
- UpdateDashboardTablesUpdateDashboardTablesRegenerates optimized aggregate tables for all dashboard reports and updates their definitions to reflect the latest source data.
- Data-Integrity-Reportsub-job
markers
Why it classified as flow
Front-of-house actions found anywhere in the tree (including nested sub-jobs).