Legacy Pipeline Atlas worker-1snapshot 2026-07-15

Batch / file feed · RevFind pipeline

MNEye · Daily-Processing-Source-Data

pipeline MNEye/Daily-Processing-Source-Data · db MNEye_ProdMirror · updated 2026-02-06
classification: other · 0 Flow markers

trigger

How it's scheduled

Daily batch — once at 03:15

The Batch / file feed arrives as a scheduled drop, so one overnight pass is enough: it ingests the day's file and works the whole batch to process the visits. A tighter loop would just re-scan the same file.

Scheduled taskCadenceServerStateCoordination
\MDC\Daily - Processing - MNEyedaily at 03:15prod-db-3enabledthen runs \MDC\Daily - Processing - MNEye - Accounts, \MDC\Daily - Processing - MNEye - Visits

shape

Against the canonical Flow spine

Every Flow pipeline is a variation on one shape. Lit stages are the ones this pipeline runs; the labels match the capability lens.

Ingest
Batch / file feed
→
Eligibility & benefits
→
Estimate
→
Letters
→
Work queue & sync

computed concepts · from the SQL

What it computes

Business concepts mined from this pipeline's SQL text — the meaning behind the typed steps. Each links to the concept lens (who else computes it).

Fee schedule / contractCPT / procedure logic

task tree

What the pipeline runs

Tasks in order, each with what it does — typed stepsfrom the MDClarity task docs (click to open the reference), sub-jobsyou can expand inline (▶) to reveal their own tasks, and SQL steps summarized by Gemini (✨). Every step shows the tables it writes/reads and which earlier step it feeds from(sqlglot lineage).

other

Daily-Processing-Source-Data

5 tasks · MNEye
sub-job / Flow action SQL / native step▶ expand a sub-job inline · ↗ open it as its own page · a typed step opens its doc
  1. ▶Load-NextGen-Datasub-job · 16↗
    1. DownloadDownload
      Downloads files from remote FTP or SFTP servers to local directories with support for archiving and parallel transfers.
      server=MNEye_ProdMirror-mdcssh-sftpremote directory=/local directory={SourceDataFolder}
    2. Download FilesExecuteProcessdisabled
      command={SSHToolPath}\SSH.exeserver=MNEye_ProdMirror-mdcssh-sftpDISABLED
    3. Load AppointmentsLoadFile
      writes ImportAppointments
      data file spec=ImportAppointmentsimport mode=Replace
    4. Load PatientsLoadFile
      writes ImportPatients
      data file spec=ImportPatientsfile path={SourceDataFolder}\ExportPatient.txtimport mode=Replace
    5. Load Patient InsuranceLoadFile
      writes ImportPatientInsurance
      data file spec=ImportPatientInsuranceimport mode=Replace
    6. Load PayerLoadFile
      writes ImportPayer
      data file spec=ImportPayerfile path={SourceDataFolder}\ExportPayer.txtimport mode=Replace
    7. Load ProvidersLoadFile
      writes ImportProviders
      data file spec=ImportProvidersfile path={SourceDataFolder}\ExportProviders.txtimport mode=Replace
    8. Load ChargeMasterLoadFile
      writes ImportChargeMaster
      data file spec=ImportChargeMasterimport mode=Replace
    9. Load EncountersLoadFile
      writes ImportEncounters
      data file spec=ImportEncountersfile path={SourceDataFolder}\ExportEncounters.txtimport mode=Replace
    10. Load ChargesLoadFile
      writes ImportCharges
      data file spec=ImportChargesfile path={SourceDataFolder}\ExportCharges.txtimport mode=Replace
    11. Load AdjustmentsLoadFile
      writes ImportAdjustments
      data file spec=ImportAdjustmentsimport mode=Replace
    12. Load TransactionDetailLoadFile
      writes ImportTransactionDetail
      data file spec=ImportTransactionDetailimport mode=Replace
    13. Load Encounter InsuranceLoadFile
      writes ImportEncounterInsurance
      data file spec=ImportEncounterInsuranceimport mode=Replace
    14. Load LocationsLoadFile
      writes ImportLocations
      data file spec=ImportLocationsfile path={SourceDataFolder}\ExportLocation.txtimport mode=Replace
    15. Load EventLoadFile
      writes ImportEvent
      data file spec=ImportEventfile path={SourceDataFolder}\ExportEvent.txtimport mode=Replace
    16. Archive FilesExecuteProcess
      command=cmd.exe
  2. ▶Load-Source-Tablessub-job · 13↗
    1. Merge SourceAppointmentMergeSql
      writes SourceAppointment · reads ImportAppointments
      merge data source type=ThisDatabasetable name=SourceAppointmentmerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
    2. Load SourcePatientInsurancesqlload_source_tables__load_sourcepatientinsurance.sql
      ✨ Loads patient insurance data while identifying inactive policies and calculating the coordination of benefits order for each coverage record.
      writes SourcePatientInsurance · reads ImportPatientInsurance
    3. Merge SourcePatientMergeSql
      writes SourcePatient · reads ImportPatient
      merge data source type=ThisDatabasetable name=SourcePatientmerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
    4. Load PatientsLoadPatients
      Upsert patient records into the patient dimension.
      · reads PatientSourcePatient
      ↳ feeds from Merge SourcePatient
      custom patient query
    5. Load SourceEncounterssqlload_source_tables__load_sourceencounters.sql
      ✨ Loads encounter records by transforming imported clinical and financial data into a standardized format for tracking patient visits and balances.
      writes SourceEncounters · reads ImportEncounters
    6. Load SourceEncounterInsurancesqlload_source_tables__load_sourceencounterinsurance.sql
      ✨ Loads and refreshes encounter-level insurance coverage details, including payer information and policy dates, into the source storage table.
      writes SourceEncounterInsurance · reads ImportEncounterInsurance
    7. Load SourceChargessqlload_source_tables__load_sourcecharges.sql
      ✨ Updates source charge records with procedure codes, financial balances, and rendering provider information for specific patient encounters.
      writes SourceCharges · reads ImportCharges
    8. Load SourceAdjustmentssqlload_source_tables__load_sourceadjustments.sql
      ✨ Loads financial adjustment data, including adjudication dates and reason codes, to track modifications made to patient account balances.
      writes SourceAdjustments · reads ImportAdjustments
    9. Load SourceTransactionDetailsqlload_source_tables__load_sourcetransactiondetail.sql
      ✨ Populates detailed transaction history, including allowed amounts and payments, while ensuring data integrity by removing duplicate or outdated records.
      writes SourceTransactionDetail · reads ImportTransactionDetail
    10. Merge SourceLocationsMergeSql
      writes SourceLocations · reads FacilityImportLocations
      merge data source type=ThisDatabasetable name=SourceLocationsmerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
    11. Load Provider (moved to Load-Entities)sqlload_source_tables__load_provider_moved_to_load_entities.sqldisabled
      ✨ Synchronizes provider information, such as NPI and taxonomy codes, by merging new and updated records from multiple source systems.
    12. Load SourcePayersqlload_source_tables__load_sourcepayer.sql
      ✨ Refreshes the master list of insurance payers by importing the latest payer identification and contact data into the source table.
      writes SourcePayer · reads ImportPayerSourcePayer
    13. Merge SourcechargemasterMergeSql
      writes SourceChargeMaster · reads ImportChargeMaster
      merge data source type=ThisDatabasetable name=SourceChargeMastermerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
  3. ▶Load-Charge-Dimensionsub-job · 14↗
    1. TransformChargemaster ExecuteMergeSqlMergeSql
      writes TransformChargemaster · reads SourceChargeMaster
      merge data source type=ThisDatabasetable name=TransformChargemastermerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
    2. Add Missing Codes to Chargemaster (for now hack)sqlload_charge_dimension__add_missing_codes_to_chargemaster_for_now_hack.sql
      ✨ Inserts manual charge code entries into the chargemaster to address missing data for specific practices.
      writes TransformChargemaster · reads TransformChargemaster
      ↳ feeds from TransformChargemaster ExecuteMergeSql
    3. End to end chargemaster updatesqlload_charge_dimension__end_to_end_chargemaster_update.sql
      ✨ Builds the end-to-end mapping between chargemaster schedules, facilities, and insurance records for fee calculation.
      writes FeeScheduleRecordScheduleMapTempChargeSchedules · reads FacilityTempChargeSchedulesTransformChargeMaster
      ↳ feeds from Add Missing Codes to Chargemaster (for now hack), TransformChargemaster ExecuteMergeSql
    4. Load TempChargeSchedulessqlload_charge_dimension__load_tempchargeschedules.sqldisabled
      ✨ Builds temporary charge schedules by determining the effective start and end dates for each practice's chargemaster.
      writes TempChargeSchedules
    5. Load FeeScheduleRecordsqlload_charge_dimension__load_feeschedulerecord.sqldisabled
      ✨ Loads fee schedule records with specific amounts and modifier filters based on practice-specific chargemaster data and schedules.
      writes FeeScheduleRecord · reads SourceChargeMasterTempChargeSchedules
    6. Load ScheduleMapsqlload_charge_dimension__load_schedulemap.sqldisabled
      ✨ Updates the schedule map to link professional and facility fee schedules with their respective facilities and insurance categories.
      writes ScheduleMap · reads FacilityTempChargeSchedules
    7. Load ContractMapsqlload_charge_dimension__load_contractmap.sqldisabled
      ✨ Builds the contract mapping table by defining charge logic and bill types for various service categories across all locations.
      writes ContractMap · reads Facility
    8. Load ScheduleLoadEntities
      writes Schedule · reads ScheduleMapTempChargeSchedules
      ↳ feeds from End to end chargemaster update
      entity type=Schedulecustom query
    9. Load SourceChargeMapsqlload_charge_dimension__load_sourcechargemap.sql
      ✨ Loads a unified charge map by consolidating service-specific charge codes from the source master file for designated practices.
      writes SourceChargeMap · reads SourceChargeMaster
    10. Load ChargeTypesqlload_charge_dimension__load_chargetype.sqldisabled
      ✨ Builds the charge type table by mapping charge codes to display names, bill types, and specific billing rules or modifiers.
      writes ChargeType · reads MedicareRVUScheduleMedicareRVUScheduleMapSourceChargeMap
    11. Merge ChargeTypeMergeSql
      writes ChargeType · reads ChargeTypeMedicareRVUScheduleMedicareRVUScheduleMapSourceChargeMap
      ↳ feeds from Load SourceChargeMap
      merge data source type=ThisDatabasetable name=ChargeTypemerge mode=InsertUpdatebulk copy timeout seconds=300bulk copy use table lock
    12. Load ChargeType Dimensionsqlload_charge_dimension__load_chargetype_dimension.sql
      ✨ Updates the charge type dimension by populating unique charge codes and descriptions for multiple billing categories.
      writes DimensionMember · reads DimensionMemberSourceChargeMap
      ↳ feeds from Load ProcedureCategory and Subcategory DM, Load ProcedureCode Dimension, Load SourceChargeMap
    13. Load ProcedureCode Dimensionsqlload_charge_dimension__load_procedurecode_dimension.sql
      ✨ Loads the procedure code dimension by organizing unique codes and descriptions into a hierarchical structure for reporting and analysis.
      writes DimensionMember · reads SourceChargeMap
      ↳ feeds from Load SourceChargeMap
    14. Load ProcedureCategory and Subcategory DMsqlload_charge_dimension__load_procedurecategory_and_subcategory_dm.sql
      ✨ Builds hierarchical dimensions for procedure categories and subcategories to enable grouped analysis of medical services and billing codes.
      writes DimensionMember · reads SourceChargeMap
      ↳ feeds from Load SourceChargeMap
  4. ▶Load-Entitiessub-job · 3↗
    1. Load FacilityLoadEntitiesdisabled
      writes Facility · reads FacilitySourceLocations
      entity type=Facilitycustom querykey prefix=FDISABLED
    2. Load FacilityLoadEntities
      writes Facility · reads FacilitySourceAppointmentSourceEncountersSourceLocations
      entity type=Facilitycustom querykey prefix=F
    3. Merge ProvidersMergeSql
      writes Provider · reads ImportProvidersSourceEncounterssourceappointment
      sql file=sql/load_entities__merge_providers.sqlmerge data source type=ThisDatabasetable name=Providermerge mode=InsertUpdatebulk copy timeout seconds=300
  5. ▶Load-InsuranceMapsub-job · 1↗
    1. Load InsuranceMapLoadEntities
      writes InsuranceMap · reads InsuranceMapSourcePayer
      entity type=InsuranceMapcustom querykey prefix=IM