Legacy Pipeline Atlas worker-1snapshot 2026-07-15

sub-job

USDH · Process-EDI

called by Daily-Processing

sub-job USDH/Process-EDI · db USDH_ProdMirror · updated 2025-09-17

task tree

What the sub-job runs

Tasks in order, each with what it does — typed stepsfrom the MDClarity task docs (click to open the reference), sub-jobsyou can expand inline (▶) to reveal their own tasks, and SQL steps summarized by Gemini (✨). Every step shows the tables it writes/reads and which earlier step it feeds from(sqlglot lineage).

sub-job

Process-EDI

3 tasks · USDH
sub-job / Flow action SQL / native step▶ expand a sub-job inline · ↗ open it as its own page · a typed step opens its doc
  1. Download Files from SFTPDownload
    Downloads files from remote FTP or SFTP servers to local directories with support for archiving and parallel transfers.
    server=USDH_ProdMirror-mdcssh-sftpremote directory=/local directory={SourceDataFolder}
  2. ▶Load-Edi-Filessub-job · 1↗
    1. ▶Load-Edi-Filesub-job · 3↗
      1. Bulk Insert File Into Databasesqlload_edi_file__bulk_insert_file_into_database.sqldisabled
        ✨ Loads raw EDI files into staging and identifies transaction types and subtypes based on file naming conventions.
        writes EDIFileStaging · reads SINGLE_CLOB
      2. Upload File to EDIFileStaging tableExecuteProcess
        command=powershell.exe
      3. Archive FilesExecuteProcess
        command=cmd.exe
  3. ▶TransformAndParseEDIsub-job · 5↗
    1. EDITransformsqltransformandparseedi__editransform.sql
      ✨ Loads unique EDI messages from staging into the transformation table while preventing duplicate records.
      writes EDITransform · reads EDIFileStagingEDITransform
    2. EDIBatchsqltransformandparseedi__edibatch.sql
      ✨ Builds a batch of unprocessed EDI messages, excluding empty records, for subsequent parsing and transformation.
      writes EDIBatch
    3. Clean Messages in EDIBatchsqltransformandparseedi__clean_messages_in_edibatch.sql
      ✨ Updates EDI batch messages by replacing specific formatting characters to ensure compatibility with parsing logic after vendor formatting changes.
      · reads EDIBatch
      ↳ feeds from EDIBatch, EDIParser
    4. EDIParserEDIParser
      writes EDIBatch
      table name=EDIBatch
    5. ▶Load-EDI-Metadatasub-job · 3↗
      1. Load EDIFacilityMapsqlload_edi_metadata__load_edifacilitymap.sql
        ✨ Loads the facility map with unique billing and service facility details extracted from electronic claim transactions.
        writes EDIFacilityMap · reads EDIFacilityMapTxn837P
      2. Load EDIProviderMapsqlload_edi_metadata__load_ediprovidermap.sql
        ✨ Builds the provider map by combining rendering provider information from claim transactions with national provider registry data.
        writes EDIProviderMap · reads EDIProviderMapNPIDataTxn837P
      3. Load EDIPayerMapsqlload_edi_metadata__load_edipayermap.sql
        ✨ Builds a master payer mapping table by consolidating payer names and identifiers from remittance and claim transaction records.
        writes EDIPayerMap · reads TransformRemitTxn837P