Ops Alert Emails · Classified

Pipeline failures, last 6 months

Every ops/alert email — batch job failures, SQL Server alerts, Dagster+ asset alerts, and SQL Beacon health checks — parsed into a queryable substrate. The story underneath the volume: a handful of chronic feeds and transport drops pulling from a few flaky vendors.

4,694

alert emails

4,120

job failures

1,328

transport failures

customers affected

87%

HTTP retries exhausted

99%

joined to legacy-export tenant profile

Window 2025-09-09 → 2026-06-16. Counts are real (Gmail messages.list, not thread-grouping). Built from the alert substrate as of 2026-06-16; aggregates only, no patient data. Failing customers are joined to the legacy-export bundle to attribute each to its source systems & product.

4,120 alert emails ▸ 1,388 incident-days ▸ 72 distinct jobs 66% of emails are same-day retries

What's regressing right now

⚠ RegressionNWDerm · FHIR-Processingselect.nextech-api.com

389

failures in 2026-06

▲ 730% vs 46.9/mo baseline

monthly

SONDJFMAMJ

94% retries exhausted

446 transport failures

server_error422http_52411service_unavailable8

by hour (UTC)spread across the day — not a maintenance window

Transport failures by vendor, per month — the spike is one vendor.

nextechappsmmapiapisftpother

The four alert streams

4,120

Batch job failures

legacy pipeline jobs

2025-09-09 → 2026-06-16

423

SQL Beacon health checks

daily per-server checks

2026-01-27 → 2026-06-16

111

SQL Server Agent alerts

severity 016/017/020

2025-09-17 → 2026-06-16

Dagster+ asset alerts

new stack (since May)

2026-05-12 → 2026-06-15

Who's failing — by source system & product

Each failing tenant joined to its legacy-export profile, so failures roll up to the EHR family / source system they pull from and the product they run. The transport-drop subset is overlaid — it concentrates on the HL7 & ModMed file feeds and the FHIR API.

By EHR family / source system

A tenant pulling from several sources counts toward each — this reads as “failures from tenants touching X”, not a partition.

HL7 v2 messages7 tenants

1,891822 transport

FHIR API2 tenants

1,203478 transport

ModMed2 tenants

755481 transport

MSN LLC (radiology service bureau)1 tenant

56739 transport

NextGen6 tenants

38549 transport

(none recorded)4 tenants

3688 transport

eClinicalWorks (ECW)4 tenants

2092 transport

Phygen2 tenants

110

athenahealth2 tenants

9316 transport

X12 EDI claims / remits3 tenants

AWS QuickSight export1 tenant

Experity (CCP)1 tenant

Snowflake (varies)1 tenant

By product

Product is single-valued per tenant — a clean split.

ClarityFlow9 tenants

2,5931,222 transport

ClarityFlow, RevFind6 tenants

1,025106 transport

RevFind12 tenants

447

(unknown)1 tenant

Off-export — running on the new stack

4 tenants failing with no legacy pipeline profile (33 failures): AVP, MedWise, OENT, TheOregonClinic. These are the migration tell — failures from tenants the legacy export no longer describes.

all job failures transport-layer subset (remote data-pull drops)

Cutting through the noise — one cell per day a job failed

Raw email counts are dominated by retry storms. At day granularity the real shape shows: a few feeds fail every day (chronic), others detonate in short bursts.

SepOctNovDecJanFebMarAprMayJun

NWDerm FHIR-Processing

EugeneGI Flow-MM-HL7-Processing

RIA Flow-HL7-Processing

MannEye Flow-FHIR-Processing

SWOrofacial Flow-HL7-Processing

SSR Daily-Processing

SeaRad Daily-Visit-Processing-HL7

USHP RevFind-Daily-Processing-Accounts

OrthoAlliance Daily-Accounts-Processing

PAA Daily-Processing

Partnercare Revfind-Daily-Processing

SGP RevFind-Daily-Processing

each cell = one day a job failed fewer more emails/day

Alert amplification — emails vs. real incident-days

Jobincident-days ● ───── ○ emails

NWDerm FHIR-Processing

33→764 23×

SWOrofacial Flow-HL7-Processing

44→282 6×

RIA Flow-HL7-Processing

86→534 6×

PAA Sync-Patient-Portal-Data

8→45 6×

SSR Daily-Processing

39→190 5×

SeaRad Sync-Patient-Portal-Data

9→42 5×

SeaRad Daily-Visit-Processing-HL7

17→71 4×

EugeneGI Flow-MM-HL7-Processing

155→599 4×

MannEye Flow-FHIR-Processing

170→383 2×

OneGI RevFind-Daily-Processing

39→41 1×

Noisiest jobs — chronic vs. bursty

Customer · Job	Feed	Emails	Active days	Per day	Shape
NWDerm FHIR-Processing FHIR API	fhir	764	33	23.2	bursty
EugeneGI Flow-MM-HL7-Processing HL7 v2 messages	hl7	599	155	3.9	chronic
RIA Flow-HL7-Processing HL7 v2 messages	hl7	534	86	6.2	bursty
MannEye Flow-FHIR-Processing FHIR API	fhir	383	170	2.3	chronic
SWOrofacial Flow-HL7-Processing HL7 v2 messages	hl7	282	44	6.4	bursty
SSR Daily-Processing ClarityFlow	other	190	39	4.9	chronic
SeaRad Daily-Visit-Processing-HL7 HL7 v2 messages	hl7	71	17	4.2	chronic
USHP RevFind-Daily-Processing-Accounts NextGen	other	71	70	1	chronic
OrthoAlliance Daily-Accounts-Processing HL7 v2 messages	other	63	62	1	chronic
Partnercare Revfind-Daily-Processing eClinicalWorks (ECW)	other	58	58	1	chronic
PAA Daily-Processing Phygen	other	58	58	1	chronic
SGP RevFind-Daily-Processing AWS QuickSight export	other	51	51	1	chronic
PAA Sync-Patient-Portal-Data Phygen	portal_sync	45	8	5.6	bursty
SeaRad Sync-Patient-Portal-Data HL7 v2 messages	portal_sync	42	9	4.7	chronic
OneGI RevFind-Daily-Processing ModMed	other	41	39	1.1	chronic

Transport explorer — remote data-pull failures

The shape underneath the noise: a remote pull failing at the transport layer. SFTP/WinSCP drops on the HL7 file feeds; HTTP 5xx / 429 / timeouts on the FHIR & API feeds. Filter by protocol, feed family, or vendor.

Protocol

Feed

Vendor

1,328 failures · 38% retries exhausted

job failurestransport

Vendor / host	Failures	Exhausted	Top error categories
(sftp)	744	0%	connection_closed 188too_many_connections 96timeout 23
select.nextech-api.com	445	95%	server_error 422http_524 11service_unavailable 8http_520 4
apps.maxrte.com	96	93%	http_429 68service_unavailable 11bad_gateway 8server_error 5
mmapi.ema-api.com	32	3%	gateway_timeout 18http_403 8bad_gateway 6
api.availity.com	11	0%	http_401 9http_400 2

Infrastructure — DB health & server alerts

SQL Beacon — days flagged per server

10.1.104.10		83/141
PROD-DB-3		74/141
STAGING-DB-1		51/141
PROD-DB-1		42/141
STAGING-DB-3		33/141
PP-PROD-DB-1		28/141
PP-PROD-DB-2		21/21
PROD-DB-4		21/21
PROD-DB-2		21/21
STAGING-DB-2		15/23
STRAIGHTPATH-JU		7/141

SQL Server Agent alerts

PROD-DB-1	OS / path (016)	52
PROD-DB-3	OS / path (016)	41
PROD-DB-1	fatal conn (020)	11
PROD-DB-3	fatal conn (020)	3
PROD-DB-1	resource (017)	1