ReliabilityOps ExpertiseToolsField NotesContact
← Field Notes HADR · Deep dive

Everything that doesn't fail over

July 2026 · 9 min read · deep dive

The availability group did its job. The failover was clean, the databases came up SYNCHRONIZED on the new primary, the health dashboard went green. And the application was down hard.

This is the most common shape of a "failover worked but nothing works" incident, and the reason is one sentence worth tattooing somewhere: an availability group fails over the contents of the databases, and almost nothing else. Everything that lives at the server level. In the master and msdb databases. Stays behind unless something copied it there ahead of time. Here are the four that bite, in the order they bite.

1. Logins. And their SIDs

The database user travels inside the AG. The server login it maps to does not. If nobody provisioned logins on the DR replica, the app gets "login failed for user X". For a user that plainly exists inside the database. Worse and subtler: the login exists, but with a different SID than the one baked into the database user, so the mapping is silently broken.

-- the tell: same name, different SID
SELECT 'SERVER' src, name, sid FROM sys.sql_logins WHERE name='app_orders'
UNION ALL
SELECT 'DATABASE', name, sid FROM OrdersDB.sys.database_principals WHERE name='app_orders';

Tonight's fix is ALTER USER app_orders WITH LOGIN = app_orders to remap. The durable fix is scripting logins onto every replica with matching SIDs, sp_help_revlogin, or Copy-DbaLogin from dbatools. And then auditing for drift so it never recurs.

2. Agent jobs

Backups, index maintenance, ETL, the little cleanup job someone wrote in 2019 that everything quietly depends on. All of it lives in msdb. Fail over without syncing jobs and your new primary is running naked: no backups (your RPO is now silently infinite), no maintenance, no ETL. Nothing errors. It just stops happening, and you find out days later.

3. Linked servers

This is the dangerous one, because the failure mode isn't "it errors". It's "it succeeds against the wrong place." A linked server still pointed at the old primary means any job or proc that uses it is now reaching back into the site you just failed away from. Best case, prod is down and the job errors. Worst case, prod is half-alive and DR quietly writes into it. And now your data lives in two places that will never reconcile. That's split-brain, and it's the kind of thing that ends up in a regulatory finding.

4. Credentials & configuration

Database-scoped credentials, the sp_configure surface, TDE certificates, endpoint permissions. The settings nobody remembers because they were set once, years ago. These don't announce themselves. They just make one feature quietly not work until someone traces it back.

The move that catches all four

You don't find these by running a check on the new primary in isolation, because "there are 40 logins here" tells you nothing. You find them by running the same read-only audit on prod and on DR and diffing the two. A login on prod that's missing on DR is a finding. A linked server whose data_source differs is a finding. Drift is invisible until a failover makes it an outage. So make drift a thing you can see on a quiet Tuesday, not a thing you discover at 2am.

The whole philosophy in one line: the failover is the easy part. The AG handles it. The hard part is everything the AG leaves behind, and the only reliable way to manage it is to check for drift before you need the failover, not during it.
The kit runs this exact audit. Read-only, and diffable across replicas.
Open the Triage tool →