professional work

Notification Pipeline: Reliability & Observability

Fixed a notification system that was dropping alerts silently with no way to diagnose why.

Full-Stack Developer · Undisclosed · Oct 2025 — Present

context

Lead notifications were dropping silently. Users weren't getting notified about new leads and there was no log, no error, no trace anywhere in the pipeline to explain why.

the challenge

There were multiple failure modes and none of them were visible. Source lookup could fail silently, a burst of errors would permanently disable notifications instead of retrying, and the node-cron scheduler would die on server restarts without logging anything.

approach

• 1.I added hop-level audit logging before fixing anything else. Once I could see where failures were happening, the causes became obvious.
• 2.I replaced the permanent failure block with a 10-minute cooldown retry. A brief outage no longer permanently kills notifications.
• 3.I moved the background scheduler from node-cron to Inngest cron. node-cron stops silently when the server restarts. Inngest doesn't. I also added a lazy-init fallback in case the Inngest instrumentation doesn't fire on startup.

technical highlights

• —I added a notification_logs table that records every step in the pipeline as a separate row with a status. Failures are now diagnosable by query instead of requiring code changes to see what's happening.
• —There's a 3-source fallback chain. If the primary source fails, it tries the secondary, then the tertiary. The audit log records which source succeeded and which failed.
• —The cooldown retry meant I didn't need manual intervention to restore notifications after a transient outage. Previously it would stay disabled until someone noticed.

result

Notification failures are now visible in the logs. The fallback chain handles source outages. A server restart no longer silently stops background jobs.

technologies