How to Build an Alerting Pipeline in PagerDuty
An alerting pipeline is only as good as its signal-to-noise ratio. Too many alerts and your team ignores them all. Too few and critical issues go unnoticed. PagerDuty sits at the center of your alerting infrastructure, receiving signals from monitoring tools and routing them to the right responders with the right urgency.
This guide walks you through building a PagerDuty alerting pipeline that minimizes alert fatigue while ensuring critical incidents get immediate attention. You'll configure services, escalation policies, event rules, and notification settings that reflect how your team actually operates — not just the defaults.
Step-by-step guide
Map Your Services and Ownership
Create a PagerDuty service for each independently deployable component your team owns. A service in PagerDuty represents a thing that can break — your API, database, payment processor, CDN, background job system. Each service should have a clear owner (a team, not an individual). Avoid creating one catch-all service for everything, as it prevents meaningful routing and makes on-call rotations impossible to manage.
- Name services consistently using your team's naming convention — match them to your infrastructure or microservice names
- Add a description to each service explaining what it does and what customer impact looks like when it's down
- Tag services with labels like 'customer-facing,' 'internal,' and 'infrastructure' for grouping and reporting
Configure Escalation Policies
Create an escalation policy for each service that defines who gets notified and when. A typical policy has three levels: Level 1 is the on-call engineer (notified immediately), Level 2 is the engineering lead (notified after 15 minutes if unacknowledged), Level 3 is the engineering director (notified after 30 minutes). Set the escalation timeout based on your SLA — if your SLA requires 5-minute acknowledgment, Level 2 should kick in at 5 minutes, not 30.
- Create on-call schedules with weekly rotations and configure them as Level 1 in the escalation policy
- Add a 'last resort' notification target (like a team Slack channel or email group) as the final escalation level
Set Up Event Rules for Intelligent Routing
Use PagerDuty's Event Intelligence or Global Event Rules to transform and route incoming events before they create incidents. Create rules that: suppress known non-actionable alerts (like scheduled maintenance windows), set severity based on alert content (e.g., anything containing 'out of disk' is critical), and deduplicate related alerts so a cascade of failures creates one incident instead of fifty. Event rules are where you turn raw monitoring noise into actionable incidents.
- Create suppression rules for alerts that fire during planned maintenance windows
- Set up deduplication keys so alerts from the same root cause merge into a single incident
- Route alerts from different monitoring tools to the correct service based on tags or source fields
Define Urgency Levels and Notification Methods
Configure two urgency levels for each service: High urgency for incidents that require immediate response (customer-facing outages, data loss, security breaches), and Low urgency for issues that can wait until business hours (performance degradation below SLA, non-critical batch job failures). Map notification methods to urgency: high urgency triggers phone call + push notification + SMS; low urgency triggers email + push notification only. This ensures your team's phones don't ring at 3 AM for non-critical issues.
- Set up 'Support Hours' on low-urgency services so they only page during business hours
- Configure notification rules in each team member's profile to respect their notification preferences
Integrate Monitoring Sources
Connect your monitoring tools to PagerDuty via native integrations or the Events API. For each integration, configure which events should trigger incidents and which should be informational. Common integrations include Datadog (infrastructure metrics and APM), Sentry (application errors), AWS CloudWatch (cloud resource alarms), and custom webhooks from your application's health checks. Each integration should send structured payloads that include enough context for the responder to start diagnosing without opening another tool.
- Use PagerDuty's Events API v2 for custom integrations — it supports severity levels, component fields, and custom details
- Test each integration by sending a test alert and verifying it creates an incident on the correct service with the right urgency
Set Up Response Plays and Status Pages
Create Response Plays for your most common incident types — pre-configured actions that add the right responders, create a Slack channel, and update your status page with a single click. For a database outage, the response play might page the DBA on-call, add the infrastructure lead, create a #incident-db Slack channel, and post a 'Investigating' update to your status page. Having these pre-built saves critical minutes during the first moments of an incident.
- Create response plays for your top five most common incident types based on historical data
- Integrate PagerDuty with your status page provider (Statuspage, Atlassian Status) so updates flow automatically
Common mistakes
Alerting on Every Metric Anomaly
Not every spike in latency or dip in throughput requires human attention. Alert on conditions that require human intervention — thresholds that indicate customer impact or system failure, not statistical anomalies. If an alert fires and the correct response is 'wait and see,' it shouldn't be an alert.
Using the Same Escalation Policy for Everything
A one-size-fits-all escalation policy means either your critical alerts are under-escalated or your low-priority alerts over-escalate. Create separate policies for different service tiers — customer-facing production services need aggressive escalation; internal tools and staging environments can have relaxed policies.
Not Tracking Alert Frequency and Noise Metrics
If you don't measure your alert noise, you can't reduce it. Track alerts per service per week, percentage of alerts acknowledged vs. auto-resolved, and mean time to acknowledge. PagerDuty's Analytics provides these metrics — review them monthly and suppress or tune the noisiest alerts.
Skipping On-Call Handoff Documentation
When an on-call rotation changes, the incoming engineer should know about any ongoing issues, recent changes, and known flaky alerts. Create a handoff document or Slack thread at each rotation boundary summarizing the state of the world. PagerDuty's on-call handoff notifications can automate the reminder.
Tips
Use PagerDuty's Analytics dashboard to identify your noisiest services and top alert sources — fixing the top three noise generators often reduces alert volume by 50%.
Set up a 'Shadow On-Call' rotation where new team members receive the same alerts as the primary on-call but aren't expected to respond, giving them a risk-free learning period.
Create a monthly 'Alert Hygiene' ritual where the team reviews and tunes the top 10 noisiest alerts from the past month.
Use PagerDuty's Slack integration to create dedicated incident channels automatically when high-severity incidents trigger, centralizing communication.
How Vantage helps
Vantage helps product teams incorporate reliability requirements into their product specifications. When writing PRDs for features that affect system reliability, Vantage's context-aware generation ensures that alerting, monitoring, and incident response requirements are captured alongside functional requirements.