Alerts and notifications

Errors → Alerts (editors create and change alerts; anyone can mute one for themselves). Two kinds of rule, the same as Sentry's:

  • Issue alerts: something happens to an issue → notify. Each alert is about one issue.
  • Metric alerts: a metric over a time window crosses a threshold → an incident opens, and resolves when the metric is back to normal.

Both notify through the same actions: email, Slack, Microsoft Teams, Discord, PagerDuty, Opsgenie, signed webhooks, and (issue alerts only) a Linear / Jira ticket. Every notification is in the Activity tab with its status, attempts and error.

Issue alerts: when / if / then

When (any of, or all of):

ConditionChecked
A new issue is createdas the first event arrives
A resolved issue comes back (regression)as the event arrives
An issue escalates: at least a day old, 10+ events in the last hour and 5× its hourly average over the week beforeevery minute; at most once a day per rule and issue
An ignored issue is reopenedwhen a teammate (or issue workflow) reopens it
More than N events in a window (1 minute to 1 day), or N% more than the same window an hour / day / week / 30 days earlierevery minute
More than N users affected in a windowevery minute
More than N% of sessions affected in a windowevery minute, from release health sessions

With all of, at most one of the first four can be used (two state changes never happen at once); frequency conditions must hold at that moment too. Frequency-only "all of" rules fire when every count holds.

If (optional; all / any / none of): checked against the issue and its latest event.

FilterMatches
Levelthe event's level is at least / exactly / at most debug < info < warning < error < fatal
Tagkey equals, does not equal, contains, does not contain, starts with, ends with, is set, is not set. The SDK's tags, plus built-ins: environment, release, level, platform, handled, mechanism, browser, os, device, url, transaction, server_name, logger, dist, user.id, user.email, user.username, error.type
Releasethe latest release seen in the environment, a release, or a pattern (web@2.*)
Categoryerror, performance, cron, uptime, feedback
Agethe issue is older / newer than N minutes, hours, days or weeks
Assigned tounassigned, a member or a team (from issue workflow)
Platformjavascript, python…
Handledhandled (caught) or unhandled (crashed). Events without the information count as handled
Times seenthe issue has happened at least N times

Then: one or more actions, and notify at most once per issue every … (the action interval: every time, 5 minutes … 1 week). Ignored issues never alert.

  • Email: members you pick, typed addresses, issue owners and suggested assignees (when nobody owns the issue: every member, or no one), a team, or every member of the workspace. One email goes to everyone the actions reach. Members who muted the rule, or turned issue alerts off for the project in Settings → Notifications, are left out (typed addresses that are theirs too).
  • Slack, Teams, Discord, webhooks: a message per alert in the connected channel. Slack channels of the Guidely Slack app also get buttons: Resolve (or in the next release), Ignore… (forever, 1 h, 24 h, until 100 more events), Assign to me and Assign to…, which act as the Guidely member with the clicking Slack user's confirmed email, with their permissions, and update the message. In workspaces that require two-step verification or enforce single sign-on the buttons are left out (Slack can't check that sign-in) and the alert posts with its links only: see Why are the buttons missing?
  • PagerDuty: an Events API v2 trigger with dedup_key = guidely-issue-<issueId> and severity from the issue level (fatal → critical, error → error, warning → warning, else info) or the one you pick. Resolving the issue in Guidely sends resolve; a regression triggers it again.
  • Opsgenie: an alert with alias = guidely-issue-<issueId> and priority from the level (P1 … P5) or the one you pick; resolving the issue closes it.
  • Ticket: one Linear / Jira ticket per issue (test sends never open tickets).

The editor estimates how often the rule would have fired in the last 7 days (new issues, event and user counts over fixed windows, the action interval per issue, filters on each issue's latest event). Regressions, escalations and reopened issues can't be replayed from stored events and aren't counted.

Send test notification sends a test through every action at once (mutes and preferences don't apply).

Metric alerts

SourceMetricsFilters
Errorsnumber of errors, users affectedlevel (at least), tags, release, one issue
Performance (transactions)failure rate, p50 / p95 / p99 duration, Apdex, throughput, counttransaction, release
Release healthcrash-free sessions, crash-free usersrelease

Performance metrics read the hourly performance rollups, so their window snaps to whole hours (a 15-minute window reads the current hour). Release health reads the same sessions as the Releases page.

  • Window: 1, 5, 10, 15, 30 minutes, 1, 2, 4 hours or 1 day. Checked every minute (alerts.metrics).
  • Compare: the value itself, or its % change from the same window 1 hour, 1 day, 1 week or 30 days earlier (no value then: no change).
  • Thresholds: above or below; critical, optional warning, optional resolve (default: back under the lowest threshold). Crossing a threshold is strict (> 100 for "above 100").
  • Incident lifecycle: ok → warning / critical opens incident #n; warning ↔ critical; under the thresholds but not yet past the resolve threshold drops critical to warning; past it resolves. Each step is on the incident's timeline and notified: each action listens to critical, warning or both; the resolve goes to the actions that heard the incident's worst status. A source with no data keeps the state.
  • A rule muted for everyone still tracks incidents but sends nothing.

The incident page shows the metric around the incident with the thresholds, the timeline and every notification.

Muting

From the list or a rule's page: for me (you stop getting its emails) or, for editors, for everyone (nothing is sent), for 1 hour, 4 hours, 1 day, 1 week or until unmuted.

Channels (Integrations)

One connection per channel or service; each lists the alerts that use it, has Send test and its delivery log. Secrets (webhook URLs, routing keys, API keys) are stored like Slack's and returned masked; sending a masked value back keeps the stored one.

IntegrationSetupSent
Slacka channel of the Guidely Slack app, or an incoming webhook URLBlock Kit message (with triage buttons through the app)
Microsoft Teamschannel → Workflows → Post to a channel when a webhook request is received (or an Incoming Webhook)Adaptive Card
Discordchannel → Integrations → Webhooksan embed; allowed_mentions off
PagerDutyservice → Integrations → Events API V2 → Integration Key (one connection per service)Events API v2 trigger / resolve
Opsgenieteam → Integrations → API → key; US or EUcreate / close alert
Alert webhooksyour HTTPS endpoint; the signing secret is shown onceSentry-style JSON

Teams, Discord and webhook URLs are called through the SSRF guard (public hosts, standard ports, no redirects). PagerDuty and Opsgenie go to their API hosts (PAGERDUTY_EVENTS_URL, OPSGENIE_API_URL).

Webhook payload

{
  "action": "triggered",
  "data": {
    "event": { "event_id": "…", "issue_id": "ei_…", "title": "TypeError: …", "level": "error", "platform": "javascript",
               "environment": "Production", "release": "web@2.4.1", "datetime": "…", "tags": [["browser", "Chrome 130"]],
               "user": { "id": "u-1" }, "web_url": "https://…/errors/ei_…", "issue_url": "https://…/errors/ei_…" },
    "issue": { "id": "ei_…", "shortId": "NORTHWIND-42", "title": "TypeError: …", "culprit": "…", "level": "error", "status": "open",
               "platform": "javascript", "issueCategory": "error", "count": "48", "firstSeen": "…", "lastSeen": "…", "permalink": "https://…" },
    "triggered_rule": "Production errors",
    "trigger": "new"
  },
  "project": { "id": "…", "name": "Northwind CRM", "slug": "northwind" },
  "url": "https://…/errors/ei_…",
  "environment": "Production",
  "timestamp": "…",
  "actor": { "type": "application", "id": "guidely", "name": "Guidely" }
}

action is triggered (issue alerts), critical / warning / resolved with data.metric_alert (metric alerts: id, identifier, title, status, metric, value, threshold, date_started, alert_rule), or test. Headers: X-Guidely-Signature: sha256=<hex> and Sentry-Hook-Signature: <hex> (HMAC-SHA256 of the raw body with the signing secret), Sentry-Hook-Resource: event_alert | metric_alert, Sentry-Hook-Timestamp, X-Guidely-Event: alert.trigger | alert.resolve | alert.test. Code that verifies Sentry integration webhooks works unchanged.

Delivery and retries

Each notification is an AlertDelivery row sent by the worker's outbox loop: 2xx is sent; 408, 429, 5xx and network errors retry after 30 s, 1, 2 and 4 minutes (5 attempts); other 4xx and bad configuration (a removed integration, an unreadable secret) fail at once. Failed deliveries can be sent again from the Activity tab. Emails go through the mail outbox; their status is the email's.

Notification preferences (Settings → Notifications)

Per member, a default for every project plus per-project overrides:

KindDefaultCovers
Issue and metric alertsonalert emails to you as a member, owner or team (and typed addresses that are yours)
Issue workflowonissues you're subscribed to: assigned, comments, mentions, regressions
Deploys and releasesoffrelease / deploy notifications
Weekly reportoffthe Monday summary per project

plus the time zone weekly reports follow (UTC when unset).

Once a workspace admin has added the Guidely Slack app, each kind also has a Slack DM switch (off for every kind until turned on, with its own per-project overrides), and the page shows which Slack account your email matches. Alert DMs carry the same buttons as channel alerts (none where Slack actions are off, as above). In HIPAA workspaces DMs only say what kind of thing happened, with a link.

Weekly report

On Mondays from 9:00 in the member's time zone (the hourly reports.weekly job), once per member, project and week, to members who turned it on: production errors and issues against the week before, new and resolved issues, top 5 issues by events and by users, the slowest transactions (p95) and the crash-free session rate. The footer's Unsubscribe link (a signed email-link token, no login) turns the report off for that project. Preview it in Admin → Email templates.

API

GET/POST /api/v1/projects/:id/alert-rules, GET/PATCH/DELETE /api/v1/alert-rules/:idissue rules (conditions, conditionMatch, filters, filterMatch, actions, cooldownMinutes, ownerId; the original onNewIssue / onRegression / spikeThreshold / spikeWindowMinutes / emailRecipients / slackIntegrationId / ticketIntegrationId fields are still accepted and returned)
POST /api/v1/projects/:id/alert-rules/previewthe 7-day estimate for an unsaved rule
GET /api/v1/projects/:id/alert-rules/deliveries?ruleId=fired alerts with their notifications
POST /api/v1/alert-rules/:id/test, POST /api/v1/metric-alerts/:id/testtest sends
POST/DELETE /api/v1/alert-rules/:id/snooze, …/metric-alerts/:id/snoozemute `{ scope: "me" \"everyone", until? } / unmute ?scope=`
GET/POST /api/v1/projects/:id/metric-alerts, GET/PATCH/DELETE /api/v1/metric-alerts/:idmetric rules
POST /api/v1/projects/:id/metric-alerts/previewthe metric over recent windows
GET /api/v1/projects/:id/metric-incidents, GET /api/v1/metric-incidents/:idincidents, with timeline, chart series and notifications
GET /api/v1/projects/:id/alert-activity, POST /api/v1/alert-deliveries/:id/retrythe activity log
GET /api/v1/projects/:id/alert-optionsmetric sources (connected or not) and teams
POST /api/v1/integrations/:id/test-alertIntegrations → Send test
GET/PATCH /api/v1/notification-preferencesyour preferences

Rules saved before when / if / then

Rules created before (new issue / regression / spike, email + Slack + ticket) keep doing exactly what they did. Migration 20261222000001_alert_rules_v2_backfill writes their conditions and actions (spikeThreshold T within W minutes, "at least T", becomes "more than T − 1 events in W minutes"), and rows without them are read from the original columns, which stay in step with every save.

Code map

  • apps/web/src/server/services/error-alerts.ts: issue rules, firing (ingestion, status changes, the minutely frequency / escalation job), preview, mutes, history.
  • alert-rule-schema.ts (validation, old-rule mapping), alert-filters.ts (filters), alert-actions.ts (recipients, dispatch), alert-channels.ts (payloads, sending, retries), alert-activity.ts (log, tests), alert-snooze.ts, alert-hooks.ts (issue workflow hooks), alert-metrics.ts (metric sources), metric-alerts.ts, notification-preferences.ts, weekly-report.ts.
  • Jobs: alerts.spikes (every minute), alerts.metrics (every minute), reports.weekly (hourly); deliveries go out with the outbox loop.

For other parts of error monitoring

  • Issue workflow registers assignment, ownership and teams with registerIssueWorkflowHooks (alert-hooks.ts), calls onIssueStatusChanged(issueId, from, to) when it changes an issue's status (resolve, unignore, archive-until-escalating), may call onIssueChanges(projectId, env, [{ issueId, trigger: "escalating" }]) for its own escalation, and checks shouldNotify(userId, projectId, "workflow") before emailing.
  • Performance and release health plug into setMetricProvider (alert-metrics.ts documents the expected transactionMetric, crashFreeRate, sessionCount and slowestTransactions signatures).
  • Monitors can notify through dispatch / queueDeliveries with an AlertMessage and the same actions.