Alerts
An alert watches one query's result and tells you when it crosses a line.
How it works
An alert is checked after its query runs, not on a timer of its own. An alert on a query that is not scheduled only fires when somebody runs the query — which is the single most common surprise, and the fix is to give the query a schedule.
It reads one column — its value in the first row, or its minimum or maximum across rows — compares it, and is then either triggered or ok. Notifications are sent on the change between those states, not on every check, so a metric sitting above its threshold for a day is one message rather than a hundred.
Setting one up
- Write a query returning the number you care about. One row is easiest.
- New Alert, choose the query, the column, the comparison and the value.
- Add destinations — email, Slack, a webhook, and others.
- Optionally write a custom template for the message.
Rearm
How long before a still-triggered alert notifies again. Zero means it tells you once and stays quiet until it recovers and trips a second time.
Attachments
An alert can carry pictures of up to five dashboards or queries, rendered by a headless browser. It is off unless a renderer is configured, and the published image does not include one — it runs in its own optional container.
Any dashboard can be pictured; it does not have to be shared first. The renderer is let in with a pass good for that one dashboard, for five minutes, read-only, and spent as soon as the picture is taken. Access is checked against the alert’s owner each time, so an attachment stops arriving if they lose sight of it.
Email and Slack (app) carry the pictures; the Slack webhook cannot. An incoming webhook has no way to upload a file, so an alert sent through it says a threshold was crossed and shows nothing — which for an alert on a chart is most of the information missing. The Slack (app) destination posts as the app an administrator connected under Settings → Slack and uploads them. Both destinations exist: the webhook needs nothing but a URL, and this needs the app.
A picture that will not upload never stops the alert — the threshold was still crossed and somebody still needs to know — but the message says which one is missing, because an alert about a chart arriving with no chart and no explanation reads as the alert being broken.