路由
amtool config routes test 不计算 active_time_intervals。warning 且 team=platform 在工具里会落到 slack-platform,即使不在 Asia/Shanghai 的工作时间。这是工具的行为,页面没有改时间。
amtool config routes show
Routing tree:
.
└── default-route receiver: default
├── {alertname="Watchdog"} receiver: deadman-webhook
├── {severity="critical"} continue: true receiver: pagerduty-oncall
├── {severity="critical"} continue: true receiver: slack-incidents
├── {team="payments"} receiver: slack-payments
├── {team="orders"} receiver: email-oncall
├── {severity="warning"} receiver: slack-platform
└── {severity="info"} receiver: null
断言
| 用例 | 期望 receivers | 实际 | 结果 |
|---|---|---|---|
| critical_platform_instance_down | pagerduty-oncall,slack-incidents | pagerduty-oncall,slack-incidents | passed |
| critical_orders_fast_burn | pagerduty-oncall,slack-incidents,email-oncall | pagerduty-oncall,slack-incidents,email-oncall | passed |
| critical_payments | pagerduty-oncall,slack-incidents,slack-payments | pagerduty-oncall,slack-incidents,slack-payments | passed |
| warning_orders_slow_burn | email-oncall | email-oncall | passed |
| warning_payments_queue | slack-payments | slack-payments | passed |
| warning_platform_memory | slack-platform | slack-platform | passed |
| info_discarded | null | null | passed |
| watchdog_deadman | deadman-webhook | deadman-webhook | passed |
| unclassified_default | default | default | passed |
inhibit
- InstanceDown 抑制同一 instance 上的其他告警。
- critical 抑制同一 alertname 和 service 的 warning。
- OrdersApiErrorBudgetBurnFast 抑制 service=orders-api 的 ServiceHighErrorRate。
每条 inhibit 都设置了 equal。工作时间是周一到周五 09:00–18:00 Asia/Shanghai。维护窗口是周日 02:00–04:00 Asia/Shanghai。
模板渲染
下面是 amtool template render 用一份样例 JSON 渲染的文本。样例不是发出去的通知,也不是一次事故。
Slack
[FIRING:1] OrdersApiErrorBudgetBurnFast *OrdersApiErrorBudgetBurnFast* severity=critical Orders API availability error budget is burning fast Sample payload for template rendering. Not a sent notification and not an incident. Runbook: https://runbooks.larkspur.example/OrdersApiErrorBudgetBurnFast Alerts: 1
Email 主题
[FIRING:1] OrdersApiErrorBudgetBurnFast runbook https://runbooks.larkspur.example/OrdersApiErrorBudgetBurnFast