ObservaStack

路由

amtool config routes test 不计算 active_time_intervals。warning 且 team=platform 在工具里会落到 slack-platform,即使不在 Asia/Shanghai 的工作时间。这是工具的行为,页面没有改时间。

amtool config routes show

Routing tree:
.
└── default-route  receiver: default
    ├── {alertname="Watchdog"}  receiver: deadman-webhook
    ├── {severity="critical"}  continue: true  receiver: pagerduty-oncall
    ├── {severity="critical"}  continue: true  receiver: slack-incidents
    ├── {team="payments"}  receiver: slack-payments
    ├── {team="orders"}  receiver: email-oncall
    ├── {severity="warning"}  receiver: slack-platform
    └── {severity="info"}  receiver: null

断言

用例期望 receivers实际结果
critical_platform_instance_downpagerduty-oncall,slack-incidentspagerduty-oncall,slack-incidentspassed
critical_orders_fast_burnpagerduty-oncall,slack-incidents,email-oncallpagerduty-oncall,slack-incidents,email-oncallpassed
critical_paymentspagerduty-oncall,slack-incidents,slack-paymentspagerduty-oncall,slack-incidents,slack-paymentspassed
warning_orders_slow_burnemail-oncallemail-oncallpassed
warning_payments_queueslack-paymentsslack-paymentspassed
warning_platform_memoryslack-platformslack-platformpassed
info_discardednullnullpassed
watchdog_deadmandeadman-webhookdeadman-webhookpassed
unclassified_defaultdefaultdefaultpassed

inhibit

每条 inhibit 都设置了 equal。工作时间是周一到周五 09:00–18:00 Asia/Shanghai。维护窗口是周日 02:00–04:00 Asia/Shanghai。

模板渲染

下面是 amtool template render 用一份样例 JSON 渲染的文本。样例不是发出去的通知,也不是一次事故。

Slack

[FIRING:1] OrdersApiErrorBudgetBurnFast
*OrdersApiErrorBudgetBurnFast* severity=critical
Orders API availability error budget is burning fast
Sample payload for template rendering. Not a sent notification and not an incident.
Runbook: https://runbooks.larkspur.example/OrdersApiErrorBudgetBurnFast
Alerts: 1

Email 主题

[FIRING:1] OrdersApiErrorBudgetBurnFast runbook https://runbooks.larkspur.example/OrdersApiErrorBudgetBurnFast