Custom Grafana alert templates
The default Grafana alerts sent to slack look ugly and are difficult to grok quickly. The value is passed as a string containing the value and labels for example
Value [ var='C' labels={__name__=ecs_containerinsights_running_task_count_average, cluster_name=cluster,
exported_job=ecs_containerinsights, instance=prometheus-cloudwatch-exporter:9106, job=cloudwatch,
service_name=service} value=1 ], [ var='B' labels={__name__=ecs_containerinsights_running_task_count_average,
cluster_name=cluster, exported_job=ecs_containerinsights, instance=prometheus-cloudwatch-exporter:9106,
job=cloudwatch, service_name=service2} value=1 ]
When something is going wrong you don't want to try and figure out what that means.
The Grafana docs don't make this obvious but templating is done in two parts,
Grafana templates are not the easiest things to customise. The process is documented here.
The first is in contact points > message tempaltes
This determines the body of the message sent to the alert channel e.g slack
This is a sample template
{{ define "singlealert1" }} # this is a child template used by the main template for rendering alerts
{{ if gt (len .Annotations.alertmessage) 0 }}{{ .Annotations.alertmessage }}{{ end }} # annotations are set in the alerts this simply outputs the alertmessage annotation
{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }} {{ end }}
{{ end }}
{{ define "customalert1" }} # this is the main template and should be the name of the template
{{ if gt (len .Alerts.Firing) 0 }}
**Firing** {{ range .Alerts.Firing }} {{ template "singlealert1" .}} {{ end }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
**Resolved** {{ range .Alerts.Resolved }} {{ template "singlealert1" .}} {{ end }}
{{ end }}
{{ end }}
The above is quite simple and that’s intentional since the templates don’t receive the value they receive a string of the value + labels which is difficult to convert to a value.
The second part is in the alert itself, in the “Summary and annotations” section, add a new annotation named “alertmessage” and fill it with the variables you want e.g.
Mount {{ $labels.intance }} {{ $labels.mountpoint }} free disk {{ $values.B.Value | printf "%.2f" }}%
This will be output as a string in the alert.
The alert will now look like this Mount ServerName / free disk 10%
It is also important to use the newer math expressions in the alerts to instead of the classic conditions otherwise the labels won't populate to the annotations.