Creating alerts based on metrics
Let us create the first alert. We'll update our go-demo_main
service by adding a few labels:
docker service update \ --label-add com.df.alertName=mem \ --label-add com.df.alertIf='container_memory_usage_\ bytes{container_label_com_docker_swarm_service_name=\ "go-demo_main"} > 20000000' \ go-demo_main
Note
Normally, we should have labels defined inside our stack file. However, since we'll do quite a few iterations with different values, we'll be updating the service instead modifying the stack file. That way we'll be able to iterate faster.
The label com.df.alertName
is the name of the alert. It will be prefixed with the name of the service stripped from underscores and dashes (godemomem
). That way, a unique alert name is guaranteed.
The second label (com.df.alertIf
) is more important. It defines the expression. Translated to plain words, it takes the memory usage limited to the go-demo_main
service and checks whether it is bigger than 20 MB...