TheCompWiz
Well-known member
- Joined
- Apr 20, 2016
- Messages
- 142
- Points
- 18
The monitoring got noisier than the servers. Disk warnings firing every night while a backup writes and clearing an hour later, load spikes during log rotation, certificate notices for domains that renew themselves anyway. After a few months of that I had stopped reading any of it, which is the same as not having it.
First thing I cut was anything that resolves without me. If an alert has never once required me to do something, it is not an alert, it is a graph, and it belongs on a dashboard I look at weekly rather than on my phone at 3am.
What I kept is short. The site does not answer. The certificate did not renew when it should have. The backup job did not finish. Disk crossed a level it has never crossed before. Four things, and every one of them means something is genuinely wrong.
The one I nearly cut and am glad I did not is the backup completion check. It stayed quiet for the right reason for about a year and then stayed quiet for the wrong reason, and I would not have found that any other way. Alerting on success failing to arrive, rather than on a failure arriving, is the only version of that check I trust now.
What is on your list these days, and did cutting anything turn out to be a mistake?
First thing I cut was anything that resolves without me. If an alert has never once required me to do something, it is not an alert, it is a graph, and it belongs on a dashboard I look at weekly rather than on my phone at 3am.
What I kept is short. The site does not answer. The certificate did not renew when it should have. The backup job did not finish. Disk crossed a level it has never crossed before. Four things, and every one of them means something is genuinely wrong.
The one I nearly cut and am glad I did not is the backup completion check. It stayed quiet for the right reason for about a year and then stayed quiet for the wrong reason, and I would not have found that any other way. Alerting on success failing to arrive, rather than on a failure arriving, is the only version of that check I trust now.
What is on your list these days, and did cutting anything turn out to be a mistake?
