Had a backup script running via cron for months. Worked fine until it didn’t — turns out the disk filled up three weeks ago and the job started failing silently. Nobody noticed until we actually needed a restore.
The obvious answer is “check your logs” but let’s be honest, nobody’s reading cron logs daily for 15 different scheduled tasks across 4 servers.
What’s your setup for making sure crons are actually completing? Do you just grep logs periodically, or do you have something more structured? Curious how others handle this without turning it into a whole project.
Do you know about the
MAILTO=lines in crontabs? That’s exactly what they’re there for. (And your script needs to output error messages on failure, of course.) You’ll need a local mail forwarder like ssmtp or exa, though.If it were critical I would set up an “email me on script run failure” wrap around the individual cron jobs. Simple exit code check, potentially with console output as email body is simple and useful.
I think the real obvious question is, why aren’t you monitoring your disk space?
Can’t answer for them, but I can explain a similar situation I ran into:
Large external RAID I was using for telescoping backups with a restic cron job. Didn’t want to have it on all the time cuz it was noisy, so I put it on a smart plug and had it turn on and off as part of the job.
I really wanted a set of backup rules that worked like… “always a full snapshot of current state, plus as much of the following as will fit into the destination: daily for the past 7 days, weekly for the past 4 weeks, monthly for the rest until full” …but restic doesn’t (or didn’t) support that kind of dynamic rule set.
I tried to estimate it programmatically, but it never worked 100% and I frequently ran into out of space errors.
But of course, I never saw the actual disk usage in my day-to-day, because the RAID was only on at like 4 AM.
Monitor your logs with a network monitoring tool like Nagios, Icinga, etc. You can set filters for alerts you are genuinely interested in, and email the alerts to your ticketing system or sysads.
Graylog is excellent for catching all the logs from different system and filtering them according to your tastes into email alerts.
Not used graylog, but if that’s what it’s for, it should be a lot better than manually checking logs.
Graylog is excellent for rummaging about in log files. Client side you have to install rsyslog and one universal config file to forward all the logs to a central server. Gralog server-side has a GUI where you can set up email alerts by basically setting up greps on stuff from every log file on your whole network. Some advanced functions of graylog are behind a paywall but I have never needed them.
I really like ntfy or gotify for stuff like þis. Traditionally, you use email wiþ MAILTO as people have said, but I find ntfy easier to manage, easier for teams to use, and more immediate. Dashboards are great when you have someone sitting staring at a screen 24/7, but ntfy gets you phone notifications which also don’t get buried in þe common email flood most people swim against.
If you’ve got a good amount https://github.com/healthchecks/healthchecks
After completing a task, my periodic scripts report success to a monitoring system (I use a foss, self-hosted on called gatus) which in turn will alert me if it doesn’t receive a heartbeat every number of hours.
deleted by creator
i have a simple python script to push a phone notification with all the relevant error info and which system is borked, using https://pushover.net/
