From NixNet
1,991 bytes added ,  05:20, 18 November 2021
add cronjob alerts
== Failure emails notifications ==TODOWe use XMPP for most of our internal communication and have a dedicated MUC for alerts. There are tools such as [ sendxmpp] that allows for piping messages to arbitrary MUCs and contacts. However, that <code>sendxmpp</code> implementation isn't available in all repositories. [ go-sendxmpp] is an alternative written in Go that provides pre-built binaries for a large number of platforms. Download the binary, make it executable, then move it to <code>/usr/local/bin</code>. Put the following script in <code>/usr/local/scripts/alert</code> then make it executable. It takes input from <code>stdin</code>, stores it in a variable, then uses that variable in a message template that contains the cronjob's name and the machine's hostname. All of that is stored in another variable then piped to <code>go-sendxmpp</code>. The <code>go-sendxmpp</code> invocation requires TLS, sets its name to <code>Alert</code>, then relays whatever is passed on <code>stdin</code> to the <code></code> MUC. <pre>#!/bin/bash STDIN=$(</dev/stdin) MSG="$1 cronjob on $(cat /etc/hostname) exited with an error.\n\`\`\`\n$STDIN\n\`\`\`" echo -e "$MSG" | /usr/local/bin/go-sendxmpp -tr 'Alert' -c</pre> Before attempting to run the script, create <code>/root/.sendxmpprc</code> with the following contents, replacing USERNAME and PASSWORD with your bot's credentials. <pre>username: USERNAMEpassword: PASSWORD</pre> With all of this in place, run the following and see if the setup works. <pre>echo 'This is a test' | /usr/local/scripts/alert Test</pre> The output should look something like this. <pre>18/11/21 | 00:15:41 Alert: Test cronjob on hostname just exited.```This is a test``` </pre> If everything works as expected, you can now use it in cronjobs like so. <pre>@daily a command to do something 2>&1 | /usr/local/scripts/alert 'Cronjob title'</pre>
= Backups =
Yes, backups for the backups are necessary. Specifically, backups of backup ''keys'' are necessary; if you lose the key, you lose your backups. There is no way around this. While saving your key in a password manager, in a text file on your PC, on someone else's PC (please don't ever do this), or even on another server are all potential solutions, it's good to have an offline copy Just In Case™. We use [ paperbackup] because it's easily read by both machines (QR codes) and humans (plain text); run your key through this, print the resulting PDF, put it in a folder, put the folder in a box, and put the box somewhere very very safe.