I highly doubt I've invented something new, but I did independently invent it, so I have that going for me, which is nice. Plus I'm stitching together lots of great open source pieces which we all have going for us.

Anyhow, this is kind of difficult to explain. And complicated… with lots of moving parts… but is super useful once you get it to work.

As high level as possible, when I do a git push, another machine reloads new configuration files. And notifies me. All within a second.

So here is a more detailed breakdown of the techno-magical dance.

1. on my laptop I edit some Nagios config files
2. when I git push a webhook is fired
3. a process catches the webhook and PUBlishes to a git channel with the relevant information using Redis
4. another process SUBscribes to that git channel and runs git pull
5. the git pull invokes the post-merge git hook
6. the post-merge script verifies the config and reloads Nagios and notifies Rocket.Chat (but Slack is also easy)

git push webooks

After editing and committing some changes locally, I run git push that updates our GitLab server. We use GitLab but any git server that supports webhooks will work, like GitHub.

The repo on GitLab has a webhook integration configured to send out push notifications:

Webhooks:

http://rancher.example.com:9000/
Push Events

push-notifier - the gitlab webhook listener

We use Rancher because it's awesome. If you're never heard of Rancher it's (among other things) a pretty web app that controls Docker.

So in my Rancher cluster I have a container running a webhook listener. This webhook listener converts incoming webhooks to message bus PUB messages.

In our case we're using Redis as our PUB/SUB message bus but you could use RabbitMQ or any number of other equivalent technologies.

My little app is called push-notifier because it notifies things when a push happens. The other half of that channel is called push-subscriber.

Anyhow, after receiving a notification from GitLab push-notifier then PUBs some JSON to the git channel.

{
    "user_name": "bob",
    "repo_url": "/ops/nagios_conf.git"
}

push-subscriber - the push-notifier listener

So on the nagios machine we have something listening for the PUB messages and then running git pull

Here's a systemd service file for that:

# /etc/systemd/system/nagios-conf-subscriber.service

[Unit]
Description=Nagios Conf Subscriber
BindTo=network.target

[Install]
WantedBy=multi-user.target

[Service]
User=nagios
Group=nagios
Type=simple
ExecStart=/opt/push-subscriber/venv/bin/python /opt/push-subscriber/push-subscriber.py \
    --redis_host rancher.example.com --redis_port 9001 \
    --channel git \
    --working_dir /etc/nagios_conf

push-subscriber.py is a bit messy since it potentially needs to be able to handle many different repos on the same machine, but it all boils down to mostly this:

while True:
  sub = listen_for_publish_msg_from_redis()
  if sub.repo == 'nagios_conf':
    run_command('cd /etc/nagios; git pull')

the git hook

So now we've got some new code in our repo located in /etc/nagios. In case you weren't aware, your local git repo has a hooks directory that can contain scripts that are run by git when certain events happen. In our case the git pull event maps to the post-merge event.

# .git/hooks/post-merge

#!/bin/bash

set -e

msg="successfully updated nagios config via git hooks"

err_exit() {
    msg="incoming nagios config has errors, not reloading"
}

# tell the world about our new config via rocket.chat
on_exit() {
    URL='https://chat.example.com/hooks/secret-token/secret-url'

    data="{'channel': '#ops', 'text': '$msg'}"
    # convert ' to " and " to ' to make valid json
    data=`echo $data | tr "\"'" "'\""`
    curl --silent -H 'Content-type: application/json' -d "$data" $URL
}

# on an error run err_exit function
trap err_exit ERR

# when script exits run on_exit
trap on_exit EXIT

cd /etc/nagios

echo "checking config"
/usr/local/nagios/bin/nagios -v /etc/nagios/nagios.cfg > /dev/null

echo "reloading nagios"
/usr/bin/sudo /bin/systemctl reload nagios.service

ProTip

Your .git/hooks directory can itself be a git repo! This is how you can distribute hooks among your team.

git clone http://gitlab.company.com/nagios_conf.git /etc/nagios
rm -rf /etc/nagios/.git/hooks
git clone http://gitlab.company.com/nagios_conf_hooks.git /etc/nagios/.git/hooks

Conclusion

Since nobody actually reads my blog I've excluded lots of code, namely push-notifier and push-subscriber as well as my rancher.compose. Maybe I'll get around to open sourcing/making available these pieces and then update this post.