Prometheus is really good at pulling metrics but it needs help if you want to test if a given host is up with a simple ping.
In this post I'll show you my config that gets a list of hosts from consul (plus some static hosts) and then ping them to monitor if they're up or not.
If a host goes down then fire an alert.
I thought I learned my lesson years ago but good lord did I over pack for this trip. I think because I was going on a “working vacation” I tricked myself into bringing too many what-ifs and nice-to-haves.
So here's what I brought with a secondary column for should-have-brought.
I'm just standing up a full “Hashi Stack” with Consul, Nomad, and Vault so likely lots of future posts.
One part that should have been simple but took lots of trial and error was getting authentication to work to our FreeIPA installation.
Here are the steps required (assumes vault and freeipa already installed and working)…
I thought I understood async programming as I'd written a few non-trivial apps in the past (using Boost ASIO)
Who has two thumbs and suffers from async programming Dunning-Kruger effect? This guy!
So here's the first of hopefully many recipes to make async programming under Python 3.6 a little easier.
The curl manpage isn't super clear on this, but to read from stdin you need to prepend the dash with a @.
json_producer | curl -d @- -H 'Content-Type: application/json' http://example.com/json_consumer
You should always use the logging module instead of just littering print statements all over your code. You will very quickly thank yourself for taking a bit of extra time.
Having said that, setting up logging is a bit of a pain so here's the pattern I use 90% of the time.
Uploading to Minio (or S3) in a script is a bit tricky.
Update: now on GitHub with some Python versions.
#!/bin/bash
# usage: ./minio-upload my-bucket my-file.zip
bucket=$1
file=$2
host=minio.example.com
s3_key='secret key'
s3_secret='secret token'
resource="/${bucket}/${file}"
content_type="application/octet-stream"
date=`date -R`
_signature="PUT\n\n${content_type}\n${date}\n${resource}"
signature=`echo -en ${_signature} | openssl sha1 -hmac ${s3_secret} -binary | base64`
curl -v -X PUT -T "${file}" \
-H "Host: $host" \
-H "Date: ${date}" \
-H "Content-Type: ${content_type}" \
-H "Authorization: AWS ${s3_key}:${signature}" \
https://$host${resource}
I highly doubt I've invented something new, but I did independently invent it, so I have that going for me, which is nice. Plus I'm stitching together lots of great open source pieces which we all have going for us.
Anyhow, this is kind of difficult to explain. And complicated… with lots of moving parts… but is super useful once you get it to work.
As high level as possible, when I do a git push, another machine
reloads new configuration files. And notifies me. All within a second.
Alkali v0.5.5 has now escaped.
Code at github.com and docs at readthedocs.org.
Major features are: