We currently host some of our customers on our private cloud and as a result we need to manage an ssl certificate for them as well. Traditionally we've used a standard SSL provider but this was not only pricey, more importantly it was a lot of work for the support staff to get certs, update certs, deploy certs, etc.
Now that's is so easy for me to create new containers and VMs with Proxmox, that's
exactly what I've been doing. However, I like to be a good netizen and use https.
Here's my nginx config that lets me easily add a new LetsEncrypt certificate to a new vm/container for a new webapp.
While trying to proxy my main nginx instance to a GitLab docker container I wasted hours and hours and hours trying to fix the following error:
fatal: unable to access 'https://gitlab.burgundywall.com/kneufeld/myproject.git/': \
SSL read: error:00000000:lib(0):func(0):reason(0), errno 54
It turns out that nginx config option ssl_session_cache is super f'n important to not screw up. I'm not totally sure what the problem is, but in my main server clause i had
ssl_session_cache shared:SSL:1m;
and I didn't have any such option in my server gitlab stanza. So something something something I could not do any git commands via https.
And even with logging everything looked okay
GIT_CURL_VERBOSE=1 git clone https://gitlab.burgundywall.com/kneufeld/myproject.git
Cloning into 'myproject'...
* Couldn't find host gitlab.burgundywall.com in the .netrc file; using defaults
* Trying 192.168.5.6...
* Connected to gitlab.burgundywall.com (192.168.5.6) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /usr/local/etc/openssl/cert.pem
CApath: none
* NPN, negotiated HTTP1.1
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=gitlab.burgundywall.com
* start date: Sep 3 16:53:00 2016 GMT
* expire date: Dec 2 16:53:00 2016 GMT
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
> GET /kneufeld/myproject.git/info/refs?service=git-upload-pack HTTP/1.1
Host: gitlab.burgundywall.com
User-Agent: git/2.9.3
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache
* SSL read: error:00000000:lib(0):func(0):reason(0), errno 54
* Closing connection 0
fatal: unable to access 'https://gitlab.burgundywall.com/kneufeld/myproject.git/': SSL read: error:00000000:lib(0):func(0):reason(0), errno 54
except it didn't work.
Anyhow, when I finally figured out that ssl_session_cache was the issue and did some reading I just made sure that each ssl server has it's own cache.
ssl_session_cache shared:SSL-gitlab:1m;
Hours.
Comments: 1
This is a quick and dirty config guide to setup git-http-backend with uwsgi from supervisord on nginx. I did this on Fedora 20 so on a different distro some paths will likely be different.
I've also ignored security in this post as I have ssl termination handled elsewhere. This will send your password in the clear if you don't have ssl setup correctly! Don't forget to setup git permissions as well. This config allows any authorized user (somebody who can login) to push.
OMG I HATE MY LIFE SOMETIMES
So… I'll hopefully write more about this later, but the TL;DR is do not use monitor.py and uwsgi.
So if you have something like this in your wsgi.py, comment it out with extreme prejudice.
import monitor
monitor.start(interval=60)
Symptoms include, but not limited to:
- workers not shutting down
- workers appear to hang and cause 50x errors
- workers not restarting if you send appropriate signals
- workers appear to work once or twice then hang
- workers never work
- randomness and non-determinism all over the f'in place
- little to no helpful log output
- propensity to start adding lots of config variables that you don't understand
- propensity to start drinking
- wondering htf everybody else seems to use this software without a problem
- wondering if perhaps computers where a poor career choice after all
I haven't looked into too closely but I suspect the fact that the monitor opens up it's own thread and runs forever has something to do with it. Open files, open sockets, stuff like that.