We currently host some of our customers on our private cloud and as a result we need to manage an ssl certificate for them as well. Traditionally we've used a standard SSL provider but this was not only pricey, more importantly it was a lot of work for the support staff to get certs, update certs, deploy certs, etc.
While trying to proxy my main nginx instance to a GitLab docker container I wasted hours and hours and hours trying to fix the following error:
fatal: unable to access 'https://gitlab.burgundywall.com/kneufeld/myproject.git/': \ SSL read: error:00000000:lib(0):func(0):reason(0), errno 54
It turns out that nginx config option
ssl_session_cache is super f'n important to not screw up. I'm not totally sure what the problem is, but in my main
server clause i had
and I didn't have any such option in my
server gitlab stanza. So something something something I could not do any git commands via
And even with logging everything looked okay
GIT_CURL_VERBOSE=1 git clone https://gitlab.burgundywall.com/kneufeld/myproject.git Cloning into 'myproject'... * Couldn't find host gitlab.burgundywall.com in the .netrc file; using defaults * Trying 192.168.5.6... * Connected to gitlab.burgundywall.com (192.168.5.6) port 443 (#0) * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /usr/local/etc/openssl/cert.pem CApath: none * NPN, negotiated HTTP1.1 * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=gitlab.burgundywall.com * start date: Sep 3 16:53:00 2016 GMT * expire date: Dec 2 16:53:00 2016 GMT * issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 * SSL certificate verify ok. > GET /kneufeld/myproject.git/info/refs?service=git-upload-pack HTTP/1.1 Host: gitlab.burgundywall.com User-Agent: git/2.9.3 Accept: */* Accept-Encoding: gzip Pragma: no-cache * SSL read: error:00000000:lib(0):func(0):reason(0), errno 54 * Closing connection 0 fatal: unable to access 'https://gitlab.burgundywall.com/kneufeld/myproject.git/': SSL read: error:00000000:lib(0):func(0):reason(0), errno 54
except it didn't work.
Anyhow, when I finally figured out that
ssl_session_cache was the issue and did some reading I just made sure that each ssl server has it's own cache.
This is a quick and dirty config guide to setup
nginx. I did this on
Fedora 20 so on a different distro some paths will likely be different.
I've also ignored security in this post as I have ssl termination handled elsewhere. This will send your password in the clear if you don't have ssl setup correctly! Don't forget to setup git permissions as well. This config allows any authorized user (somebody who can login) to push.
OMG I HATE MY LIFE SOMETIMES
So… I'll hopefully write more about this later, but the TL;DR is do not use
So if you have something like this in your
wsgi.py, comment it out with extreme prejudice.
import monitor monitor.start(interval=60)
Symptoms include, but not limited to:
- workers not shutting down
- workers appear to hang and cause 50x errors
- workers not restarting if you send appropriate signals
- workers appear to work once or twice then hang
- workers never work
- randomness and non-determinism all over the f'in place
- little to no helpful log output
- propensity to start adding lots of config variables that you don't understand
- propensity to start drinking
- wondering htf everybody else seems to use this software without a problem
- wondering if perhaps computers where a poor career choice after all
I haven't looked into too closely but I suspect the fact that the monitor opens up it's own thread and runs forever has something to do with it. Open files, open sockets, stuff like that.