GitLab CI for
Omnis Studio

testing, ci, docker and you

EurOmnis 2018

by Kurt Neufeld @ Artsman

About Me

  • Programmed a bunch in C++
  • Then programmed a bunch in Python
  • Been running Linux at home and work for 20+ years
  • Been at Artsman for just over a year in Ops role
  • Expert copy-paster from StackOverflow, I guess you could say I'm full stack
  • Ran Omnis Studio for the first time 2 days ago

Salient Points

  1. Where the industry came from
  2. Where the industry is going
  3. Where Artsman came from
  4. Where Artsman is going
  5. Examples & Demo?

Industry History

1940's?

Industry History

1950's?

Industry History

1960's

Industry History

1970's

Industry History

1980's

Industry History

  • OK, I'm getting bored of this now
  • 1990 - file servers, client/server architecture
  • 2000 - the web explodes
  • 2010 - virtualization and the cloud

Industry Prognostication

Or guessing if English isn't your first language

  • containers
  • containers
  • probably some more containers
  • more cloud

Artsman History

  • started 30 years ago by a Scot on a dare
  • we sell ticketing software
  • Omnis for native OSX & Windows
  • virtual OSX on our collocated datacenter

Artsman Prognostication

I have some input on this one

  • Omnis now supports Linux
  • cloud Linux
  • containers!

So...

What does all this Linux and containers have to do with Omnis and us here today?

Glad you asked

Definitions

CI
Continuous Integration
Continuous Integration
automated build and testing of all commits
Container
like a VM but shares the host kernel

Modern Stack

Applications Omnis, etc
Container Docker, LXC
OS Linux, no other viable options
Machine Physical, Amazon, Google, VMWare

Artsman Stack

Apps LXC moving towards Docker
Orchestration Nomad; Kubernetes (not Rancher)
LXC or VM apps in LXC, Docker in VMs
OS Proxmox; VMWare, Amazon
Machines 2 datacenters, east and west

Artsman Stack (future)

Docker running apps
Orchestration probably Kubernetes
OS Proxmox (or similar)
Hybrid Cloud Private & Public

Artsman Stack

  • Hashi Stack
    • Consul
    • Nomad
    • Vault (future)
  • Monitoring
    • Nagios
    • Prometheus (sorta)
    • Grafana (sorta)
  • ELK Stack
    • Elastic Search
    • Log Stash
    • Kibana
  • OS
    • Proxmox
  • Configuration
    • Ansible

Artsman Stack

Other things we can't live without

  • Pfsense
  • PostgreSQL
  • Python
  • Redis
  • Ceph
  • FreeIPA
  • nginx
  • Rocket.Chat
  • Gitlab
  • Fabio
  • Minio
  • Open Source
  • Omnis Studio!

Artsman Stack (future)

More specifically...

  • Get Theatre Manager Server running on Linux
  • Get off of virtual OSX
  • Radically reduces hardware requirements
  • Check up on things once a week from a beach

How to get there...

That's a lot of changes, sounds dangerous

It can be, but with lots of tests (which we have) then it's manageable.

So specifically, how to do that?

Workflow

  1. write some sweet code
  2. git commit
  3. git push this is where magic happens
  4. a new artifact is built
  5. tests automatically get run
  6. you could auto deploy here

Write Some Sweet Code

You're on your own with this one.

git commit

You are using git right?!?

git push

I recommend GitLab (self hosted or paid service) or GitHub (paid service).

Through the magic of webhooks, scripts are run when pushes are made. Some of those scripts can:

  • run tests
  • build artifacts
  • deploy code
  • notify via chat, email, etc

Moving Parts

So Many Moving Parts

  • gitlab to initiate everything
  • docker to run gitlab-runners
  • gitlab-runner accepts jobs from gitlab
  • jobs get containers from docker-registry
  • jobs do work; run tests, build artifacts
  • artifacts get saved in gitlab
  • containers get pushed to docker-registry
  • nomad re-deploys new containers
  • etc etc etc

I did all that before lunch

.gitlab-ci.yml (simplified)


stages:
  - build
  - test

build_deploy_image:
  stage: build
  image: ${docker_registry}/public/alpine:3
  script:
    - docker build --pull -t ${tag_branch} -f Dockerfile .
    - docker push ${docker_registry}/${docker_image}:${CI_COMMIT_REF_SLUG}

pytest:
  stage: test
  image: ${docker_registry}/${docker_image}:${CI_COMMIT_REF_SLUG}
  services:
    - postgres:10
  script:
    - ./make_database
    - pytest -s

Automated Tests

  • computers don't forget to run them
  • stop deploy chain if they fail
  • notify developer immediately on problem
  • can't start building on broken foundation

Demo Preconditions

  • gitlab, nomad, consul, fabio
  • base docker image with homnis in it

Demo

deploy new version of omnis app

  • edit and commit omnis app
  • ci will build new docker image
  • manually deploy new docker image

Demo Deficiencies

  • should never check-in generated code (ie: .lbs)
  • should compile dir tree into .lbs
  • no tests written (homnis has issues)

Production Differences

  • dns entry for service
  • config in load balancer
  • random port

The End

questions?