@ wrote... (1 month, 1 week ago)

My cluster was throwing warning Legacy BlueStore stats reporting detected and we could just not abide that.

Here's a simple way to upgrade:

cd /var/lib/ceph/osd
ls                                   # note your osd numbers

ceph osd set noout

for n in 7 8 9 10 11 12 ; do 
  systemctl stop ceph-osd@$n.service; 
  ceph-bluestore-tool repair --path ceph-$n; 
  systemctl start ceph-osd@$n.service; 
  systemctl status ceph-osd@$n.service;
done

# I like to wait until the cluster goes back to green 
# before doing the same on the next host

ceph osd unset noout
Category: tech, Tags: ceph
Comments: 0
@ wrote... (1 month, 2 weeks ago)

Sometimes an lxc container will refuse to start, usually after something goes wrong on the host. The key is to manually start the container and wait for some timeout or something, then the container will start properly.

Update: this does more or less the same but with less work.

pct list   # note the vmid, replace 100 101 below as appropriate

for m in 100 101; do lxc-start -n $m & done

# after a minute or so you'll see the background tasks finish in the console
pct list   # lxc containers should be running

# Note, that `systemd` will still show the tasks as failed since it didn't
# start them, so now lets stop/start them properly
for m in 100 101; do pct shutdown $m; pct start $m & done

The following is still valid but I now use the above method.

[root@proxmox1 ~]
# pct start 125
pJob for pve-container@125.service failed because a timeout was exceeded.
See "systemctl status pve-container@125.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@125' failed: exit code 1

[root@proxmox1 ~]
# /usr/bin/lxc-start -n 125 -F

# good long wait, over a minute
# login and then `poweroff`

[root@proxmox1 ~]
# pct start 125 && echo $?
0

tl;dr

  1. lxc-start -n 125 -F
  2. login and poweroff
  3. pct start 125
Category: tech, Tags: lxc, proxmox
Comments: 0
@ wrote... (3 months, 4 weeks ago)

Before upgrading to Proxmox 6 you need to upgrade to Corosync 3. Here's an ansible playbook that will automate that…

more…

Category: tech, Tags: ansible, proxmox
Comments: 0
@ wrote... (7 months, 1 week ago)

We recently upgraded our network to 10 Gbit and were really hoping to see monumental speed increases in our ceph cluster.

One of our benchmarks was pgbench and to say we were sad would be an understatement…

more…

Category: tech, Tags: ceph, postgresql
Comments: 0
@ wrote... (8 months, 2 weeks ago)

There are lots of posts about setting up CD with Jenkins and Kubernetes but I haven't found any describing how to do it with Nomad and Gitlab.

So here's how I did it…

more…

Category: tech, Tags: cd, ci, gitlab, hashistack, nomad
Comments: 5
@ wrote... (10 months, 3 weeks ago)

I also found the docs for consul connect to be confusing. They don't clearly differentiate the difference between the client and server proxy.

Some declarations that are worth stating explicitly:

  • consul acl needs to be setup first, see consul acl for more info
  • acl and intention are used somewhat interchangeably here
  • client side consul connect proxies can only talk to other consul connect proxies
  • client side consul connect proxies can not talk directly to a service
  • the docs explaining -service vs -listen vs -upstream are terrible
  • I'll use the term proxy to mean consul connect process
  • the term service refers to the actual service (eg. redis)
  • the term server proxy refers to the proxy that connects to a real service
  • the term client proxy refers to the proxy that clients connect to

Having said all that, service mesh sounds like they're worth having.

Mitchell Hashimoto at least partly agrees with me.

more…

Category: tech, Tags: consul
Comments: 0
@ wrote... (10 months, 3 weeks ago)

I found the otherwise great consul docs to be very obtuse and confusing and maybe even wrong.

I'm running these commands against my home setup which only has a single consul server. In a more realistic setup you'll need to duplicate the config changes on all your consul servers and then restart them one at a time.

Ran against consul 1.4.0

more…

Category: tech, Tags: consul
Comments: 2
@ wrote... (11 months ago)

I opened an issue on GitHub several months ago against the awesome fabio asking for a simple feature. To help foster community and contributors (I assume) the maintainer showed me the file to edit.

The problem, and why I ignored it for several months, was that I don't know Go. The patch itself is crazy simple, just editing an html template. The problem is the Go environment. How do you build, how do you test, etc…

Anyhow… today was the day I'd tackle this… what could go wrong?

more…

Category: tech, Tags: hatecomputers
Comments: 0
@ wrote... (11 months ago)

On Tuesday, December 11, 2018 I received a phising email, redacted version on pastebin.

Thankfully I used a unique password because there it was, in the clear. The unique password showed me that it was for http://osnews.com.

I reached out to them immediately and got a response promptly but it didn't mention anything about disclosure. So I then asked David if he was planning to tell his users about the breach and he replied he would by the end of the week.

Here's a snippet of his response:

The very old custom CMS that OSNews runs on hasn’t been meticulously
updated, and it does appear that someone got ahold of our user data.

On Monday, with still no announcement I sent another email and asked again. David replied he would announce by end of day. Although several content posts have been added in the last week there has still been no announcement of the security breach.

So, after one week, I'm announcing for them.

  • osnews.com has been hacked
  • osnews.com kept user passwords in the clear
  • those email/password tuples are now in the wild

Damn.

Category: tech, Tags: osnews
Comments: 3
@ wrote... (1 year ago)

Sometimes you don't have a favicon.ico or it's not in your staticfiles because you have a single page javascript or you're just tired of seeing 404s in your debug output.

Here's how you can serve up a hard coded icon (or any file really) directly from django.

I got this transparent icon from transparent-favicon.info

wget http://transparent-favicon.info/favicon.ico
base64 favicon.ico

I put this in my main project urls.py but feel free to put wherever.

# project/urls.py

def favicon(request):
    from textwrap import dedent
    from django.http import HttpResponse
    import base64

    icon = """\
    AAABAAEAEBACAAEAAQCwAAAAFgAAACgAAAAQAAAAIAAAAAEAAQAAAAAAgAAAAAAAAAAAAAAAAAAA
    AAAAAAAAAAAA////AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    AAAAAAAAAAAAAAAAAAAAAAAAAAD//wAA//8AAP//AAD//wAA//8AAP//AAD//wAA//8AAP//AAD/
    /wAA//8AAP//AAD//wAA//8AAP//AAD//wAA"""
    icon = dedent(icon)
    icon = base64.b64decode(icon)

    return HttpResponse(icon, content_type="image/x-icon")

urlpatterns += [
    url(r'^favicon.ico', favicon, name='favicon'),
]
Category: tech, Tags: django
Comments: 0