@ wrote... (4 months, 1 week ago)

I thought I understood async programming as I'd written a few non-trivial apps in the past (using Boost ASIO)

Who has two thumbs and suffers from async programming Dunning-Kruger effect? This guy!

So here's the first of hopefully many recipes to make async programming under Python 3.6 a little easier.

more…

Category: tech, Tags: async, python
Comments: 0
@ wrote... (5 months, 2 weeks ago)

You should always use the logging module instead of just littering print statements all over your code. You will very quickly thank yourself for taking a bit of extra time.

Having said that, setting up logging is a bit of a pain so here's the pattern I use 90% of the time.

more…

Category: tech, Tags: python
Comments: 0
@ wrote... (6 months, 1 week ago)

Uploading to Minio (or S3) in a script is a bit tricky.

Update: now on GitHub with some Python versions.

#!/bin/bash

# usage: ./minio-upload my-bucket my-file.zip

bucket=$1
file=$2

host=minio.example.com
s3_key='secret key'
s3_secret='secret token'

resource="/${bucket}/${file}"
content_type="application/octet-stream"
date=`date -R`
_signature="PUT\n\n${content_type}\n${date}\n${resource}"
signature=`echo -en ${_signature} | openssl sha1 -hmac ${s3_secret} -binary | base64`

curl -v -X PUT -T "${file}" \
          -H "Host: $host" \
          -H "Date: ${date}" \
          -H "Content-Type: ${content_type}" \
          -H "Authorization: AWS ${s3_key}:${signature}" \
          https://$host${resource}
Category: tech, Tags: linux, python, shell
Comments: 0
@ wrote... (1 year, 3 months ago)

Alkali v0.5.5 has now escaped.

Code at github.com and docs at readthedocs.org.

Major features are:

  • adding BoolField
  • adding CSVStorage, load/save csv files
  • queries are bit more efficient/fast, don't copy as many objects by default
  • some doc cleanups
Category: tech, Tags: alkali, python
Comments: 0
@ wrote... (1 year, 4 months ago)

You can now nicely render markdown in your console/terminal with ConsoleMD in glorious color.

Note the subtlety changing hues of the subtopics, the brilliance of auto-incrementing list counters, the dulcet notes of the embedded python, the elegant italics of the block quote. ConsoleMD has it all! At least with regards to my small test files.

Code is on github.com

Category: tech, Tags: consolemd, python
Comments: 0
@ wrote... (1 year, 5 months ago)

Alkali v0.5.4 has now escaped.

Code at github.com and docs at readthedocs.org.

Major features are:

  • implemented Query.distinct
  • implemented Query.annotate
  • implemented Query.aggregate functions: Sum, Count, Min, Max
  • Fields are now descriptors on Model instance
  • added ForeignKey field
  • models now cascade delete when ForeignKey instance is deleted
  • minor speed ups
  • IntField now has auto_increment property
  • Query returns copies of Model instances
  • added signals on model creation/deleting/saving/etc
  • better documentation
Category: tech, Tags: alkali, python
Comments: 0
@ wrote... (1 year, 5 months ago)

I had a hell of a time trying to make a nice help command while using click.

I wanted the ability to have a help sub-command for each real command.

mycommand --help
mycommand help

mycommand foo --help
mycommand help foo

It's actually fairly easy (once you know how).

CTX_SETTINGS=dict(help_option_names=['-h','--help'])

@click.group(context_settings=CTX_SETTINGS)
@click.pass_context
def cli(ctx, **kw):
    ...

@cli.command()
@click.pass_context
def foo(ctx):
    ...

@cli.command()
@click.argument('topic', default=None, required=False, nargs=1 )
@click.pass_context
def help(ctx, topic, **kw):
    if topic is None:
        print ctx.parent.get_help()
    else:
        print cli.commands[topic].get_help(ctx)
Category: tech, Tags: python
Comments: 0
@ wrote... (1 year, 6 months ago)

Packages change pretty frequently when you're using Homebrew on OSX, including Python.

I noticed that I had several instances of lots of large packages installed (Boot, MacVim, Python, etc) so I ran brew cleanup and that freed up approx 9 gigs of disk! Unfortunately it also broke many virtual environments since the version of python that they were pointing at no longer existed.

What to do what to do…

Based on this Stack Overflow question I wrote a script to update a single virtual environment or all of them.

Note, this only updates virtual enviroments created with virtualenvwrapper. Also, you may have to reinstall your packages, but hopefully not.

#!/bin/bash

if [ -z "$1" ]; then
    echo "usage: $0 virtualenv|all"
    exit 1
fi

if [ -z "$WORKON_HOME" ]; then
    echo "you must export WORKON_HOME"
    exit 1
fi

if [ "$1" = "all" ]; then
    echo "udpating all virtualenvs"
    cd $WORKON_HOME

    for name in $(find . -type d -maxdepth 1 -mindepth 1|xargs -n1 basename); do
        $0 $name
    done

    exit 0
fi

cd $WORKON_HOME
virtenv="$WORKON_HOME/$1"

echo "deleteing $1"

rm -f  $virtenv/.Python
rm -f  $virtenv/bin/pip
rm -f  $virtenv/bin/pip2
rm -f  $virtenv/bin/pip2.7
rm -f  $virtenv/bin/python
rm -f  $virtenv/bin/python2
rm -f  $virtenv/bin/python2.7
rm -fr $virtenv/include
rm -f  $virtenv/lib/python2.7/* 2> /dev/null
rm -fr $virtenv/lib/python2.7/distutils
rm -f  $virtenv/lib/python2.7/site-packages/easy_install.*
rm -fr $virtenv/lib/python2.7/site-packages/pip
rm -fr $virtenv/lib/python2.7/site-packages/pip-*.dist-info
rm -fr $virtenv/lib/python2.7/site-packages/setuptools
rm -fr $virtenv/lib/python2.7/site-packages/setuptools-*.dist-info

source `which virtualenvwrapper.sh`

echo "creating $1"
mkvirtualenv -q $1
Category: tech, Tags: osx, python, virtualenv
Comments: 0
@ wrote... (1 year, 6 months ago)

There are dozens of similar programs on the the ol' internetz and here's mine.

I'm not a cryptographer.

more…

Category: tech, Tags: python
Comments: 0
@ wrote... (1 year, 11 months ago)

Well, I finally released alkali into the wild.

Alkali is a simple database that makes it very easy to specify the on-disk format of your data. This makes it easy to use your existing data files as tables in a database. Plus the api is based off of Django models.

This is my first real project that I've released and it's a surprising amount of work. I have a new level of appreciation for all the libraries that I just blithely download and use without a second's thought.

For instance, it took about the same amount of time to write the docs as it did to write the actual code. Plus there are a lot of moving pieces to release open source software the right way.

  • use git as your source code control
  • write documentation, learn Sphinx and reStructuredText
  • when you push to github, triggers are fired
  • push release to PyPi, writing setup.py is very non-trivial, learn how that works
  • try to do some marketing on Reddit. Given my zero karma I suck at marketing and/or programming.

So yeah… please go check out alkali!

Category: tech, Tags: alkali, python
Comments: 0