The basic idea is that you write the large (or repeated) comment once in a sensible place, and reference it in the myriad other places where you may want the extra context.

Here’s an example of one:

# Note [CharField length 191]
# ===========================
#
# MySQL VARCHAR fields with an index or constraint and the utf8mb4 encoding
# are limited to 191 characters in length.
# This is because MySQL assumes that each char may be 4 bytes long,
# and the maximum length of an InnoDB index is 767 bytes.
#
# 767 / 4 = 191.75
#
# See https://www.grouparoo.com/blog/varchar-191

Now when you write code, you can “link” to the note without massive comments breaking up your reading of the code. This is especially useful if you might be tempted to repeat comments in lots of places.

For example, the code below “links” to the note we defined above next to each CharField. The note doesn’t even have to be in the same file, as you can search for the heading to find it anywhere.

class MyModel(models.Model):
    # Note [CharField length 191]
    value = models.CharField(max_length=191, ...)


class MyOtherModel(models.Model):
    # Note [CharField length 191]
    name = models.CharField(max_length=191, ...)

Thanks to jml for bringing this style to my attention. I hope it’s useful!

1. A default must not injure production or, through omission, allow production to come to harm.
2. A default must ease local development except where such a default would conflict with the First Law.
3. A default must reflect the live environment as long as such reflection does not conflict with the First or Second Laws.

This format is inspired by Isaac Asimov’s Three Laws of Robotics.

Like Asimov’s Laws, I expect there are some odd cases when they’re pushed to their limits.

This script runs on a regular cron-job on my server, and solves the issue nicely:

# fix_firewall.py
import subprocess
import sys


def run_command(command):
    command = command.split()
    return subprocess.run(command, check=True, stdout=subprocess.PIPE).stdout


def get_remote_ip_address(domain):
    stdout = run_command(f'host {domain}')
    return stdout.strip().rsplit(maxsplit=1)[-1].decode()


def get_firewalled_ip_and_rule_number(port):
    stdout = run_command('ufw status numbered')
    for line in stdout.decode().splitlines():
        if f'{port}/tcp' in line:
            number, *_, ip = line.rsplit(maxsplit=4)
            return number.strip('[] '), ip


def remove_old_rule(rule_number):
    run_command(f'ufw --force delete {rule_number}')


def add_new_rule(ip, port):
    run_command(f'ufw allow from {ip} proto tcp to any port {port}')


if __name__ == '__main__':
    _, domain, port = sys.argv
    remote_ip = get_remote_ip_address(domain)
    rule_number, firewalled_ip = get_firewalled_ip_and_rule_number(port)
    if remote_ip != firewalled_ip:
        print(f'Changing IP from {firewalled_ip} to {remote_ip}.')
        remove_old_rule(rule_number)
        add_new_rule(remote_ip, port)

This script has to be run as root so that it can modify the firewall rules.

It’s run like this:

python3 fix_firewall.py my-domain.example.com 12345

# Somewhere in settings.py...
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Then, when you create an app, or install a library, you’re faced with a choice: do you add the new item above or below Django’s apps?

Well, it’s good practice to add new apps above Django’s, unless you have a specific reason not to. I’ve seen this done the other way around a lot, so it might not be what you expect.

This is because when Django is looking for something, it will go through the INSTALLED_APPS list, from the top, one app at a time, until it has found what it’s looking for. Putting your code first means it takes priority.

For example, if you want to override a template in Django’s admin, you can do so by placing a modified version of the original into the templates/ directory of one of your apps.

This applies to:

• management commands,
• templates,
• template-tags,
• translations,
• probably a lot more.

It’s also handy to know that this is the order in which apps will be registered, and admin integration discovered.

Finally: don’t add the core app’s templates/ directory into TEMPLATES[...]['DIRS'] (or TEMPLATE_DIRS)! Instead, simply add each app containing templates/ to INSTALLED_APPS.

TLDR

In a Django projects, expect to see INSTALLED_APPS structured like this:

INSTALLED_APPS = [
    # The "core" (AKA "project" or "main") app.
    'core',

    # Other apps in the project.
    'foo',
    'bar',

    # Libraries.
    'orderable',
    'pgcrypto',
    'webpack_loader',

    # Django.
    'django.contrib.admin',
    'django.contrib.auth',
    ...  # Etc.
]

This ensures:

• The ‘core’ app can override everything.
• Your secondary apps can extend the libraries and Django.
• Libraries can be used to extend Django.
• As usual, Django provides a solid default.

I’ve installed Ubuntu on desktops and servers loads of times, so I’m pretty familiar with using it in a normal setting, but this presented a couple of unusual challenges.

Firstly, the box doesn’t have anywhere to plug in a monitor. Now, it’s not needed once the NAS is up and running, but it is important for the installation. It does have a suspiciously-VGA-port-shaped plate on the back, however…

Jaroslaw Zachwieja’s related blog post very helpfully explains that there are some Nvidia graphics card breakout cables that can be installed to get this working. He lists them as “XFX part: MA-BK01-LP1K”. It proved very hard to find that item in stock anywhere. I finally found a listing on eBay under the name “VGA SVGA Video Graphics Adapter Card Slot Bracket Header Cable 15 Pin 16 Hole”. Catchy. I snapped one up. I’ve just checked: now they’re sold out too.

If you’re trying to do this, and are unable to get hold of (or make) the required cable, then you might have luck using the serial port on the back instead.

With the VGA cable installed I was able to plug in a monitor, watch the default OS boot, and do a little victory jig.

After that, a had some false starts. I discovered that:

• To boot to the live USB, one must hold down the “Backup” button while turning on the machine.
• The NAS is picky about USB sticks. A Kingston DataTraveller (100 G3) 8GB did not work, but my Sandisk Ultra Fit 128GB did.
• On boot, Ubuntu showed an error and a prompt:

    graphics initialization failed
    Error setting up gfxboot
    boot:
  

  The advice online was to type help at this prompt, but that failed for me on Ubuntu 16.04. It turns out that gfxboot has improved since 16.04 came out, so 17.10 got me beyond this point, despite displaying the same error.

• One of the USB ports was faulty.

I had intended to install from one USB stick to another, using the internal 128MB drive as a boot partition. The faulty USB port meant I could only plug in one USB stick and a keyboard.

I found out that I can install to the USB I booted from if I boot with the toram boot option. The way to do this was to type install toram at the boot: prompt instead of typing help.

The standard Ubuntu Server image failed to format the USB it was running from with this method, but it worked a treat on the Ubuntu 17.10 Network installer. (Confusingly, the file was called mini.iso, unlike the other Ubuntu downloads, which, on the whole, are named descriptively.)

The installation went swimmingly from there on. I thought things might have gone wrong a couple of times, but nothing turned out to be an issue. Notably:

• The installer seemed stuck at 20% of “Creating swap file”, but it did eventually get moving again.
• My concern that automatic updates would fill up the /boot/ partition with old kernels, was based on out of date information, so I was able to turn them on.
• I was asked what software I wished to install. I selected “OpenSSH Server” (which is required to SSH into the machine over the LAN) and “Basic Ubuntu Server” (which, in retrospect, I probably didn’t need).
• Installing Grub to the Master Boot Record didn’t seem to cause the chaos I thought it might.
• Now that it’s installed, nothing after the BIOS appears on the monitor. I just get a blank screen. I suspect that there is an incorrect video setting somewhere. That doesn’t cause me any issues, however. I’m able to SSH in, and I do not plan on having a monitor plugged in.

So now I have Ubuntu installed! Yatta!

I have a small list of things to do next:

• This thing is loud. I’d like to pop some new fans in it.
• I’ll be installing Pi Hole and NextCloud.
• There’s a tiny screen on the front. I’d like to see if can get it to display anything.
• It’s a NAS… Maybe I should put some hard drives in it…

from __future__ import print_function
from collections import defaultdict
from os.path import join

# The `pkg_resources` module isn't in the standard library,
# but it's frequently available anywhere the `setuptools`
# module is (it typically comes with `pip`).
import pkg_resources


licences = defaultdict(list)

# We go over each installed package:
for package in pkg_resources.working_set:
    # Find the directory where the metadata is stored.
    info_dir = package._provider.egg_info
    if info_dir is None:
        print('No data for {}'.format(package))
        continue
    # Determine if the package is a wheel...
    is_wheel = info_dir.endswith('.dist-info')
    # ...and chose correct file.
    filename = 'METADATA' if is_wheel else 'PKG-INFO'
    filepath = join(info_dir, filename)

    # We then loop over the lines in the file...
    with open(filepath, 'r') as lines:
        for line in sorted(lines):
            # ...find the line that stores the license...
            if (line.startswith('Classifier: License :: ') or
                    line.startswith('License: ')):
                # ... and add the package to the list for it.
                license = line.rsplit(': ')[-1].strip()
                licences[license].append(package)
                break
        else:
            print('No data for {}'.format(package))

# Then all we have to do is print it out:
for licence, packages in licences.items():
    print(licence)
    for package in packages:
        print('\t', package)

Note that this only works for packages where the correct information has been added to the dependancy’s setup.py.

A few days ago, I was asked if I knew a good introduction guide. I don’t really. Rather than disappoint, I typed up some quick tips, and I was encouraged to write them down publicly. The particulars of the process are already covered very well elsewhere, so, in lieu of anything coherent, here’re a few disjointed thoughts on the subject.

To start with, you can’t go wrong with the “How to Bullet Journal” video (~4 mins) from the project’s main home page. It’s a short video explaining how the system works, and why you might want to give it a go.

If you pick it up, after a while, you might start looking for further inspiration. There are a lot of great ideas on the Bullet Journal blog. Scanning through the pictures will reveal a lot of (dauntingly artistic) alternative layouts for the main “modules”. Don’t worry if you’re not super-artistic. There’s a chance that might come with time, but I recommend focussing on the routine and utility. You can always add flair later.

I like to keep a lot of lists in the book, and transfer them to the new book once I’ve filled one up. I have lists for “movies/tv to watch”, “books to read”, “games to play”, “birthdays”, and others. Recently, as I’ve made more of an effort to cook, I’ve been recording recipes I like too.

As for the book you use, I highly recommend dotted grid-paper. I use the large Moleskine hard-cover books, but the Leuchtturm 1917 books look pretty nice too. The Leuchtturm books have slightly broader pages. They both ought to be good for fountain-pen use, if that’s your thing. It’s my thing.

Finally, don’t forget that it’s your book. By this I mean: it doesn’t matter if you stray from the guidelines, tear out a few pages, and scribble all over the others. Go nuts. Adapting, inventing you own way of doing things, and skipping the things that get in your way is greatly encouraged.

Tests with assertNumQueries failing? Want a way to inspect your SQL queries in your tests? I have an answer.

I expect you have a test that looks something like this:

class GoodTestCase(django.test.TestCase):
    def test_fantastic_stuff(self):
        with self.assertNumQueries(1):
            thing.do_fantastic_stuff()  # Queries happen here.

..and unexpectedly you’re getting an error that looks a bit like this:

FAIL: test_fantastic_stuff (greatproject.tests.GoodTestCase)
------------------------------------------------------
Traceback (most recent call last):
  File "/greatproject/tests.py", line 130, in test_fantastic_stuff
    thing.do_fantastic_stuff()
  File "/path/to/django/test/testcases.py", line 105, in __exit__
    executed, self.num

AssertionError: 2 queries executed, 1 expected

Okay, so we know something’s wrong, but we don’t know what. How do we find out what that extra query is? Well, Django can show you the queries run if you just ask it nicely. Lets throw some debug code in the test to print them out…

from django.db import connection

class GoodTestCase(django.test.TestCase):
    def test_fantastic_stuff(self):
        with self.assertNumQueries(1):
            # Get num queries run before.
            so_far = len(connection.queries)
            thing.do_fantastic_stuff()  # Queries happen here.
            for query in connection.queries[so_far:]:
                print(query['sql'])  # Print our queries

When you run this, you should get the executed queries printed in your test output:

SELECT (1) AS "a" FROM "great_model" WHERE "great_model"."id" = 15  LIMIT 1
UPDATE "great_model" SET "field" = 1 WHERE "great_model"."id" = 15

Nice, eh? That should be enough to set you on the path of debugging the issue, but I feel like going a little further (too far?) and making it pretty. Wouldn’t it have been nicer to run the following? (Note the extra context manager.)

class GoodTestCase(django.test.TestCase):
    def test_fantastic_stuff(self):
        with self.assertNumQueries(1), PrintQueries():
            thing.do_fantastic_stuff()  # Queries happen here.

I think so too, so here’s my definition of PrintQueries:

from django.db import connection

class PrintQueries(object):
    def __enter__(self):
        # Get num queries run before.
        self.so_far = len(connection.queries)

    def __exit__(self, type, value, traceback):
        for query in connection.queries[self.so_far:]:
            print(query['sql'])  # Print our queries.

I hope you find it useful! :)

1. Add the following to your /etc/postgresql/9.1/main/pg_hba.conf just below where is says # Put your actual configuration here.

   Replace meshy with your login name. (If you don’t know your username you can get it by running whoami in your terminal.)

    host    all     meshy       127.0.0.1/32            trust
    local   all     meshy                               trust
    local   all     all                                 ident
   

2. Restart postgres

    sudo -u postgres service postgresql restart
   

3. Set up postgres.

   Run postgres:

    sudo -u postgres psql
   

   Create superuser and database.

    create user meshy;
    alter role meshy with superuser;
    alter role meshy login;
    create database meshy;
   

4. This step is not required, and should NEVER be used in production. If you are fond of the data in any of your databases using this install of postgres Do Not Use This.

   That said, if the data doesn’t matter, and your only concern is speed, this should sharply increase database creation time and speed up your tests.

   Final warning: Do NOT use on machines with important data! You have been warned.

   Edit /etc/postgresql/9.1/main/postgresql.conf with:

    # WARNING! This is dangerous!
    # These settings are likely to cause corruption across all databases.
    fsync = off
    synchronous_commit = off
    full_page_writes = off
   

5. Profit.

    psql
   

I’m using this to run Django tests on Ubuntu 12.04 and 13.04, but I’m sure this is fit for many other purposes. Have fun, and happy hacking!