Avery Uslaner

(Failing to) Run a VM on Fedora 35 with the Framework Laptop

Sun, 20 Mar 2022 17:35:00 GMT

Virt-Manager (QEMU/KVM)

The first blocker encountered during setup of my Parrot OS VM is that the VM creation wizard reports that it “Failed to find a suitable default network.”

If I ignore the problem and attempt to hit finish, it tells me it is “Unable to complete install: ’internal error: No <source> ‘bridge’ attribute specified with <interface type=‘bridge’/>’”.

I’m not entirely sure what device name it’s looking for but I’ll try giving it the name of my wireless network device which I’ll grab from ip addr show. With that, I get the error “Unable to complete install: ‘Unable to add bridge wlp170s0 port vnet0: Operation not supported’”.

I’ll take that to mean the device name is valid, but it doesn’t support being used in this way? Googling around seems to suggest I can create my own bridge interface which sounds useful so we can try that.

$ virsh iface-bridge wlp170s0 br0

error: Failed to define new bridge interface br0
error: this function is not supported by the connection driver: virInterfaceDefineXML

Ah nuts. Well that error leads me to https://help.ubuntu.com/community/KVM/Networking#bridgednetworking which helpfully explains that wireless devices can’t be used for creating a network bridge. That’s a bummer because my laptop does not contain an ethernet card. So maybe I’m out of luck?

Let’s try a few more things gathered from google:

sudo ip link add br0 type bridge

Ok cool, I’ve successfully created a bridged network interface called br0, virtual machine manager happily accepts it as the device name and actually lets me boot into my new VM. So that’s pretty good. But I still haven’t specified a network device to actually bridge to so not surprisingly my VM doesn’t seem to have any network connectivity. It’s possible I’m still doomed to failure.

I’ll try linking it my built in wireless card and a USB wireless adapter:

$ sudo ip link set wlp170s0 master br0                                                                   ─╯

Error: Device does not allow enslaving to a bridge.

$ sudo ip link set wlp0s20f0u3 master br0

Error: Device does not allow enslaving to a bridge.

Balls. So I guess bridge networking is off the table until the Framework laptop comes out with an ethernet expansion card.

Alright, well I don’t see why I shouldn’t be able to pass my USB wireless adapter directly to the VM so it can get it’s own IP address that way.

Fortunately, that seems to be as easy as choosing Virtual Machine > Redirect USB Device and selecting my adapter.

I can see it in the VM with ip link show but don’t seem to be able to use it yet. Attempting to bring it up:

$ sudo ifconfig wlxe8de270574fd up

SIOCSIFFLAGS: Input/output error

$ sudo ip link set wlxe8de270574fd up

RTNETLINK answers: Input/output error

Attempting to reload the ath9k_htc kernel driver doesn’t seem to help.

Dmesg shows the error: “ath: phy3: Unable to reset hardware; reset status -5 (freq 2412 MHz)”. The only guess I have at this point is that it’s a driver issue which seems difficult to resolve when I don’t have network connectivity to attempt other drivers. I’ll attempt a different distro.

Bleh same issue with Ubuntu 20.04 :(

Setting up Geodjango on Arch Linux

Thu, 24 Jun 2021 02:02:00 GMT

Installing Postgres

sudo pacman -S postgresql

Setting up data directory within my /home partition:

mkdir -p /home/myuser/postgresdb_data/data
sudo chown -R postgres:postgres /home/myuser/postgresdb_data

Initializing data:

sudo -iu postgres
initdb -D /home/myuser/postgresdb_data/data

Edit the systemd service file:

sudo systemctl edit postgresql

Add the following:

[Service]
Environment=PGROOT=/home/myuser/postgresdb_data
PIDFile=/home/myuser/postgresdb_data/data/postmaster.pid
ProtectHome=false

Edit /home/myuser/postgresdb_data/data/postgresql.conf to listen exclusively through UNIX sockets and we’ll also set password encryption to scram-sha-256:

...
listen_addresses = ''
...
password_encryption = scram-sha-256
...

Edit /home/myuser/postgresdb_data/data/pg_hba.conf to restrict access rights to the database superuser and configure access for our django user:

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             postgres				peer
local	geodjango	db_user				scram-sha-256

Installing PostGIS

sudo pacman -S postgis

Create the spatial database:

sudo -iu postgres
createdb geodjango
psql geodjango
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;
CREATE EXTENSION fuzzystrmatch;
CREATE EXTENSION postgis_tiger_geocoder;

Create The Database User

CREATE USER db_user WITH PASSWORD 'secret-password';
ALTER ROLE db_user SET client_encoding TO 'utf8';
ALTER ROLE db_user SET default_transaction_isolation TO 'read committed';
ALTER ROLE db_user SET timezone TO 'UTC';
GRANT ALL PRIVILEGES ON DATABASE geodjango TO db_user;

Errors

At this point if I try to interact with the database, I get an error about a missing pgrouting file so apparently I’m still missing that part.

ERROR:  could not open extension control file "/usr/share/postgresql/extension/pgrouting.control": No such file or directory

Installing pgRouting

git clone git://github.com/pgRouting/pgrouting.git
cd pgrouting
git checkout v3.1.3
mkdir build
cd build
cmake ..

Ah shit, another error:

CMake Error at /usr/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Boost (missing: Boost_INCLUDE_DIR) (Required is at least
  version "1.53.0")
Call Stack (most recent call first):
  /usr/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.20/Modules/FindBoost.cmake:2345 (find_package_handle_standard_args)
  CMakeLists.txt:76 (find_package)

Install boost

sudo pacman -S boost

Back to installing pgRouting:

cd pgrouting/build
cmake ..
make
sudo make install

Ok sweet, now to create the extension:

sudo -iu postgres
psql geodjango
CREATE EXTENSION pgrouting;

Done!

Fixing RabbitMQ Sources Repository

Tue, 01 Jun 2021 15:52:00 GMT

The Error

Err:10 https://dl.bintray.com/rabbitmq-erlang/debian bionic InRelease
  403  Forbidden [IP: <redacted> 443]
E: Failed to fetch https://dl.bintray.com/rabbitmq-erlang/debian/dists/bionic/InRelease  403  Forbidden [IP: <redacted> 443]
E: The repository 'https://dl.bintray.com/rabbitmq-erlang/debian bionic InRelease' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

The Fix

Seems to be related to the shutdown of Bintray sometime around April. So the first step is to delete my /etc/apt/sources.list.d/bintray.erlang.list file which contained deb https://dl.bintray.com/rabbitmq-erlang/debian bionic erlang-22.x.

Then I replace my original /etc/apt/sources.list.d/rabbitmq_rabbitmq-server.list which looked like this:

# this file was generated by packagecloud.io for
# the repository at https://packagecloud.io/rabbitmq/rabbitmq-server

deb https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ bionic main
deb-src https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ bionic main

To this:

## Provides modern Erlang/OTP releases
##
## "bionic" as distribution name should work for any reasonably recent Ubuntu or Debian release.
## See the release to distribution mapping table in RabbitMQ doc guides to learn more.
deb http://ppa.launchpad.net/rabbitmq/rabbitmq-erlang/ubuntu bionic main
deb-src http://ppa.launchpad.net/rabbitmq/rabbitmq-erlang/ubuntu bionic main

## Provides RabbitMQ
##
## "bionic" as distribution name should work for any reasonably recent Ubuntu or Debian release.
## See the release to distribution mapping table in RabbitMQ doc guides to learn more.
deb https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ bionic main
deb-src https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu/ bionic main

Then update the keys:

## Team RabbitMQ's main signing key
sudo apt-key adv --keyserver "hkps://keys.openpgp.org" --recv-keys "0x0A9AF2115F4687BD29803A206B73A36E6026DFCA"
## Launchpad PPA that provides modern Erlang releases
sudo apt-key adv --keyserver "keyserver.ubuntu.com" --recv-keys "F77F1EDA57EBB1CC"
## PackageCloud RabbitMQ repository
curl -1sLf 'https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey' | sudo apt-key add -

Then a simple:

sudo apt-get update -y
sudo apt-get upgrade -y

And we’re good!

Switching Between Networks With wpa_supplicant

Sun, 16 May 2021 19:34:00 GMT

Background

My setup manages its Wi-Fi connection with wpa_supplicant but doesn’t have a desktop environment so it’s kind of annoying when I want to switch VLANs to manage my IoT devices.

The two networks I’m switching between are defined in /etc/wpa_supplicant/wpa_supplicant-wlp2s0.conf and their priority set by the priority variable. To make switching easier, I’ve made a script to flip the priorities and restart the wpa_supplicant systemd service.

Solution

#!/usr/bin/python3
import subprocess


with open('/etc/wpa_supplicant/wpa_supplicant-wlp2s0.conf', 'r') as f:
    filedata = f.readlines()

for i, line in enumerate(filedata):
    if 'priority=1' in line:
        filedata[i] = line.replace('priority=1', 'priority=0')
    elif 'priority=0' in line:
        filedata[i] = line.replace('priority=0', 'priority=1')

with open('/etc/wpa_supplicant/wpa_supplicant-wlp2s0.conf', 'w') as f:
    f.writelines(filedata)

subprocess.run(['systemctl', 'daemon-reload'], check=True)
subprocess.run(['systemctl', 'restart', 'wpa_supplicant@wlp2s0'], check=True)

Setting Up The Chia Network On A Synology NAS

Sun, 16 May 2021 02:21:00 GMT

This is probably dumb but I’m doing it anyway

Chia is going to be the next big crypto craze. Or at least that’s what their website tries to make the case for.

Also, it’s just kind of fun to play around with. So that’s what this post is about, especially since the HDDs in my NAS are not the recommended method of plotting and farming Chia.

Setup

I’ll be using the Virtual Machine Manager of my Synology DS918+ to run a VM with three CPU cores and 2 GB of memory. I configured it to use 3 virtual drives; one with 50 GB of space and the other two with 1 TB each. The 50 GB drive was used to install Ubuntu 20.04.

During installation of the OS, each virtual physical drive was also configured as its own LVM volume group.

Post-Install Setup

Some further configuration of the volume groups where farm-vg0 and tmp-vg0 are the names of the volume groups configured during OS installation:

sudo lvcreate -l 100%FREE -n farm farm-vg0
sudo lvcreate -l 100%FREE -n farm tmp-vg0

Making the filesystems:

sudo mkfs.ext4 /dev/farm-vg0/farm
sudo mkfs.ext4 /dev/tmp-vg0/tmp

I added the following to my fstab file to mount the logical volumes automatically:

/dev/mapper/farm--vg0-farm /mnt/farm ext4 defaults 0 0
/dev/mapper/tmp--vg0-tmp /mnt/tmp ext4 defaults 0 0

Chia Install

Following their install guide:

git clone https://github.com/Chia-Network/chia-blockchain.git -b latest --recurse-submodules
cd chia-blockchain

sh install.sh

. ./activate

Chia Setup

chia init
chia keys generate
chia start farmer
chia plots create -n 3 -b 1536 -t /mnt/tmp -d /mnt/farm -r 2

Conclusions

The above command took about 5 days to complete. That’s bad. This setup is bad. But also fun and interesting.

Installing & Configuring Mariadb for Arch Linux

Thu, 15 Apr 2021 23:13:00 GMT

Background

This website runs on a Digital Ocean Ubuntu VPS, and uses MySQL as it’s backend database. I want to use my Arch Linux laptop to develop and test new code for the website so I need a database to use for the test environment. I could use Docker which would probably be the better option if I wanted to emulate the production database on this VPS as closely as possible but I’ve never used Mariadb so I want to give it a try.

Special Considerations

My laptop is setup with separate partitions for root, home, and swap with the home partition being the largest so I’m going to be configuring Mariadb to use a location on my home partition as the data directory.

Installation

Following along with the official Arch Linux docs on Mariadb, we install the mariadb package:

pacman -S mariadb

We need to set the datadir in /etc/my.cnf.d/server.cnf so we’ll back up that file and make the change:

cp /etc/my.cnf.d/server.cnf /etc/my.cnf.d/server.cnf.bak
vi /etc/my.cnf.d/server.cnf

...
[mysqld]
datadir=/home/myuser/mariadb_data
skip-networking
socket=/tmp/mysql.sock
...
[mariadb]
socket=/tmp/mysql.sock
...

cp /etc/my.cnf.d/client.cnf /etc/my.cnf.d/client.cnf.bak
vi /etc/my.cnf.d/client.cnf

...
[client]
socket=/tmp/mysql.sock
...

Now we can run the mariadb-install-db command:

sudo mariadb-install-db --user=myuser --basedir=/usr --datadir=/home/myuser/mariadb

Create an override for the mariadb systemd unit file and run it as my user with ProtectHome set to false:

sudo systemctl edit mariadb

[Service]
User=myuser
Group=myuser
ProtectHome=false
PrivateTmp=false

Now we can run the service and if it’s successful, we can enable it to start automatically.

sudo systemctl start mariadb
sudo systemctl enable mariadb

Resizing Partitions

Sun, 14 Feb 2021 19:06:00 GMT

The Background

On my recent Arch install, I decided to create separate partitions for my root and home directories. Unfortunately, I’m a dummy and only allocated 20 GB for the root partition. As it turns out, that’s not enough for a proper workstation setup so now I have to fix my mistake.

The Fix

My particular install involves two usb drives, one with the Arch boot partition and the other with a cryptographic key to unlock the hard drive in my laptop. So to get started, I remove the usb drive with the boot partition and replace it with another usb that holds the Arch live environment I originally used for the installation. This way, I can boot into the live environment, unlock my drive, and resize the partitions while they’re not in use.

Decrypting the Drive

I used dm-crypt plain mode to encrypt the drive so the first step is to unlock it so I can see the partitions. That looks like this:

cryptsetup --cipher=aes-xts-plain64 --offset=0 --key-file=/dev/sdc --key-size=512 open --type plain /dev/sda cryptlvm

In the above code snippet, /dev/sdc is the usb drive containing nothing but my cryptographic key and /dev/sda is the drive I’m decrypting.

At this point, I can see my logical volume group which look something like this:

NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                   8:0      0   400G  0 disk
  └─cryptlvm          254:0    0   400G  0 crypt
    ├─MyVolGroup-swap 254:1    0     8G  0 lvm   [SWAP]
    ├─MyVolGroup-root 254:2    0    20G  0 lvm   /mnt
    └─MyVolGroup-home 254:3    0   372G  0 lvm   /mnt/home

At this point, I can hopefully use lvresize to reduce the size of my home partition and add that extra space to my root partition. So let’s give that a go:

lvresize -L -20G --resizefs MyVolGroup/home
lvresize -l +100%FREE --resizefs MyVolGroup/root

It worked!

Surprisingly painless. And now my root partition is twice the size it used to be so I can install more packages without fear of running out of space.

Setting up a Video Background Login Screen

Mon, 08 Feb 2021 02:16:00 GMT

Background

Recently I was inspired to switch over to Arch Linux after seeing this post by u/EmpressNoodle over at /r/unixporn.

During my Arch install, I decided to use SDDM as my display manager. One of it’s features is the ability to use videos as the background of the login screen so I figured I’d give that a go.

Configuration

First, I had to decide on the base theme that I wanted to customize. My install came with elarun, maldives, and maya (all of which live in /usr/share/sddm/themes) so I used sddm-greeter –test-mode –theme /path/to/theme to preview them and see which one I liked best. I believe Arch defaults to breeze, though I’m not exactly sure where that theme lives and it didn’t matter to me since I ended up picking the maldives theme to base my custom theme on.

To begin, I simply copied the maldives theme to my own custom theme directory so I would have the original to refer back to in case I screwed something up.

sudo cp -r /usr/share/sddm/themes/maldives /usr/share/sddm/themes/custom

At this point, I needed to edit the Main.qml file which handles how the layout looks. I used the guide here to help me get my bearings in terms of how to get it to use a video background: https://nixdaily.com/how-to/add-video-background-to-sddm-login-screen/

Here are the changes I made to the file. I’m not sure if those width and height changes had any effect at all but they didn’t hurt.

➜  ~ diff /usr/share/sddm/themes/maldives/Main.qml /usr/share/sddm/themes/custom/Main.qml
24a25
> import QtMultimedia 5.15
30,31c31,32
<     width: 640
<     height: 480
---
>     width: 1920
>     height: 1080
55,56c56,57
<     Background {
<         anchors.fill: parent
---
>     MediaPlayer {
>         id: videoPlayer
58,63c59,65
<         fillMode: Image.PreserveAspectCrop
<         onStatusChanged: {
<             if (status == Image.Error && source != config.defaultBackground) {
<                 source = config.defaultBackground
<             }
<         }
---
>         autoPlay: true
>     }
>
>     VideoOutput {
>         source: videoPlayer
>         anchors.fill: parent
>         fillMode: VideoOutput.PreserveAspectCrop

As you can see, the video background file is being pulled from config.defaultBackground. That setting is found in theme.conf in the same directory as Main.qml. It’s a simple file that looks like this:

[General]
background=background_video.mp4

Of course, you’ll need to make sure that background_video.mp4 actually exists. I tried setting the path to a video located in my home directory but it didn’t work. Not sure if that’s because my home directory is on a separate encrypted partition, but the simple fix is just to save the video into the theme directory alongside the config files.

At this point you can start testing your theme with sddm-greeter –test-mode –theme /path/to/custom/theme

I needed to install a few other packages to get my video to play properly. Your mileage my vary.

sudo pacman -S gst-libav gstreamer-vaapi

Once you have it working, the last step to to tell sddm to use the them by creating a new file in /etc/sddm.conf.d/

I named mine custom_theme.conf. It looks like this:

[Theme]
Current=custom

Final Product

You can see a short demonstration of my new login screen in the video below. The background I’m using is a lightly edited version of another YouTube video you can find here: https://youtu.be/BHACKCNDMW8

The above video isn’t actually a perfect representation of my login screen, but it’s close enough. I’m actually just using ffmpeg to record my main display while I run sddm-greeter –test-mode with my customized theme.

Automatic Bird Identification: Part III

Mon, 11 May 2020 20:12:00 GMT

Training the Model

Now we can actually write the script that will train the model!

# USAGE
# python fine_tune_birds.py --vgg vgg16/vgg16 --checkpoints checkpoints --prefix vggnet

from config import bird_config as config
import mxnet as mx
import argparse
import logging
import os

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-v", "--vgg", required=True,
                help="path to pre-trained VGGNet for fine-tuning")
ap.add_argument("-c", "--checkpoints", required=True,
                help="path to output checkpoint directory")
ap.add_argument("-p", "--prefix", required=True,
                help="name of model prefix")
ap.add_argument("-s", "--start-epoch", type=int, default=0,
                help="epoch to restart training at")
args = vars(ap.parse_args())

# set the logging level and output file
logging.basicConfig(level=logging.DEBUG,
                    filename="training_{}.log".format(args["start_epoch"]),
                    filemode="w")

# determine the batch
batchSize = config.BATCH_SIZE * config.NUM_DEVICES

# construct the training image iterator
trainIter = mx.io.ImageRecordIter(
    path_imgrec=config.TRAIN_MX_REC,
    data_shape=(3, 224, 224),
    batch_size=batchSize,
    rand_crop=True,
    rand_mirror=True,
    rotate=15,
    max_shear_ratio=0.1,
    mean_r=config.R_MEAN,
    mean_g=config.G_MEAN,
    mean_b=config.B_MEAN,
    preprocess_threads=config.NUM_DEVICES * 2)

# construct the validation image iterator
valIter = mx.io.ImageRecordIter(
    path_imgrec=config.VAL_MX_REC,
    data_shape=(3, 224, 224),
    batch_size=batchSize,
    mean_r=config.R_MEAN,
    mean_g=config.G_MEAN,
    mean_b=config.B_MEAN)

# initialize the optimizer and the training contexts
opt = mx.optimizer.SGD(learning_rate=5e-5, momentum=0.9, wd=0.0005,
                       rescale_grad=1.0 / batchSize)
ctx = [mx.gpu(0)]

# construct the checkpoints path, initialize the model argument and
# auxiliary parameters, and whether uninitialized parameters should
# be allowed
checkpointsPath = os.path.sep.join([args["checkpoints"],
                                    args["prefix"]])
argParams = None
auxParams = None
allowMissing = False

# if there is no specific model starting epoch supplied, then we
# need to build the network architecture
if args["start_epoch"] <= 0:
    # load the pre-trained VGG16 model
    print("[INFO] loading pre-trained model...")
    (symbol, argParams, auxParams) = mx.model.load_checkpoint(
        args["vgg"], 0)
    allowMissing = True

    # grab the layers from the pre-trained model, then find the
    # dropout layer *prior* to the final FC layer (i.e., the layer
    # that contains the number of class labels)
    # HINT: you can find layer names like this:
    # for layer in layers:
    # 	print(layer.name)
    # then, append the string `_output` to the layer name
    layers = symbol.get_internals()
    net = layers["drop7_output"]

    # construct a new FC layer using the desired number of output
    # class labels, followed by a softmax output
    net = mx.sym.FullyConnected(data=net,
                                num_hidden=config.NUM_CLASSES, name="fc8")
    net = mx.sym.SoftmaxOutput(data=net, name="softmax")

    # construct a new set of network arguments, removing any previous
    # arguments pertaining to FC8 (this will allow us to train the
    # final layer)
    argParams = dict({k: argParams[k] for k in argParams
                      if "fc8" not in k})

# otherwise, a specific checkpoint was supplied
else:
    # load the checkpoint from disk
    print("[INFO] loading epoch {}...".format(args["start_epoch"]))
    (net, argParams, auxParams) = mx.model.load_checkpoint(
        checkpointsPath, args["start_epoch"])

# initialize the callbacks and evaluation metrics
batchEndCBs = [mx.callback.Speedometer(batchSize, 50)]
epochEndCBs = [mx.callback.do_checkpoint(checkpointsPath)]
metrics = [mx.metric.Accuracy(), mx.metric.TopKAccuracy(top_k=2),
           mx.metric.CrossEntropy()]

# construct the model and train it
print("[INFO] training network...")
model = mx.mod.Module(symbol=net, context=ctx)
model.fit(
    trainIter,
    eval_data=valIter,
    num_epoch=65,
    begin_epoch=args["start_epoch"],
    initializer=mx.initializer.Xavier(),
    arg_params=argParams,
    aux_params=auxParams,
    optimizer=opt,
    allow_missing=allowMissing,
    eval_metric=metrics,
    batch_end_callback=batchEndCBs,
    epoch_end_callback=epochEndCBs)

A few important things to note:

This script makes use of a pretrained imagenet model so you’ll need to provide those pretrained weights: params & symbols
I’ve included top k 2 accuracy which is a literally useless metric but the 2 can just be incremented to a more useful metric when I have more categories of training data ready
The hyperparameters used here are probably terrible but it doesn’t make sense to worry about those when we still have so little training data
I’m training with a single GTX 1080 graphics card so my ctx variable is set to [mx.gpu(0)]. Add more cards to that list and change the NUM_DEVICES variable in the config file if you have more cards to train with

Checking the Output

The good news is that it runs! An exert of my training_0.log file looks like this:

INFO:root:Epoch[0] Train-accuracy=0.710938
INFO:root:Epoch[0] Train-top_k_accuracy_2=1.000000
INFO:root:Epoch[0] Train-cross-entropy=0.704148
INFO:root:Epoch[0] Time cost=4.002
INFO:root:Saved checkpoint to "/ML/Birds/checkpoints/vggnet-0001.params"
INFO:root:Epoch[0] Validation-accuracy=0.812500
INFO:root:Epoch[0] Validation-top_k_accuracy_2=1.000000
INFO:root:Epoch[0] Validation-cross-entropy=0.392680
INFO:root:Epoch[1] Train-accuracy=0.812500
INFO:root:Epoch[1] Train-top_k_accuracy_2=1.000000
INFO:root:Epoch[1] Train-cross-entropy=0.394014
INFO:root:Epoch[1] Time cost=3.758
INFO:root:Saved checkpoint to "/ML/Birds/checkpoints/vggnet-0002.params"
INFO:root:Epoch[1] Validation-accuracy=0.906250
INFO:root:Epoch[1] Validation-top_k_accuracy_2=1.000000

Classes: 2

Training Images: 372

Validation Images: 72

We can also graph the data from the training logs to get a better idea of how it’s doing.

The graphs look pretty good but that’s just because the model is able to overfit super quickly since there are so few data to learn from. I also haven’t yet fixed the problem of images without birds being incorrectly classified so there’s no reason too read to much into these results. Since we know the data being fed to the model isn’t high quality yet, we know that the output can’t be of high quality. So really this is a good example of why you should be skeptical of the output of machine learning models until you’ve thoroughly scrutinized them.

Now that I know that the training pipeline is actually functional, I can focus my efforts on building a more robust training set. Then we can fuddle with the hyperparameters and actually start thinking more critically about the results that we’re getting.

Automatic Bird Identification: Part II

Sun, 10 May 2020 20:00:00 GMT

In this post I’ll be concentrating on creating the lst and rec files needed for model training using the training data that we generated in Part I of this series.

The model will use the mxnet framework which reads rec files as training data. First we’ll create the lst files guided by a config file that we can define now:

The Config File

from os import path

# top level path to our bird related directories
BASE_PATH = "/ML/Birds"

# based on the base path, derive the images path
IMAGES_PATH = path.sep.join([BASE_PATH, "DATA"])

LABELS_PATH = "/bird_ml/dataset.csv"

MX_OUTPUT = BASE_PATH
TRAIN_MX_LIST = path.sep.join([MX_OUTPUT, "lists/train.lst"])
VAL_MX_LIST = path.sep.join([MX_OUTPUT, "lists/val.lst"])
TEST_MX_LIST = path.sep.join([MX_OUTPUT, "lists/test.lst"])

TRAIN_MX_REC = path.sep.join([MX_OUTPUT, "rec/train.rec"])
VAL_MX_REC = path.sep.join([MX_OUTPUT, "rec/val.rec"])
TEST_MX_REC = path.sep.join([MX_OUTPUT, "rec/test.rec"])

# Label encoder used to convert species names to integer categories
LABEL_ENCODER_PATH = path.sep.join([BASE_PATH, "output/le.cpickle"])

# define the RGB means from the ImageNet dataset
R_MEAN = 123.68
G_MEAN = 116.779
B_MEAN = 103.939

# define the percentage of validation and testing images relative
# to the number of training images
NUM_CLASSES = 2
NUM_VAL_IMAGES = 0.15
NUM_TEST_IMAGES = 0.15

# define the batch size
BATCH_SIZE = 32
NUM_DEVICES = 1

The RBG means are taken from the original Imagenet paper by Simonyan and Zisserman. That way we can fine tune an existing Imagenet trained model rather than training our model entirely from scratch.

This first model will only attempt to classify 2 classes since we don’t have much training data yet and we only want to make sure our data processing pipeline is functioning properly. We’re not expecting a high quality model just yet.

Building the Record File Dataset

While our image dataset is already created, our model actually needs to consume rec files rather than raw image files so creating those will be our next step. The rec files are essentially just images compressed into the record file format. Their creation will be handled by the im2rec.py file included with the MxNet framework, but we’ll use our own script to create the lst files that serve to pair each image filepath with it’s encoded bird species class.

from config import bird_config as config
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
import progressbar
import pickle
import os

# read the contents of the labels file, then initialize the list of
# image paths and labels
print("[INFO] loading image paths and labels...")
rows = open(config.LABELS_PATH).read()
rows = rows.strip().split("\n")[1:]
trainPaths = []
trainLabels = []

# loop over the rows
for row in rows:
    # unpack the row, then update the image paths and labels list
    filename, species = row.split(",")
    filename = filename[filename.rfind("/") + 1:]
    trainPaths.append(os.sep.join([config.IMAGES_PATH, filename]))
    trainLabels.append(species)

# now that we have the total number of images in the dataset that
# can be used for training, compute the number of images that
# should be used for validation and testing
numVal = int(len(trainPaths) * config.NUM_VAL_IMAGES)
numTest = int(len(trainPaths) * config.NUM_TEST_IMAGES)

# our class labels are represented as strings so we need to encode
# them
print("[INFO] encoding labels...")
le = LabelEncoder().fit(trainLabels)
trainLabels = le.transform(trainLabels)

# perform sampling from the training set to construct a a validation
# set
print("[INFO] constructing validation data...")
split = train_test_split(trainPaths, trainLabels, test_size=numVal,
                         stratify=trainLabels)
(trainPaths, valPaths, trainLabels, valLabels) = split

# perform stratified sampling from the training set to construct a
# a testing set
print("[INFO] constructing testing data...")
split = train_test_split(trainPaths, trainLabels, test_size=numTest,
                         stratify=trainLabels)
(trainPaths, testPaths, trainLabels, testLabels) = split

# construct a list pairing the training, validation, and testing
# image paths along with their corresponding labels and output list
# files
datasets = [
    ("train", trainPaths, trainLabels, config.TRAIN_MX_LIST),
    ("val", valPaths, valLabels, config.VAL_MX_LIST),
    ("test", testPaths, testLabels, config.TEST_MX_LIST)]

# loop over the dataset tuples
for (dType, paths, labels, outputPath) in datasets:
    # open the output file for writing
    print("[INFO] building {}...".format(outputPath))
    f = open(outputPath, "w")

    # initialize the progress bar
    widgets = ["Building List: ", progressbar.Percentage(), " ",
               progressbar.Bar(), " ", progressbar.ETA()]
    pbar = progressbar.ProgressBar(maxval=len(paths),
                                   widgets=widgets).start()

    # loop over each of the individual images + labels
    for (i, (path, label)) in enumerate(zip(paths, labels)):
        # write the image index, label, and output path to file
        row = "\t".join([str(i), str(label), path])
        f.write("{}\n".format(row))
        pbar.update(i)

    # close the output file
    pbar.finish()
    f.close()

# write the label encoder to file
print("[INFO] serializing label encoder...")
f = open(config.LABEL_ENCODER_PATH, "wb")
f.write(pickle.dumps(le))
f.close()

Running our build dataset script will create the lst files in the directory defined by the config file we created earlier. Now that we have those, we can run the im2rec script to generate the rec files.

To make things easy I grabbed the im2py script off of MxNet’s github and put it in my project directory:

cd /bird_ml/
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/im2rec.py

To make generation of the rec files easier, I wrote a small script to create and organize them.

build_rec_files.py:

#!/bin/bash

/bird_ml/venv/bin/python /bird_ml/im2rec.py \
    --resize 256 --quality 100 --encoding '.jpg' /ML/Birds/lists ""

mv /ML/Birds/lists/*.idx /ML/Birds/rec
mv /ML/Birds/lists/*.rec /ML/Birds/rec

Now that we have our rec files, we’re ready to start training! I’ll handle that in the next post.

Automatic Bird Identification: Part I

Thu, 07 May 2020 03:04:00 GMT

Intro

I have a bird feeder with a Raspberry Pi camera pointed at it. That raspberry pi creates a video feed of the bird feeder accessible on my local network. My local network has a Synology 918+ with a IP camera license. The Synology configured IP camera detects motion from birds (and wind unfortunately), and saves a short video of the motion events to disk.

[Image unavailable: raspberry-pi-zero-bird-cam.jpg]

The Plan

At first I will manually watch each motion event video to organize the videos by bird species. I’ll do this by placing each video into a folder with the name of the bird species as the folder’s name. Then, using Python I can split the videos into individual images which I can use to train a machine learning model that automatically identifies the bird species that is visiting the bird feeder.

I’m planning to base my machine learning model off of another machine learning problem described in a book containing example code. To minimize the amount of new code I have to write, I’ll organize the training images in the same way as the example code I’m following. So that is to say all images will be saved into the same directory while a csv file is used to describe their file locations and category labels. I’ll need to write the code to make that happen though.

The Code

The script below traverses the video files organized by species and saves each video frame to it’s own file. The dataset.csv file referenced by the DATA_LOG variable keeps track of all of the file paths and their respective labels.

At this point, there is a problem because significant portions of each video contain no birds when they either haven’t landed yet or have flown away. This means some frames are incorrectly labeled as a particular species when in fact they contain no birds at all. To address this, I’ll move any frames that don’t contain birds into a folder called “Empty feeder”. Later iterations of the below script will traverse this folder, so that these frames can be correct labeled as containing no birds so our future machine learning models can also learn to recognize images without birds in them.

import csv
import os
import time

import cv2

from imutils.video import FileVideoStream

DATA_PATH = "ML/Birds/DATA"
DATA_LOG = "bird_ml/dataset.csv"


def get_video_file():
    for root, dirs, files in os.walk("ML/Birds"):
        try:
            dirs.remove(DATA_PATH)
        except ValueError:
            pass

        for file in files:
            try:
                file_extension = file.split(".")[1]
            except IndexError:
                continue

            if file_extension == "mp4":
                yield os.path.join(root, file)


def split_and_label(video_file, file_number):
    print("[*] Processing", video_file)
    video_stream = FileVideoStream(video_file).start()
    # Allow the buffer some time to fill
    time.sleep(2.0)

    while video_stream.more():
        frame = video_stream.read()
        if frame is None:
            break
        else:
            write_path = os.path.join(DATA_PATH, str(file_number) + ".jpg")
            print("\t[*] Saving file number", file_number)
            cv2.imwrite(write_path, frame)
            file_number += 1
            label = video_file.split('/')[-2]
            with open(DATA_LOG, 'a') as dataset:
                fieldnames = ['Image Filename', 'Species']
                writer = csv.DictWriter(dataset, fieldnames=fieldnames)
                writer.writerow({'Image Filename': write_path, 'Species': label})

            # Sleep again so we don't overtake the buffer
            time.sleep(0.01)

    return file_number


def get_next_filename():
    last_row = open(DATA_LOG, 'r').readlines()[-1]
    file_path = last_row.split(',')[0]
    file_name = file_path.split('/')[1]
    file_number = file_name.split('.')[0]
    return int(file_number) + 1


def main():
    next_file_num = get_next_filename()
    for video_file in get_video_file():
        next_file_num = split_and_label(video_file, next_file_num)

        # Delete the original video file so we don't duplicate training data
        os.remove(video_file)


if __name__ == "__main__":
    main()


def main():
 next_file_num = get_next_filename()
 for video_file in get_video_file():
 next_file_num = split_and_label(video_file, next_file_num)

 # Delete the original video file so we don't duplicate training data
 os.remove(video_file)


if __name__ == "__main__":
 main()

Securing Django + Nginx With TLS

Sun, 09 Feb 2020 00:25:00 GMT

Installing Certbot

We first need to install certbot from Let’s Encrypt since that will do most of the hard work for us.

sudo apt-get update
sudo apt-get install software-properties-common
sudo add-apt-repository universe
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot python-certbot-nginx

Now that certbot it installed, we can grab a certificate. We’ll have daphne consume it so we don’t need certbot to mess with our nginx file.

sudo certbot certonly --nginx

The tool will prompt you for an email address and ask you to agree to a few things before it saves the certificate.

At this point, I need to update my supervisord config file so Daphne can actually make use of it. That config should like something like this:

[fcgi-program:asgi]
# Set Django environment variables
environment=DJANGO_SETTINGS_MODULE="photo_blog.settings.production",SECRET_KEY="<redacted>"

# TCP socket used by Nginx backend upstream
socket=tcp://localhost:8000

# Directory where your site's project files are located
directory=/home/myusername/averyuslaner.com

# Each process needs to have a separate socket file, so we use process_num
# Make sure to update "mysite.asgi" to match your project name
command=/home/myusername/venv/bin/daphne -e ssl:443:privateKey=/etc/letsencrypt/live/averyuslaner.com/privkey.pem:certKey=/etc/letsencrypt/live/averyuslaner.com/fullchain.pem -u /run/daphne/daphnee%(process_num)d.sock --fd 0 --access-log - --proxy-headers mysite.asgi:application

# Number of processes to startup, roughly the number of CPUs you have
numprocs=1

# Give each process a unique name so they can be told apart
process_name=asgi%(process_num)d

# Automatically start and recover processes
autostart=true
autorestart=true

# Choose where you want your log to go
stdout_logfile=/var/log/asgi.log
redirect_stderr=true

Daphne can’t handle TLS unless you have the correct packages installed for Twisted. To meet those requirements, we’ll pip install them:

pip install Twisted[tls,http2]

Then we need to update supervisord:

sudo supervisorctl reread
sudo supervisorctl update

Boom, website secured.

Configuring MySQL for Django

Fri, 07 Feb 2020 16:42:00 GMT

Installing System Packages

The first step is obviously to install MySQL itself, which is easiest to do using your systems package repository. In my case, I’m using Ubuntu 18.04 on a Digital Ocean droplet.

sudo apt-get update
sudo apt-get install mysql-server libmysqlclient-dev

Securing the Database

Now we can work on some initial setup of the new database server. Conveniently, MySQL provides a tool that walks you thru the basics of securing the installation. The tool will ask you to decide on a password for the MySQL root user, so before we get started I strongly suggest you get a password manager ready to create and store a strong password. If you don’t already have a password manager, I’ll recommend Bitwarden, which is free and open-source.

sudo mysql_secure_installation

At this point, you’ll be guided thru a series of questions. For the greatest level of security, just answer ‘yes’ to all of the prompts. I personally declined installing the password checking plugin since I can just use my password manager to create and save long complicated passwords.

The output of the tool will look like this:

Securing the MySQL server deployment.

Connecting to MySQL using a blank password.

VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?

Press y|Y for Yes, any other key for No: n
Please set the password for root here.

New password:

Re-enter new password:
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.


Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.

By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.


Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
- Dropping test database...
Success.

- Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.

All done!

We have a little bit more work to do though because the root password we set isn’t actually being used. By default, the root user’s authentication is set to use the auth_socket plugin, which only checks if the socket user name matches the MySQL user name. You can confirm this by connecting to MySQL and running the following query:

sudo mysql
mysql> SELECT user,authentication_string,plugin,host FROM mysql.user;
+------------------+-------------------------------------------+-----------------------+-----------+
| user             | authentication_string                     | plugin                | host      |
+------------------+-------------------------------------------+-----------------------+-----------+
| root             |                                           | auth_socket           | localhost |
| mysql.session    | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | mysql_native_password | localhost |
| mysql.sys        | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | mysql_native_password | localhost |
| debian-sys-maint | *EUFHL2894JFLJS819FIUSFKLJDFDFJFLSDFJFPOF | mysql_native_password | localhost |
+------------------+-------------------------------------------+-----------------------+-----------+
4 rows in set (0.00 sec)

We’ll want to change the plugin from auth_socket to mysql_native_password. To do so, execute the following command using the same password you came up with earlier:

mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'yourpasswordhere';

Now we can flush the privileges and confirm the change:

mysql> FLUSH PRIVILEGES;
mysql> SELECT user,authentication_string,plugin,host FROM mysql.user;
+------------------+-------------------------------------------+-----------------------+-----------+
| user             | authentication_string                     | plugin                | host      |
+------------------+-------------------------------------------+-----------------------+-----------+
| root             | *IJKHL4567JFLJS093FIUSFKLJDFDFJFLSDFJFLMV | mysql_native_password | localhost |
| mysql.session    | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | mysql_native_password | localhost |
| mysql.sys        | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | mysql_native_password | localhost |
| debian-sys-maint | *EUFHL2894JFLJS819FIUSFKLJDFDFJFLSDFJFPOF | mysql_native_password | localhost |
+------------------+-------------------------------------------+-----------------------+-----------+
4 rows in set (0.00 sec)

Configuring the Database For Django

Now that we have MySQL nice and secure, we can start getting it ready for our Django application. We’ll start by creating a new database and a user with access to that new database. Again, you’ll need to come up with a strong password for your database user.

mysql> CREATE DATABASE mydatabasename CHARACTER SET UTF8;
mysql> CREATE USER myuser@localhost IDENTIFIED BY 'youruserspasswordhere';

When we give our new user access to the database that Django will use and flush the privileges again:

mysql> GRANT ALL PRIVILEGES ON personalsite.* TO myuser@localhost;
mysql> FLUSH PRIVILEGES;

Finally, we need to feed MySQL timezone information to avoid issues with django datetimes. To do that, we’ll use the mysql_tzinfo_to_sql tool. When prompted for a password, use the password you created for the MySQL root user.

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -D mysql -u root -p

Done with MySQL. Go ahead and exit so we can move on.

mysql> exit

Configuring Django

If you already have a virtual environment setup to use with your Django application activate it now. Otherwise, you can create one using virtualenv. You can download the tool using pip like this:

sudo apt-get install python3-pip
pip3 install virtualenv

With virtualenv installed, you can specify a python version and directory to create the virtual environment. In my case, I’ll be using Python 3.8 to create a virtual environment in a directory called ‘venv’ in my home directory. If you want to see how to install Python 3.8 from source for your virtual environment, check out my post on Setting Up Django with Nginx + Daphne.

virtualenv -p python3.8 ~/venv

Active the new environment by sourcing it’s activation script:

source ~/venv/bin/activate

If you don’t yet have a django project ready to go, you can get started by creating a new directory, installing django inside of it, and starting a new project. Note the period (‘.’) at the end of the last command which specifies that you want to install django in the current directory.

mkdir ~/myprojectdirectory
cd ~/myprojectdirectory
pip install django
django-admin.py startproject nameofmyproject .

Now we can actually install mysqlclient which will allow our python app to talk to the database:

pip install mysqlclient

Before starting the Django server, you’ll want to make sure you have your database settings configured in Django. In a default django install, you can find these settings in ~/myprojectdirectory/nameofmyproject/settings.py.

Within this file, look for the DATABASES environment variable. By default, it looks like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

Instead of using sqlite3 though, we want to use MySQL. Modify the code to something like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'OPTIONS': {
            'read_default_file': '/home/myusername/mysql_django.cnf',
            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
            'isolation_level': 'read committed',
        },
    }
}

The above code instructs django to read our database credentials from a configuration file so we’ll need to create that. The init_command and isolation_level arguments also set some other database level settings that are recommended in the Django docs.

Make sure you save your configuration file in the same location that you specified above:

nano /home/myusername/mysql_django.conf

Within the file, add something like the following, substituting in the vales you used when configuring MySQL:

[client]
database = mydatabasename
user = myusername
password = 'mystrongpassword'
host = 127.0.0.1
port = 3306
default-character-set = utf8

And with that done we can finally get django started! We just need to run a few django management commands to get our migrations in order. Make sure you’re in your django project directory where the manage.py script lives.

python manage.py makemigrations
python manage.py migrate

Hopefully this command successfully sets up of your tables in your newly created database. If you run into an error, be sure to double check your configuration file to make sure you gave Django all of the correct credentials to connect to the database. Once the migrations are done, we can finally run the server. Make sure you use the IP address of your own server so you can actually connect to it.

python manage.py runserver 203.0.113.10:8000

In the above command, I’m assuming the remote server running the web app has an IP address of 203.0.113.10 and I’m serving the application thru port 8000. Now from my local computer I can connect to the running web application by typing http://203.0.113.10:8000 into my browser.

If the runserver command seems to be working on your remote server but you still can’t connect to the web app from your local machine, you may need to make sure that port 8000 isn’t being blocked by a firewall on your remote machine.

Now that you have MySQL working with Django, you’ll want to serve it with something other than the runserver command since it isn’t appropriate for use in a production environment. For help with that, check out my blog post on Setting Up Django with Nginx + Daphne.

Setting Up Django with Nginx + Daphne on Ubuntu 18.04

Thu, 06 Feb 2020 23:52:00 GMT

My Environment

I’ll be deploying a Django 3.0 web application on a Digital Ocean droplet running Ubuntu 18.04. The web app will be served by Daphne and Nginx.

Installing Things

Python

I’ll be running Django using Python 3.8.1 which isn’t currently available as a package on Ubuntu so I’ll be installing it from source. The files are available on python.org.

You could use wget to download the files straight from the remote machine, but I chose to just download it locally and transfer it to my remote home directory with scp.

scp Python-3.8.1.tar.xz username@203.0.113.10:/home/username

Dependencies

Before we install Python, we need to make sure we have a few packages installed:

sudo apt-get update && sudo apt-get install libbz2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev zlib1g-dev libffi-dev

Now we get go ahead with our Python install. I personally like to install third-party application in the /opt directory, so I’ll do that here.

Move it over, configure, make, test, and install. I’m using altinstall so I we don’t overwrite our primary python installation because I don’t see any downsides to doing so and it seems less likely to result in unintended consequences.

sudo mv ~/Python-3.8.1.tar.xz /opt
cd /opt
sudo tar -xvf Python-3.8.1.tar.xz
sudo rm Python-3.8.1.tar.xz
cd Python-3.8.1/
./configure --enable-ipv6 --enable-optimizations
make
make test
sudo make altinstall

Sweet, so now we have python3.8 installed. Calls to python3 should still be linked to the system default, which in my case is python 3.6. To use 3.8, you’ll actually need to call python3.8 explicitly.

So let’s get a virtual environment setup that we can use to run our Django application. I’ll use virtualenv which we can install via pip. I’ll install the python3 version on my system python to do it:

sudo apt-get install python3-pip
pip3 install virtualenv

Back in our home directory we can use virtualenv to create the environment in a new directory we’ll name ‘venv’.

cd
virtualenv -p python3.8 venv

Django

Since I already have a Django application ready to go, I’ll activate our new virtual environment and use the requirements file from the Django application to install everything the app needs.

source ~/venv/bin/activate
cd ~/averyuslaner.com
pip install -r requirements.txt

My web app uses MySQL for it’s database so I need to install the following before I can successfully pip install mysqlclient. If you’re interested in more details on how to use MySQL with Django, you can check out my other blog post on Setting up MySQL for Django.

By the way, if you don’t have your own requirements file, the only things you need to pip install to follow along are django 3.0+ and daphne.

Once you have at least those two things installed, we can test daphne’s ability to serve the app. It will make use of the asgi.py file that lives in the same directory as the settings.py file on a default install. If you’re upgrading django from version 2 or below, you’ll need to add it yourself.

daphne -b 203.0.113.10 -p 8000 mydjangoproject.asgi:application

The above command binds the application to the given IP and port numbers respectively. You’ll want to replace the example address with the IP address of your own remote machine. If all goes well, your web application should now be accessible from your local machine by entering http://203.0.113.10:8000 into your browser.

Supervising Daphne

Launching daphne manually isn’t ideal so let’s manage it with supervisord. We can start to attack this problem at along with configuring nginx so let’s install both now.

sudo apt-get install supervisor nginx

Next, we write a configuration file for daphne, which we save inside /etc/supervisor/conf.d/.

sudo nano /etc/supervisor/conf.d/averyuslaner.conf

Within the file, add something like what I have below. Note that I’m supplying the absolute path to daphne that’s inside my virtual environment.

[fcgi-program:asgi]
# Set Django environment variables
environment=DJANGO_SETTINGS_MODULE="mydjangoproject.settings.production",SECRET_KEY="<redacted>"

# TCP socket used by Nginx backend upstream
socket=tcp://localhost:8000

# Directory where your site's project files are located
directory=/home/myusername/averyuslaner.com

# Each process needs to have a separate socket file, so we use process_num
# Make sure to update "mydjangoproject.asgi" to match your project name
command=/home/<user>/venv/bin/daphne -u /run/daphne/daphne%(process_num)d.sock --fd 0 --access-log - --proxy-headers mydjangoproject.asgi:application

# Number of processes to startup, roughly the number of CPUs you have
numprocs=1

# Give each process a unique name so they can be told apart
process_name=asgi%(process_num)d

# Automatically start and recover processes
autostart=true
autorestart=true

# Choose where you want your log to go
stdout_logfile=/var/log/asgi.log
redirect_stderr=true

Now we tell supervisor to update so it can read the configuration file:

sudo supervisorctl reread
sudo supervisorctl update

Serving From Nginx

Next, we create the configuration file for nginx which we’ll save in /etc/nginx/sites-available/:

sudo nano /etc/nginx/sites-available/averyuslaner.com

Within the file, add this:

upstream daphne-backend {
    server localhost:8000;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    location / {
        try_files $uri @proxy_to_app;
    }
    location @proxy_to_app {
        proxy_pass http://daphne-backend;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
    }
}

Once that’s done, we simply symlink the file into nginx’s sites-enabled directory and reload nginx.

cd /etc/nginx/sites-enabled
sudo ln -s ../sites-available/averyuslaner.com .
sudo service nginx reload

That should do it! Your web application should now be accessible at your server’s IP address. If that isn’t the case, check your log files to see where the problem might be. Relevant log files can be found in the following places:

Nginx - /var/log/nginx/error.log
Supervisord - /var/log/supervisor/supervisord.log
Daphne process - /var/log/asgi.log

Configuring Ubuntu 18.04 + CUDA 10.0 + NVIDIA GPU For Deep Learning With Tensorflow & OpenCV Python Bindings

Wed, 24 Oct 2018 13:52:00 GMT

Guides Exist for Ubuntu 16.04; Less so for 18.04

This guide will essentially adapt existing guides for 16.04 and address the areas where those guides need to be altered or worked around to achieve the desired results. This solution isn’t as well supported as 16.04 so if you’re just looking for an easy way to get started with machine learning, you should probably look elsewhere. Here we’re all about breaking things and figuring out how to fix them.

This guide draws especially heavily from one written by Adrian Rosebrock over at PyImageSearch.com. And if you’re reading this post, you might also be interested in his latest book, Deep Learning For Computer Vision With Python.

Connecting To Your Development Machine

I’m going to assume you’ll be interacting with your development machine remotely over SSH. This is better than running everything locally anyway because running the X11 server that runs your Graphical User Interface (GUI) while you set up your development environment can eff things up.

If you have no other choice but to run these commands locally, you can switch to another virtual console by pressing:

CTRL + ALT + F3

…logging in and then running:

sudo service lightdm stop

Updating Your Environment

Make sure your package lists are up-to-date:

sudo apt-get update && sudo apt-get upgrade -y

Installing Necessary Packages

This is the first area where we start to run into differences between configuring 16.04 and 18.04.

First, let’s install all of the packages that install painlessly for both systems:

sudo apt-get install build-essential cmake git unzip pkg-config
sudo apt-get install libjpeg-dev libtiff5-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev
sudo apt-get install libgtk-3-dev
sudo apt-get install libhdf5-serial-dev graphviz
sudo apt-get install libopenblas-dev libatlas-base-dev gfortran
sudo apt-get install python-tk python3-tk python-imaging-tk

The following packages require a bit more work:

libjasper-dev\
libpng12-dev

libjasper-dev for Ubuntu 18.04

Thanks to a handy SE answer, we can fix this issue pretty easily:

cd ~/Downloads
wget http://security.ubuntu.com/ubuntu/pool/main/j/jasper/libjasper-dev_1.900.1-debian1-2.4ubuntu1.2_amd64.deb
wget http://security.ubuntu.com/ubuntu/pool/main/j/jasper/libjasper1_1.900.1-debian1-2.4ubuntu1.2_amd64.deb
sudo apt-get install ./libjasper-dev_1.900.1-debian1-2.4ubuntu1.2_amd64.deb ./libjasper1_1.900.1-debian1-2.4ubuntu1.2_amd64.deb

libpng12-dev for Ubuntu 18.04

This package has since been dropped. Instead, we use libpng-dev which conveniently was already installed when we installed libgtk-3-dev.

Installing Python 2.7 and Python 3 Header Files

We’ll be compiling OpenCV with python binding so we’ll need these development packages to do so:

sudo apt-get install python2.7-dev python3-dev

Preparing NVIDIA Drivers

We’ll need to change out the default graphics drivers so we’ll need to install some packages in preparation for that change:

sudo apt-get install linux-image-generic linux-image-extra-virtual
sudo apt-get install linux-source linux-headers-generic

Now we need to disable the default Nouveau driver by creating a blacklist file. Open up the nano text editor with this command:

sudo nano /etc/modprobe.d/blacklist-nouveau.conf

To this file, add the following lines:

blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off

You can now close out the editor by hitting CTRL + X, being careful to select ‘Y’ when prompted to save your changes.

Now we disable Nouveau in the Kernel Mode Settings (KMS):

echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf

The above command edits the file without us having to open a text editor. The first part of the command, echo, returns everything after it, in this case options nouveau modeset=0. The vertical pipe character | in the middle, indicates that we want that output to be fed into the other part of our command, in this case tee. We append that command with sudo because we need admin writes to edit the file located at /etc/modprobe.d/nouveau-kms.conf. The tee command splits what is fed to it between the given file and the terminal output so we can see what it is doing. In this case, we’ve also included the -a flag which instructs the tee command to append the input to the file rather than just overwriting everything in the file.

Now we update run the command that makes note of our previous changes so they will be respected at boot time:

sudo update-initramfs -u

And now we can restart the machine:

sudo reboot

Downloading CUDA Toolkit

Now we download the CUDA Toolkit from the NVIDIA website. Because we like the pain and frustration of the bleeding edge, we’ll be using CUDA Toolkit 10.0 so we’ll select Linux > x86_64 > Ubuntu > 18.04 > deb (network).

If you’ve navigated to the website on a machine that isn’t the one you’re configuring, just copy the link address to the download file and use wget on your Ubuntu machine:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb

Now we just install it:

sudo dpkg -i cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda-10.0

We need to update our path variable so the computer knows where CUDA lives.

Note that I’m editing my ~/.profile file which only gets read when the user logs in. You could replace the ~/.profile part of the following command with ~/.bashrc which I imagine some people would recommend, but those people aren’t me.

echo "export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}" >> ~/.profile

To refresh the PATH variable in your current session, you can either close and reopen a new terminal session or use the source command:

source ~/.profile

You can check to make sure /usr/local/cuda-10.0/bin is included in your path variable by echoing it:

echo $PATH

After rebooting we can verify that things are working properly. Check that the NVIDIA Persistence Daemon is running:

sudo systemctl status nvidia-persistenced

It should say something like:

nvidia-persistenced.service - NVIDIA Persistence Daemon
Loaded: loaded (/lib/systemd/system/nvidia-persistenced.service; static; vendor preset: enabled)
Active: active (running)

Now check that the NVIDA driver version:

cat /proc/driver/nvidia/version

You would expect something like:

NVRM version: NVIDIA UNIX x86_64 Kernel Module 410.48 Thu Sep 6 06:36:33 CDT 2018
GCC version: gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)

Check the version of CUDA Toolkit:

nvcc -V

It should say something like:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

Now let’s get the CUDA samples and make sure they run correctly. Change into whatever directory you’d like to download them to and run:

cuda-install-samples-10.0.sh .

There should now be a directory called NVIDIA_CUDA-10.0_Samples in your current directory. Let’s cd into it and compile the samples:

Note that the make command uses a single CPU core by default but will go much faster if you tell it to use more. You can find out how many cores your system has available by using the command:

lscpu | grep -E ‘^Thread|^Core|^Socket|^CPU(’

That will give you output like:

CPU(s): 12
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1

So in my make command I can say make -j6 to make things go much faster than if I had just executed make without any additional arguments.

cd NVIDIA_CUDA-10.0_Samples/
make

Once that’s done, we’ll execute the deviceQuery script to check if our GPU is working with CUDA. From the same directory you ran the make command, you can run the script like this:

Note the period (.) at the beginning of the command.

./bin/x86_64/linux/release/deviceQuery

This command gives you a bunch of info and at the bottom it should hopefully say:

Result = PASS

Installing cuDNN (CUDA Deep Neural Network library)

Downloading cuDNN requires the creation of an account on the developer.nvidia.com website so it’s probably easier to download the files on the machine you’re sshing from and then copy them over using scp.

The three downloads we’re looking for are cuDNN v7.3.1 for CUDA 10.0. Specifically we’ll be installing the runtime and development libraries plus the code samples and user guide so look for:

cuDNN v7.3.1 Runtime Library for Ubuntu18.04 (Deb)
cuDNN v7.3.1 Developer Library for Ubuntu18.04 (Deb)
cuDNN v7.3.1 Code Samples and User Guide for Ubuntu18.04 (Deb)

When you have all of those downloaded, you can copy them to your development machine. You’ll need to replace < user > and < dev-machine-ip > with their appropriate values. Make sure you run this command from the same directory you downloaded the files to.

scp libcudnn7* < user >@< dev-machine-ip >:/home/<user>/Downloads/

Now, back on the development machine, let’s install them. We copied them over to users Download directory so you can look for the files there. When you find them, install them like so:

sudo dpkg -i libcudnn7_7.3.1.20-1+cuda10.0_amd64.deb
sudo dpkg -i libcudnn7-dev_7.3.1.20-1+cuda10.0_amd64.deb
sudo dpkg -i libcudnn7-doc_7.3.1.20-1+cuda10.0_amd64.deb

We can verify the installation by running some sample code. First, we copy it over to our home directory:

cp -r /usr/src/cudnn_samples_v7/ $HOME

Now we can change into the directory and compile the samples. As before we can speed up the make command by giving it the number of CPU cores we have available (e.g. make -j4).

cd  $HOME/cudnn_samples_v7/mnistCUDNN
make clean && make

We then run the sample (again, note the period at the beginning):

./mnistCUDNN

If all went well, you should see at the bottom of the output:

Test passed!

Setting Up Our Virtual Python Environments

Back in our home directory, let’s setup a virtual environment for our python code to run. This is standard practice since it separates our own python libraries from the system level ones and also helps us stay organized.

There are quite a few ways to do this, so feel free to use whatever method you want or do what I do:

cd ~
sudo apt-get install python3-venv
python3 -m venv venv/dl4cv

If you followed my way, we created our virtual environment in a directory called dl4cv which you could have named whatever you want. This directory itself is located in the venv directory in your home directory.

Now, whenever you want to work inside of your virtual environment, you’ll need to activate it:

source venv/dl4cv/bin/activate

When activated, you’ll notice that the name of your virtual environment appears in parentheses to the left side of your command prompt.

Once inside our virtual python environment, we’ll be using the python package manager pip to install new python packages. First, let’s upgrade it to the latest version:

pip install --upgrade pip

Next, let’s install the python package NumPy which we’ll need to compile and install OpenCV:

pip install numpy

We now need to download both opencv and opencv_contrib to our Downloads directory:

cd ~/Downloads
wget -O opencv.zip https://github.com/opencv/opencv/archive/3.4.3.zip
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/3.4.3.zip

Unzip both of the archives we just downloaded:

unzip opencv.zip
unzip opencv_contrib.zip

Now we change directory into the ~/opencv-3.4.3/ directory, create a new directory called build and configure OpenCV inside of it:

cd ~/Downloads/opencv-3.4.3/
mkdir build && cd build

cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_CUDA=ON -D INSTALL_PYTHON_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=~/Downloads/opencv_contrib-3.4.3/modules -D BUILD_EXAMPLES=ON -DBUILD_opencv_cudacodec=OFF ..

When it’s done make note of the output, focusing on the regions as highlighted in the image below.

[Image unavailable: OpenCV Build Terminal Output]

You’ll want to make sure that both your Python and NumPy versions are the ones from your virtual environment.

If for some reason they aren’t you can always delete the build folder and repeat the above steps, make sure you have the commands correct and that you have activated your virtual python environment before running the command.

As an aside, I had originally tried to compile with CUDA support as well but it ends up not playing well at compile time so that’s why it’s disabled in this example.

Now we are ready to compile. This command again runs faster if you specify more cores, but if you run into errors, try running make clean and then rerunning make without specifying additional cores.

make -j4

Then we install and update our file links before returning to our home directory:

sudo make install
sudo ldconfig
cd ~

To make our OpenCV bindings available to our virtual python environment, we need to symbolically link them, which is kind of like copying but without the copying part:

cd ~/venv/dl4cv/lib/python3.6/site-packages/
ln -s /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so cv2.so
cd ~

Time to test it out! Make sure you have your virtual python environment activated and just type python and hit enter to enter the virtual environment.

From there you can try importing the OpenCV library:

import cv2
cv2.__version__

No errors? Congrats!

We’ll install a few more python packages before adding Tensorflow support:

pip install scipy matplotlib pillow
pip install imutils h5py requests progressbar2
pip install scikit-learn scikit-image

Tensorflow with GPU support can be pip installed for earlier versions of CUDA + cuDNN but not for the latest versions that we’ve installed so we’ll need to build tensorflow from source ourselves.

Let’s start by downloading the source into our Downloads/ folder:

cd ~/Downloads
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow

By default, we’re on the master branch of the source code which is more likely to have bugs and stability issues. Let’s switch to the latest release version:

git checkout r1.11

We can test out the code branch with Bazel but we’ll need to install it. First the dependencies:

sudo apt-get install openjdk-8-jdk

Then we can install Bazel itself:

cd ~/Downloads
wget https://github.com/bazelbuild/bazel/releases/download/0.18.0/bazel_0.18.0-linux-x86_64.deb
dpkg -i bazel_0.18.0-linux-x86_64.deb

…and change back into our tensorflow directory to run the test (expect it to take awhile):

bazel test -c opt -- //tensorflow/... -//tensorflow/compiler/... -//tensorflow/contrib/lite/...

You’ll see plenty of WARNING messages, but we just want to make sure things don’t completely go to shit. For me, my results were:

INFO: Build completed, 23 tests FAILED, 27775 total actions

The documentation offers no clues as to whether 23 errors is good or bad, so I’ll just assume it’s totally fine.

Now we run the configure script which essential surveys the user for which values to use to compile with. The things we care about are telling it to compile with the python executable in our virtual environment, enabling CUDA support, and telling it the correct versions of our tools. We’ll just leave everything else as their default values.

So the only non-default answers we need to give are:

Do you wish to build TensorFlow with CUDA support? [y/N]: Y
Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 9.0]: 10.0
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7.0]: 7.3\
Please specify the NCCL version you want to use. If NCCL 2.2 is not installed, then you can use version 1.3 that can be fetched automatically but it may have worse performance with multiple GPUs. [Default is 2.2]: 1.3

Now we make the package builder:

bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

…build the package:

./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

And install it using pip and change back to our home directory:

pip install /tmp/tensorflow_pkg/tensorflow-1.11.0-cp36-cp36m-linux_x86_64.whl
cd ~

We can test by opening up another python interactive shell and trying to import tensorflow:

import tensorflow
tensorflow.__version__

If you don’t get any errors, you can move onto installing Keras:

pip install keras

If you open up another Python shell and import keras, you should get a message saying:

Using TensorFlow backend.

If you do, you’re done! Go play with some deep learning models. Maybe start here?

How to Politic: A Salt Lake County Democrats Guide to Partial Political Competence

Wed, 04 Apr 2018 01:26:00 GMT

Caucus Night

A friend encouraged me to attend the Democratic Party’s Caucus Night. To be perfectly honest, I still don’t know if that event was organized by the State, County or National Democratic Party, or some combination thereof. Regardless, I consider myself a democrat and I wanted to get more involved in creating change in my local community so I decided to go.

That same friend directed me to vote.utah.gov. There you can enter your street address to figure out your precinct number and the location of your caucus.

Cool, but what the fuck is a caucus anyway?

By definition, a caucus is “a meeting at which local members of a political party register their preference among candidates running for office or select delegates to attend a convention.” At this particular caucus, we were selecting delegates to attend the Salt Lake County Democratic Convention.

Attending the caucus

Unfortunately, since most of the attendees are veteran political activists, there aren’t very clear directions as to where to go or what to do. I wish there had been a short posterboard explaining that I need to check in, fill out the form, and then go wait in the auditorium. Instead, I essentially just walked inside and put on my best “help me, I’m so confused” face until a volunteer approached me and asked if I had registered yet. I explained that I had not, and was directed to a table of computers labeled “ADA Registration.” I still have no idea what ADA stands for.

I approached the nearest woman sitting behind a computer who looked up at me, apparently expecting something.

“Umm, what do you need from me?” I asked, already feeling stupid.

“I just need a government-issued ID that lists your current address,” she offered merrily.

“Okay sure. The address on here is actually old, so I also brought the piece of mail if that works.”

“No problem. Looks like you’re already registered to vote and the address matches so you’re good to go! Just fill this form out, and we’ll soon get started in the auditorium to my left here.”

With the form in hand, I made my way to the nearest table, grabbed a pen and filled it out. The woman had also circled my precinct number on the top of the form since I would apparently need it later. The form requested basic contact info and offered me the ability to express interest in various event caucuses, which are essentially just more groups of people that share or are interested in specific topics.

After bonding with a fellow democrat by nodding as she complained about how long everything was taking, we made our way into the auditorium to listen to people with supposed relevance to our political affiliation.

Speeches and Instructions

Some people whose names and titles I’ve already forgotten explained to the small crowd that we are the power that will turn the state blue. Later, I overheard a woman comment that she found the whole thing “uplifting,” and “inspiring.” Personally, I was hoping for something a little more educational in terms of how exactly this whole thing works. Luckily, they did get to that eventually.

A bit later, the meeting transitioned to a brief explanation of what we would be voting on. Namely, local precinct leaders, some of whom would also act as delegates to vote at the upcoming Salt Lake County Democratic Convention. They apologized for not having a projector available to explain how the forms worked but assured us that we would be able to figure it out on our own and if we had any questions people would be walking around to address them. Each precinct had its own meeting location and we were directed to meet there once the previous precinct Chair or other member of the precinct had gathered the relevant materials from up front.

I decided to be optimistic and assume that the previous Chair of my precinct was in attendance and would gather the paperwork. I made my way to my precincts designated table. Thankfully, other precinct members had indeed shown up and soon joined me at the table. We quickly came to agreement that none of us wanted to serve as any elected official but we would if we had to. And we did have to.

At this point I knew roughly that as a delegate, I would need to attend the aforementioned Democratic County Convention to vote there. With this is mind, I apprehensively volunteered myself to serve as a delegate. Later, my friend explained to me that this means I was also a State delegate. I’m still rather baffled that I could become an elected official for something I didn’t/don’t fully understand.

What now?

My first major responsibility is to attend the County Convention on April 14th. There I know I have to vote, but as of the time of writing, I don’t really know who is running, much less what they stand for. So that’s great.

So far I’ve figured out my districts (House District 28 & Senate District 4) by going to le.utah.gov so that seems like a pretty good first start.

I’ll figure out the rest later. Stay tuned.

Docker-Compose in Production

Fri, 23 Mar 2018 18:04:00 GMT

Production is like development but with more stress and frustration

Just when I had my Docker Cloud deployment all figured out, Docker went ahead and ruined everything by announcing they were dropping support for it in May. So here I am trying to figure out how to use Docker Compose in production. Luckily, I already had a working implementation of Docker Compose for development. Unfortunately, docker didn’t want to make things too easy for me.

I wish it was as easy as provisioning a new host with Docker Machine, targeting the remote host by configuring the DOCKER_HOST environment variable, and then firing off my docker-compose up command. Sadly, that is not the case. While I can get everything running perfectly on my local host with docker-compose up, when I target the remote machine, my containers complain that their run scripts don’t exist:

ERROR: for cts_worker_1  Cannot start service worker: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"./run_celery.sh\": stat ./run_celery.sh: no such file or directory": unknown

ERROR: for cts_web_1  Cannot start service web: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"./run_web.sh\": stat ./run_web.sh: no such file or directory": unknown

ERROR: for web  Cannot start service web: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"./run_web.sh\": stat ./run_web.sh: no such file or directory": unknown

ERROR: for worker  Cannot start service worker: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"./run_celery.sh\": stat ./run_celery.sh: no such file or directory": unknown
ERROR: Encountered errors while bringing up the project.

It seems this is because the run scripts are added as relative volume mounts:

web:
  volumes:
      - .:/code

Since the notation for this mounting is [host_path]:[container_path] and my DOCKER_HOST environment variable is set to the remote machine, docker is actually looking in the /root folder of the remote docker machine I provisioned. I’d prefer if it didn’t work this way, but it does so I’ll have to make do.

One potential solution is to use the relatively new docker-machine mount command to sync my code to the remote machine. I however, couldn’t get this to work in a way that made sense to me. For one, the docs don’t provide much help. They also make it seem like the command is only designed to sync files from the machine to localhost rather than the other way around. Anyway, I tried executing the following in root directory of the source code I wanted synced to my remote machine:

docker-machine mount [machine-name]:[machine-mount-point] .

Fuse, presumably the underlying tool dealing with the mount, complained that the directory wasn’t empty, and invited me to use the “nonempty” option if I was feeling brave. My frustration was close enough to bravery so I tried it, and oddly, the option doesn’t even seem supported by docker machine. Out of curiosity, I also created a new empty folder and mounted that, which worked, but that doesn’t actually solve my issue. Maybe I could copy my code into that new folder, but then I’d have a duplicate local copy of everything which would be silly.

I considered docker-machine scp but quickly realized how inefficient it was for dealing with code changes.

Then, after reading what seemed like hundreds of GitHub issues and SO questions, I realized that the solution was actually much more simple. I had been mislead from the very beginning by following the Docker docs for setting up a django project. I didn’t have any need to mount the code as a volume at all since my Dockerfile copied the code into the container already:

FROM python:3

ENV PYTHONUNBUFFERED 1

RUN apt-get update && apt-get install -y \
    binutils \
    libproj-dev \
    gdal-bin \
    libgeoip1 \
    python-gdal

RUN mkdir /code
WORKDIR /code

COPY requirements.txt /code/

RUN pip install -r requirements.txt

COPY . /code/

After deleting that relative bind mount volume, everything worked hunky dory.

Switching To Containerized Deployment

Fri, 02 Mar 2018 21:12:00 GMT

Getting Started

Like so many of my projects, I’m coming back to this one after several months of not touching it so I need to reorient myself a bit and then continue stumbling forward until I end up with something that works. The end goal is to be able to deploy a series of docker containers that make up a web application using Docker Cloud.

So far I have three docker files; one for Postgres, another that expands on Postgres to install PostGIS, and the third that incorporates these and all the other required services for my Django web application. I’ve also already created a private repository on Docker Cloud.

Step one is to deploy a node in Docker Cloud.

Deploy a Node

The needs of my web application are light at the moment so I’ve disabled swarm mode in Docker Cloud. For the sake of cost, I’ll be sticking with a single server instance.

I’ll be launching on Digital Ocean which I’ve linked to my Docker Cloud account. Digital Ocean recently increased the sizes of their droplets and those changes don’t yet seem to be reflected in the node creation options on Docker Cloud so I’m uncertain what size droplet I’ll actually end up with. I’m aiming for the $10/month size which should be 2 GB of RAM 50 GB of disk space and a single vCPU. On Docker Cloud I’m choosing what used to be the $10/mo option, listed as 1 GB of RAM, 30 GB of disk space, and a single vCPU. Hopefully when I create this, it will either create the upgraded version of the droplet or allow me resize it afterwards.

[Image unavailable: Docker Cloud Node Creation Form]

As it turns out, the created droplet does reflect the options in Docker Cloud rather than the recent upgrade to droplet sizes on Digital Ocean. This is bad, because the node created by Docker Cloud still costs $10/month but with 1 less GB of RAM and 20 GB less disk space. So the first thing I’m going to do is power off the newly created droplet and resize it through the Digital Ocean web interface.

Powering off the droplet involves it’s own problem solving of course since the Docker Cloud automated deployment doesn’t copy over any ssh keys you may have added to your Digital Ocean account. We have to do that ourselves and the process isn’t documented very clearly.

Connecting to the Node

I could just hard reset the droplet thru the Digital Ocean web interface but I’ll want to ssh to the node at some point so I may as well tackle this problem first. The Docker Cloud docs don’t go into much detail on this topic, but you apparently need to create an application stack that will copy over your ssh key to all of your nodes.

It should look like this:

authorizedkeys:
  autodestroy: always
  deployment_strategy: every_node
  environment:
    - AUTHORIZED_KEYS=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDY....
  image: 'dockercloud/authorizedkeys:latest'
  volumes:
    - '/root:/user'

You can get the value for AUTHORIZED_KEYS by running the more command on whichever public key you want to use as your key pair to connect to the containers. Make sure you include all of the gobbilty-gook except for the last part that includes your email or .local address.

Interestingly, this granted me access to the node droplet, but after shutting it down, using the command shutdown now, I can no longer connect. Digital Ocean reports that the droplet is still running so I’m not sure if the droplet never shutdown, or if Docker Cloud immediately restarted it after it went down. Attempting to restart the authorized keys stack file has no effect.

So I’ll just go ahead and attempt a hard reset thru Digital Ocean to see what happens…

…and it worked! I was able to resize the droplet to get an extra gig of RAM and 20 gigs of disk space so that’s pretty swell. After turning the droplet back on, I can also ssh into it as the root user so my public key is still there and functioning as it should.

Deploying the Web App

Now that we have our node configured properly, we need to add services to it. I’ll be using images from a private repository on Docker Hub. The repository is already setup, but is currently empty, so I’ll need to add my images to it first.

First to login on the command line with my Docker Hub credentials:

docker login --username=dockerhubusername

I’m prompted for my password, after which I’m told I’ve successfully logged in. Now, I tag the image I want to push to the repository:

docker tag cc0988358705 dockerhubusername/repository-name:django

And push the to the repository:

docker push dockerhubusername/repository-name

Since I already have a docker-compose.yml file which I can been using for testing, I can copy that file and adapt for use as my docker-cloud.yml file. The first change will be to point the image keys in the file to the remote repository I pushed my images to. For my app, I need two from my private repository (django and PostGIS) and two from their own repositories (Redis and RabbitMQ).

I can remove the reference to the volume containing my code since that will already exist in the image the service is pulling from. I also need to remove the build and depends_on config values since those aren’t supported.

One thing I don’t like is that I have to hardcode my passwords into the Docker Cloud config file. Unfortunately, I couldn’t figure out a reliable workaround since the most obvious solutions were made much harder by the fact that unlike in Docker Compose config files, specifying an env_file also is not supported by Docker Cloud.

Other than that, the config file stays the same compared to the Compose config.

Product Review: FitBit Blaze

Thu, 18 Jan 2018 19:03:00 GMT

Are Smartwatches Even Worth Buying?

My experience with smartwatches began with the original kickstarter funded Pebble which I received in March of 2013. Since then, I have considered smartwatches to be a useful accessory that I prefer to have with me as much as possible.

Here are a few reasons why:

Massively useful while driving

While driving, I can monitor incoming texts, emails, and calls with a simple glance of my wrist. I essentially don’t even have to take my eyes of the road since my wrist it just below my windshield.

Greatly relieves the burden of two-factor authentication

I enable two-factor auth on every account that offers it. This can be annoying, and slow down normal web usage, but that cost is significantly reduced getting an auth code is as simple as glancing at my wrist.

Saves time and minimizes distractions

Again, things that would normally involve me pulling out my phone to investigate notifications are reduced to quick glances. I can quickly ignore spammy calls, emails, or other messages that would have otherwise required me to pull my phone out. And while I’m happy to concede that this time-savings it minimal, I’m definitely the sort of person who ends up distracted by something unimportant whenever I find myself thumbing listlessly through my phones apps. By minimizing the number of times I feel tempted to even pull the phone out, I’m sure I’m saving myself from droves of silly social media posts in the long run.

FitBit Blaze Features

Those are the main benefits of smartwatches in general. So where does the FitBit Blaze fit in? Well it certainly has more features, but largely I don’t find them terribly appealing. I’ll go through them now:

Heart Rate Monitor

This feature is cool. At first. Ultimately though, the data produced by this sensor isn’t useful to me 99% of the time. The scenario it is potentially useful is when I’m biking and I can use my heart rate as a metric to see how hard my body is working. But then again, can’t I just, you know, feel how much effort I’m exerting?

When selecting future smartwatch products to buy, I’ll consider a heart rate monitor to be an unnecessary feature that I don’t want to pay for.

On-screen workouts

This feature is laughably bad which is why it is particularly tragic how much focus FitBit has placed on it. Looking at your watch for guidance on your workout is not convenient. It’s really that simple. At least for me.

Mayyybe the on-screen workouts would make more sense to you if you’re the sort of person whose workout clothes and/or activities prohibit you from bringing your phone with you. Otherwise, workout guidance is just better suited to being a phone application. Putting it on my watch just isn’t useful or convenient for me.

Multi-sport tracking and connected GPS

The one workout feature of the FitBit that I do like is for cycling. Much like while driving, it’s much easier for me to check my watch for notifications compared to my phone. And I also get the added benefit of being able to see things like my elapsed time, speed, distance, and pace. This is an area that a smartwatch can really shine and I wish the Blaze worked harder to make it great.

The thing is, I actually greatly prefer my old Pebble watch when it came to tracking cycling workouts. It integrated with third-party apps, and best of all, it was painless to check during my rides.

The Blaze trips over itself with its flashy color touchscreen and animations. Because of these, the screen can’t stay on all the time or the battery life would be abysmal. Instead, you’re forced to use things like gesture controls or to activate the screen using a button. The former is often awkward and error prone. I often find myself raising my wrist in a silly exaggerated movement just to try and wake my watch up so I can check the display. Clicking a button is the more reliable way to wake the screen up, but greatly diminishes the benefit of having a smartwatch in the first place.

Checking the display should be quick, easy, and intuitive. For the FitBit Blaze, this is not the case.

Guided breathing sessions

A useless gimmicky feature I don’t even care to go into detail about. Again, if you’re into this sort of thing, why not just use a phone application? There is no benefit to having this on your watch.

Conclusion & Summary

For its cost, the FitBit Blaze does not even come close to being worth the money. In my opinion, the design team has gone off the rails by focusing on features that are much better suited to your smartphone and stand little to no benefit by being ported onto a watch interface. The greatest benefits of a smartwatch are sub-satisfactorily included on the Blaze and don’t seem to be the focus of software updates over the 14+ months I have owned and used the product.

After FitBit acquired Pebble, I had hoped to see the FitBit Blaze absorb many of the features that made the Pebble great in my eyes. With the Pebble, I could create predefined text messages so I could send quick simple responses with the click of a button. I still can’t do that with my Blaze.

With my Pebble, I could quickly and easily skip through songs playing on my phone, begin playing music through services like Pandora, and change volume. On my Blaze, the music menu is awkward to get to, often takes longer to connect than if I had just pulled out my phone, can’t control volume, and can’t launch other music services on my phone.

On my Pebble, I could create my own applications and download other custom applications made by others in the developer community. For the Blaze, there is no app store so you can’t extend the functionality of your smartwatch. This is honestly the feature that I miss most about Pebble and probably the main feature I would look for in any future purchase.

As an aside, I’d like to add that my first Blaze actually got stuck in a reboot loop after 11 months of gentle use. I can say the tech support for this problem was good and I received a replacement free of charge since it was still under warranty. That said, I was almost disappointed to receive a new one because I had been looking forward to buying a different product.

I remain unimpressed with FitBit and their ability to produce quality smartwatches. In the future, I will be putting my money on a different company.

Building a Server Monitor: Pyramid Setup

Sat, 13 Jan 2018 21:53:00 GMT

Building a Server Monitor

A few days ago I realized how nice it would be to have a web interface to manage my various servers. So today I’m going to start building one. I’ll also be using this as an excuse to try out the Python web framework Pyramid for the first time. As usual, this post will be less of a guide, and more of a record of my fumbling through the void that is the internet, trying to make things do what I want.

First off, we’ll create the pyramid project. I’m using PyCharm as my IDE, it actually handles a lot of the initial project setup as far as my environment goes.

According to the Pyramid docs, its helpful to start with a cookiecutter version of a project using a package of the same name, so I guess I’ll obey. I happen to be on my Windows box while starting this project, so I’ll be installing it using conda:

conda config --add channels conda-forge
conda install cookiecutter

Now we have a few options for which cookiecutter to use. The pyramid-cookiecutter-alchemy project template sounds good to me since it uses persistent storage controlled by SQLAlchemy which I like a lot.

cookiecutter gh:Pylons/pyramid-cookiecutter-alchemy

This command then asks what you want to call the project and repo. Annoyingly, I couldn’t call both things server_monitor because of directory conflicts with either my virtual environment, or my PyCharm project folder. Whatever it was, I just had to settle calling my repo serv_mon instead.

Oh…

I now realize that I was essentially reimplementing the steps that PyCharm already took for me. So I just started a Pyramid project inside of a Pyramid project. Hurrumph.

I’m just going to wipe the premade PyCharm files so I can follow along with the Pyramid docs. The silver lining is that now I can just repeat the steps above and actually name my project and repo server_monitor!

Now I need to install the project into my virtual environment. For some reason I get some crazy nonsense when I try to install using PyCharm’s terminal:

Downloading SQLAlchemy-1.2.0.tar.gz (5.5MB)
    33% |██████████▊                     | 1.8MB 2.3MB/s eta 0:00:02Exception:
Traceback (most recent call last):
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\req\req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\download.py", line 821, in unpack_url
    hashes=hashes
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\download.py", line 659, in unpack_http_url
    hashes)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\download.py", line 882, in _download_http_url
    _download_url(resp, link, content_file, hashes)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\download.py", line 603, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\utils\hashes.py", line 46, in check_against_chunks
    for chunk in chunks:
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\download.py", line 571, in written_chunks
    for chunk in chunks:
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\utils\ui.py", line 141, in iter
    self.next(n)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\_vendor\progress\__init__.py", line 73, in next
    self.update()
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\_vendor\progress\bar.py", line 79, in update
    self.writeln(line)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\_vendor\progress\helpers.py", line 68, in writeln
    print(line, end='', file=self.file)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 141, in write
    self.write_and_convert(text)
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 169, in write_and_convert
    self.write_plain_text(text, cursor, len(text))
  File "C:\Users\Smith\Anaconda2\envs\server_monitor\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 175, in write_plain_text
    self.wrapped.flush()
OSError: raw write() returned invalid length 180 (should have been between 0 and 90)
    33% |██████████▊                     | 1.8MB 2.3MB/s eta 0:00:02

I assume it has something to do with the reference to cursor, but I’m not actually going to bother trying to figure out for sure because the install works just fine in my normal command prompt:

cd project_dir
pip install -e .

Now we install the testing dependencies because we’re good developers who test things:

pip install -e ".[testing]"

And now we can run our tests using:

pytest -q

Which runs the two sample tests that were created by our cookiecutter template.

We can also start our project using pserve and our development configuration:

pserve development.ini

This launches the development web server. But when I navigate over to it in my browser, I’m told I need to do more stuff first:

Pyramid is having a problem using your SQL database.  The problem
might be caused by one of the following things:

1.  You may need to run the "initialize_server_monitor_db" script
    to initialize your database tables.  Check your virtual
    environment's "bin" directory for this script and try to run it.

2.  Your database server may not be running.  Check that the
    database server referred to by the "sqlalchemy.url" setting in
    your "development.ini" file is running.

After you fix the problem, please restart the Pyramid application to
try it again.

On Windows, this is a little annoying because the database initialization executable is in the directory of my virtual environment but I need to pass it the development.ini file as an argument:

C:\Users\Smith\Anaconda2\envs\server_monitor\Scripts\initialize_server_monitor_db.exe development.ini

Cool, now we can actually start our server.

[Image unavailable: Pyramid Project Initial Landing Page]

Next, I’ll actually write some code myself.

Protecting Sensitive Information in Docker Compose

Wed, 03 Jan 2018 18:06:00 GMT

Hide Yo Passwords!

If you have multiple containers in a swarm service, this won’t be the path for you. If you do have a docker swarm service, check out docker secrets.

I however, don’t have a swarm, and I don’t see any benefit in converting my setup into a swarm of size one, as the docker secret docs offer as a suggestion. So instead, I’ll be using good ol’ environment variables. The problem however, is that I want to share as much of my code publicly as possible but without exposing any sensitive information. To this end, I’ll be using the handy env_file option in my docker-compose file.

You can specify the file location like so:

version: '3'

services:
  # PostGIS database
  db:
    env_file: /opt/db_secrets.env
    image: my_postgis

  # Redis
  redis:
    image: redis:3.2.11
    hostname: redis

  # RabbitMQ
  rabbit:
    hostname: rabbit
    image: rabbitmq:3.7.2
    env_file: /opt/rabbit_secrets.env

  # Django web server
  web:
    build: .
    env_file: /opt/django_secrets.env
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
    depends_on:
      - db

I’ve opted to store my environment variable files in the /opt directory, far from my source control directory so I don’t accidently end up pushing anything I don’t intend to make public.

Make sure to define the environment variables within those files using the following notation, with one credential per line:

username=password
other_username=another_password

Now, the only thing left to do is modify the file permissions for the sake of the docker container:

sudo chown root:docker rabbit_secrets.env
sudo chmod 740 /opt/rabbit_secrets.env

And now you’re ready to roll.

Import SQL file into Docker Compose Container

Fri, 22 Dec 2017 23:30:00 GMT

Get in there!

It took me a while to figure out how to properly import a .sql dump file into a docker container I had started with docker-compose. The recommended way was to import the data on creation of the container by adding the dump file to the /docker-entrypoint-initdb.d directory. I wanted to avoid this if possible because the dump file is a backup of a production database but the docker-compose file is shared in a public repository. I just didn’t like the idea of referencing the database file in a public repository; I’ve made the mistake of pushing sensitive info public repositories before and it is such a pain to clean up those mistakes afterwards. It seemed safer and more logical to do the import after the container had been initialized, especially since the data is purely for testing, and isn’t essential to the function of the application.

Unfortunately, this wasn’t as easy as it seemed like it should be.

Initially I tried something like this:

docker-compose exec -u postgres service_name psql db_name < /path/to/dump/backup_file.sql

But I would run into the following error:

gTraceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 124, in perform_command
  File "compose/cli/main.py", line 467, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
Failed to execute script docker-compose

This error is well discussed, and provides several solutions depending on what you’re trying to do. Several users seem only to need to pass the -T parameter to their docker-compose exec command. For me, the command seems to hang. By checking the logs, I can see that it does actually try to execute, but fails with many errors complaining how relations don’t exist and whatnot.

The key to salvation is somewhat buried the comments of the issue discussion. It relies on docker exec, rather than docker-compose exec and keeps the stdin open with the -i flag. Or something like that. The important thing is that it worked.

docker exec -u postgres -i $(docker-compose ps -q service_name) psql db_name < /path/to/dump/backup_file.sql

Machine Learning Practice: Part II

Mon, 27 Nov 2017 01:57:00 GMT

Building a Dataset

We were introduced to a few basic concepts in part one of this series, but going forward it will probably be easier to start experimenting with data that is actually relevant to the problem that we want to solve. As refresher, that problem is scan news articles and determine if they are relevant to natural resource conservation.

To start simple, we’ll grab two articles; one about some sort of ecological conservation topic, and another that has no relation to the topic. To do this, we’ll use the newpaper library (more specifically the newpaper3k library since I’m using Python 3):

from newspaper import Article

# Create the article objects
conservation_article = Article('http://www.sltrib.com/news/environment/2017/11/24/restoring-utahs-damaged-landscapes-ephraim-center-provides-vital-seed-stocks-to-heal-public-lands/')
regular_article = Article('http://www.sltrib.com/news/nation-world/2017/11/26/guns-were-black-friday-must-haves-going-by-the-fbis-record-203086-background-check-requests/')

Now that we have chosen our articles, we need to download and parse them before we do much with them:

# Download and parse the articles
conservation_article.download()
regular_article.download()
conservation_article.parse()
regular_article.parse()

At this point we can use some tools built into the newspaper to easily extract information that might be important to us. We’ll start by just taking a look at the summary data:

# Use newspapers built-in natural language processing
conservation_article.nlp()
regular_article.nlp()

# Now we have should have access to some interesting info
print('Conservation article summary:', conservation_article.summary)
print('\nRegular article summary:', regular_article.summary)

While the summaries themselves aren’t perfectly sensible from a human-readable standpoint, they do manage to pull out some key ideas that clearly illustrate the differences between the two articles. In the summary of the conservation article we can also see some words and phrases that will likely be important for classifying conservation articles (e.g. ‘rehabilitation’, ‘reseeding’, ‘habitat restoration’).

At this point it makes sense to start gathering some articles and manually labelling them so we can get a better sense of which words are more important for training a classifier. To do this, we’ll make some small changes and additions to the way we gather articles. For one, we’ll create two list to divide the articles that we gather:

from newspaper import Article

conservation_articles = []
regular_articles = []

# Conservation articles
conservation_article = Article('http://www.sltrib.com/news/environment/2017/11/24/restoring-utahs-damaged-landscapes-ephraim-center-provides-vital-seed-stocks-to-heal-public-lands/')
conservation_articles.append(conservation_article)
con_art1 = Article('https://www.nrcs.usda.gov/wps/portal/nrcs/detail/national/newsroom/features/?cid=nrcseprd1367450')
conservation_articles.append(con_art1)

# Regular articles
regular_article = Article('http://www.sltrib.com/news/nation-world/2017/11/26/guns-were-black-friday-must-haves-going-by-the-fbis-record-203086-background-check-requests/')
regular_articles.append(regular_article)
reg_art1 = Article('http://thehill.com/homenews/administration/362121-democrats-pull-out-of-white-house-meeting-with-trump')
regular_articles.append(reg_art1)

This will make it easy to label the articles by type in the form of tuples:

labeled_articles = [('conservation', article) for article in conservation_articles]
labeled_articles += [('regular', article) for article in regular_articles]
print(labeled_articles)

And now we can neatly organize the download and parsing of all of our articles at once:

for label, article in labeled_articles:
    article.download()
    article.parse()

We’ll want to keep adding to this list of articles, being careful to keep the two lists roughly the same length. While it would be easy to populate our list of regular articles by just grabbing whatever we can find from a random news source, we would end up with a highly imbalanced dataset. Training a classifier on such imbalanced data typically results in poor predictions since the classifier can just predict the majority class to achieve a high accuracy. We want to use a balanced number of labels so the classifier can focus in on the things that are actually informative to the problem; the words in the article.

Machine Learning Practice: Part I

Sat, 25 Nov 2017 23:17:00 GMT

Getting Started

I’ll mostly be following along with the online book Natural Language Processing with Python by Steven Bird, Ewan Klein, and Edward Loper while referring to Machine Learning in Python by Michael Bowles, but I’ll be aiming to apply the lessons to problems that interest me personally. You can find the code associated with this blog series on my GitHub.

The ultimate goal of this machine learning practice is to produce a binary classifier for determining if a news article is related to natural resource conservation. I hope to use this classifier to learn more about current trends in the field of conservation that may also reveal interesting trends in how grant money is allocated across different study areas.

Examining the Problem

Ultimately I’ll be using text processing to determine the subject of the article. One of the challenges involved in text processing is dealing with the large number of attributes. Essentially, every word in an article could be considered it’s own attribute, and you could assign an even greater number of attributes to a machine learning problem if you also decide to include combinations of words and phrases.

Generally speaking, when a machine learning problem has many attributes, you’ll need to feed the algorithm more training data in order to build a reliable model. This is a problem in its own right, since I don’t yet have a training dataset. Essentially, I’ll need to manually find articles I consider to be related to natural resource conservation and label them as such. The words used in these articles can then be extracted so the algorithm can begin to associate them the category of conservation.

Some algorithms are better than others depending on the number of attributes and the size of the dataset you can supply it. Since we already know there will be lots of attributes and I likely won’t have the time to generate a huge list of articles manually labeled as conservation-related, there’s a good chance a penalized linear regression algorithm will be our best choice.

Setting Up

I’ll be using python 3.5.3. For a full list of other requirements, check out the requirements.txt file on the GitHub repo.

First though, I’ll be using the NLTK library in a Jupyter notebook:

pip install nltk
pip install jupyter

Using an Example Dataset

Before trying to create my own dataset of classified news articles, it probably makes more sense to use an existing example dataset so we can get a better idea of how to organize our own.

To get some example data, you can use the examples from the nltk book. In your interpreter, simple import nltk and use the download_gui() function. I’ll only be downloading the “book” collection for now.

[Image unavailable: NLTK Book Examples Download GUI]

These examples will give us a lot to work with but “text7” is probably the most relevant to our ultimate goal since it is an extract of the Wall Street Journal.

Exploring the Data

We can first get an idea of how much information we have by using the len() function on our text data:

from nltk.book import text7

print(len(text7))

We find out that there are 100676 words and punctuation marks contained in the text7 object. Each of these items is known as a token, the majority of which won’t be very useful in determining the genre of the content. We can use some of the features of the nltk library to begin paring down the data into just the useful stuff.

Logically, if we want to extract to subject of a body of text, we can probably eliminate things like punctuation and duplicate words. Such a process will also greatly reduce the amount of data we ultimately have to save and sort through when we create our own dataset. That said, the frequency of word occurrence could provide valuable information about the nature of the text. We can compromise by using something like this to try to extract the more meaningful words:

from nltk import FreqDist

fdist = FreqDist(text7)
# Create a list of hopefully important words
focused_words = sorted(w for w in set(text7) if len(w) > 5 and fdist[w] > 7)
print(focused_words)

Here we create a sorted list of words with more than 5 characters in length and occur in the text more than 7 times. Browsing through the list, we can see that we’ve successfully eliminated the majority of “filler” words in the English language and have been left mostly with words that are more meaningful.

Timelapse video from static images

Thu, 02 Nov 2017 21:36:00 GMT

Getting things in order

For one of my nonprofit’s projects, I used a DSLR camera on a tripod to capture the transformation of a common garden green bean seed from germination to small seedling. If it interests you, you can find the code on GitHub here.

Unfortunately, an earlier iteration of the code I wrote to control the camera did not account for the fact that the script may need to restart at one point or another during the ~3 week period I was recording the timelapse. This created some issues with filenames, specifically images being overwritten by newer photographs or given duplicate names.

This is problematic because the photos need to be sorted in order before they can stitched together into one cohesive video. I had hoped to use a file naming convention that would make this simple, but for the above reasons, I’ll need to resort to parsing the EXIF data to determine the correct chronological order of the photographs.

Sorting files using EXIF metadata

Luckily writing a script to rename the files based on the original creation datetime of the image isn’t too difficult. First we establish where the images are located and where we want them to be saved. For me it was:

# The directory where our images are located
PHOTO_DIRECTORY = '/home/<user>/Documents/Work/CTS/timelapse/Images/'
# The directory we want to save our renamed images
PROCESSED_DIRECTORY = 'processed/'

We’ll to import a few packages for the processing:

from datetime import datetime
import os
import PIL.Image
import PIL.ExifTags

Now we need to iterate through the images in specified in the directory we designated above. We’ll also be sure to exclude any other directories that may be located there:

for file in os.listdir(PHOTO_DIRECTORY):
    # Make sure to skip directories-- we are only interested in image files
    path = os.path.join(PHOTO_DIRECTORY, file)
    if os.path.isdir(path):
        continue
    print('Processing ', file)

From here, it’s just a matter of reading the ‘DateTimeOriginal’ key in the EXIF data and using it as the new filename:

img = PIL.Image.open(PHOTO_DIRECTORY + file)
    exif_date = {
        PIL.ExifTags.TAGS[k]: v for k, v in img._getexif().items() if k in PIL.ExifTags.TAGS
        and PIL.ExifTags.TAGS[k] == 'DateTimeOriginal'
    }
    image_time = datetime.strptime(exif_date['DateTimeOriginal'], '%Y:%m:%d %H:%M:%S')
    # Name the new file using the timestamp from the EXIF data
    new_filename = image_time.strftime('%m-%d-%Y_%H-%M') + '.jpeg'
    img.save(PHOTO_DIRECTORY + PROCESSED_DIRECTORY + new_filename, 'JPEG')
    print('Saveing as {} to {}.'.format(new_filename, PHOTO_DIRECTORY + PROCESSED_DIRECTORY))

And we now have a directory filled with our image files named by the time they were shot which makes them easy to sort in the correct order. This has the added benefit of eliminating any duplicate images that may have existed in the original directory.

Converting images to video

For the video conversion, I followed this handy guide from the Raspberry Pi Foundation. The default settings included in this guide work well for YouTube but if you’d like to tweak them for an alternative YouTube format, you may want to refer to their style guide.

From the directory with your processed images, making the video involves just two commands assuming you have the package mencoder already installed:

ls *.jpg > stills.txt

mencoder -nosound -ovc lavc -lavcopts vcodec=mpeg4:aspect=16/9:vbitrate=8000000 -vf scale=1920:1080 -o timelapse.avi -mf type=jpeg:fps=24 mf://@stills.txt

That’s it! If you want to check out the code in it’s entirety or propose any improvements of your own, you can check out the code on GitHub.

Django + Wagtail Custom Users

Wed, 13 Sep 2017 06:00:00 GMT

The Problem

As I said in my last post, I’m aiming to add the ability for users to log in to the website using django-registration. It should be pretty simple, especially considering I got it working just fine on my nonprofit’s website. Unfortunately, this website is essentially all about creating silly problems for myself so I can learn by fixing them. And thus, I have run into a problem that I did not encounter on my other domain:

AttributeError at /accounts/register/
Manager isn't available; 'auth.User' has been swapped for 'custom_user.CustomUser'

Argh!

I had created a custom wagtail user model pretty much just because Wagtail showed me how to in their documentation. And now Django doesn’t know how to create users like normal.

There are of course a number of ways to solve this. I will be using the one I can figure out first.

The Struggle

Google has led me to a SO answer that makes it seem like an easy fix. We shall see.

Essentially all I have to do is instruct Django to use my model instead of the one provided in django.contrib.auth:

from django.contrib.auth import get_user_model

User = get_user_model()

So now I just have to figure out exactly where to place that code. And… after some more googling I found that the Django docs themselves describe how to address this issue. And this is why I love Django.

from django.contrib.auth.forms import UserCreationForm
from myapp.models import CustomUser

class CustomUserCreationForm(UserCreationForm):

    class Meta(UserCreationForm.Meta):
        model = CustomUser
        fields = UserCreationForm.Meta.fields + ('custom_field',)

But since we’re using django-registration, we should defer to their docs first. The process looks similar to the official docs:

from registration.forms import RegistrationForm

from mycustomuserapp.models import MyCustomUser

class MyCustomUserForm(RegistrationForm):
    class Meta:
        model = MyCustomUser

After making the appropriate changes in my custom user app’s forms.py file, I get the following error:

django.core.exceptions.ImproperlyConfigured: Creating a ModelForm without either the 'fields'   attribute or the 'exclude' attribute is prohibited;

It instructs me to edit my custom user creation form. For now I add the most basic implementation by specifying the fields from the parent class:

fields = RegistrationForm.Meta.fields

Now when attempting to create a new user using the registration form I get a timeout error with the following (truncated) traceback:

Traceback:

File "/home/<user>/venv/averyuslaner.com/lib/python3.5/site-packages/registration/backends/hmac/views.py" in create_inactive_user
  55.         self.send_activation_email(new_user)

File "/home/<user>/venv/averyuslaner.com/lib/python3.5/site-packages/registration/backends/hmac/views.py" in send_activation_email
  98.         user.email_user(subject, message, settings.DEFAULT_FROM_EMAIL)

File "/home/<user>/venv/averyuslaner.com/lib/python3.5/site-packages/django/contrib/auth/models.py" in email_user
  366.         send_mail(subject, message, from_email, [self.email], **kwargs)

File "/home/<user>/venv/averyuslaner.com/lib/python3.5/site-packages/django/core/mail/__init__.py" in send_mail
  62.     return mail.send()

File "/home/<user>/venv/averyuslaner.com/lib/python3.5/site-packages/django/core/mail/message.py" in send
  348.         return self.get_connection(fail_silently).send_messages([self])

Exception Type: TimeoutError at /accounts/register/
Exception Value: [Errno 110] Connection timed out

So now it’s just a matter of configuring the connection back to the mail server which also happens to be the same server running this website. I just needed to remove the Digital Ocean firewall since I already had UFW configured to handle all the appropriate firewall rules.

Since the error timed out while trying to send the activation email, the user was actually created and saved to the database. I’m just going to delete that user and try again from the beginning.

I’m also going to start matching up the django-registration form to match my custom user model fields. So I need to add some name fields which is as simple as editing the fields attribute in my CustomUserForm:

fields = ['first_name', 'last_name'] + RegistrationForm.Meta.fields

Sweet, and now we successfully get an activation email sent to the email address supplied when submitting the registration form.

But we also get a new error when trying to access the activation link sent to our new user:

django.urls.exceptions.NoReverseMatch: Reverse for 'login' not found. 'login' is not a valid view function or pattern name.
[14/Sep/2017 12:04:24] "GET /accounts/activate/complete/ HTTP/1.1" 500 158304

So I just need to throw in the default urls from django.contrib.auth:

url(r'', include('django.contrib.auth.urls'))

At this point everything seems to work properly but when we login, we get a 404 error because the default redirect page for a django login is /accounts/profile which doesn’t point anywhere yet. Were going to specify where that redirection should occur and create a template for it:

# Django auth settings added to base settings file
LOGIN_REDIRECT_URL = 'edit_profile'

The template is pretty basic:

{% extends "base.html" %}
{% load home_extras %}

{% block title %}Edit your profile{% endblock %}

{% block heading_text %}Edit Your Profile{% endblock %}

{% block content %}
    <div>
        <p>
            Use the form below to edit your profile.
        </p>
        <p>
            Use whatever name you'd like to be identified with while on this website. If
            you leave it blank, we'll identify you as <b>{{ user.username }}</b>, your username.
        </p>

        <form method="POST" action="." class="form-horizontal">
            {% csrf_token %}
            {% for error in form.non_field_errors %}
                <div class="form-group has-error text-danger">
                    {{ error }}
                </div>
            {% endfor %}
            {% for field in form %}
                <div class="form-group has-error text-danger">
                    {{ field.errors }}
                </div>
                <div class="form-group">
                    {{ field.label_tag }}
                    {{ field|add_css:"form-control" }}
                </div>
            {% endfor %}
            <button class="btn btn-primary form-group">Save</button>
        </form>
    </div>
{% endblock %}

But we still need to create the view to serve it up:

@login_required
def edit_profile(request):
    user = CustomUser.objects.get(username=request.user.username)
    form = ProfileForm(request.POST or None, instance=user)
    if form.is_valid():
        form.save()
        return redirect('user_profile', request.user.username)
    return render(request, "accounts/edit_profile.html", {'form': form})

And create the form referenced in the above view:

class ProfileForm(forms.ModelForm):
    first_name = forms.CharField(
        required=True,
        widget=forms.TextInput(attrs={'placeholder': 'First Name'})
    )
    last_name = forms.CharField(
        required=False,
        widget=forms.TextInput(attrs={'placeholder': 'Last Name'})
    )
    email = forms.EmailField(
        required=True,
        widget=forms.TextInput(attrs={'placeholder': 'Email'})
    )

    class Meta(object):
        model = CustomUser
        fields = ['first_name', 'last_name', 'email']

    def __init__(self, *args, **kwargs):
        # Instance is a CustomUser object
        instance = kwargs.get('instance', None)
        if instance:
            kwargs.setdefault('initial', {}).update({'first_name': instance.first_name,
                                                     'last_name': instance.last_name,
                                                     'email': instance.email})
        super().__init__(*args, **kwargs)

    def save(self, commit=True):
        # Instance is a CustomUser object
        instance = super().save(commit=commit)
        instance.first_name = self.cleaned_data['first_name']
        instance.last_name = self.cleaned_data['last_name']
        instance.email = self.cleaned_data['email']
        if commit:
            instance.save()
        return instance

Upon saving the edit form, our view redirects to the normal profile view which we also need to create under templates/accounts/user_profile.html:

{% extends "base.html" %}
{% load humanize %}

{% block title %}{% firstof user_obj.profile.name user_obj.username %}{% endblock %}

{% block heading_text %}User Profile{% endblock %}

{% block content %}
    <div>
        <h3>
            {% firstof user_obj.profile.name user_obj.username %}
        </h3>

        <p>
            Hey internet person.
        </p>

        <p>
            You can edit your profile <a href="{% url 'edit_profile' %}">here</a>. If you want. No pressure.
        </p>
    </div>
{% endblock content %}

Cool, now we have everything working properly so we just need to make some improvements to the nav page. We need to hook up the login button to the actual login function:

{% if user.is_authenticated %}
    <ul class="nav navbar-nav navbar-right">
        <li class="dropdown">
            <a href="#" class="dropdown-toggle" data-toggle="dropdown">My Account <b class="caret"></b></a>
                <ul class="dropdown-menu">
                    <li><a href="{% url 'user_profile' user %}">Profile</a></li>
                    <li><a href="{% url 'logout' %}">Logout</a></li>
                </ul>
        </li>
    </ul>
{% else %}
    <a class="navbar-text navbar-right" href="{% url 'registration_register' %}">Register</a>
    <form method="POST"
                    action="{% url 'login' %}?next={% if request.path == '/logout/' %}/{% else %}{{request.path}}{% endif %}"
                    class="navbar-form navbar-right">
        {% csrf_token %}
        <div class="form-group">
            <input type="text" name="username" placeholder="Username" id="id_username" class="form-control">
        </div>
        <div class="form-group">
            <input type="password" name="password" placeholder="Password" id="id_password"
                class="form-control">
        </div>
        <button type="submit" class="btn btn-success">Sign in</button>
    </form>
{% endif %}

And voilà! We now have a working registration process!

Website updates

Wed, 13 Sep 2017 00:00:00 GMT

And we’re back!

I’m finally coming back to my personal website so I can make it less bad. And more good.

I’ve spent the better part of two days updating all of the packages on the VPS as well as those in the python development environment. It’s kind of just fun to update things. It makes me feel better.

With style

I refactored a bunch of the project code to better handle things like file permissions and environment variables. You can check it out on the GitHub page if you’re curious. And of course in the process of doing so I briefly exposed ALL of this project’s sensitive information to the entire internet.

For about 2 minutes.

I actually checked the logs to see if anyone had managed to discover the directory traversal vulnerability I had unwittingly creating while learning the difference between “alias”, “root”, and “autoindex” in my nginx config files. Don’t worry, no one had. Looks like my website won’t be taking the place of Equifax in the headlines just yet.

Anyway, as you might have noticed, I also made some stylistic changes to the front page already. Functionally, they are sterile. The little login form in the top nav bar doesn’t actually work at the time of writing, but I pushed it anyway because YOLO.

Registration

I’ll be working on adding registration functionality to the website next. Yes that’s right, I’m announcing that I plan to start collecting sensitive user information in the same blog post in which I casually admitted to exposing the credentials of the database to the internet. Don’t worry though, I have it ALL under control!

MySQL: From remote to local

Sun, 15 Jan 2017 00:00:00 GMT

Ok so now that we have our secure backup location, it’s time to actually grab everything from the database. The first command will allow us to securely communicate with the remote server over SSH. If we didn’t do this, all of the website traffic would be vulnerable to interception since it would be unencrypted in transit (with the exception of the user passwords which are stored encrypted already).

ssh -f -L3310:localhost:3306 <redacted> -N

The above command will forward traffic directed at local port 3310 to the remote machine’s port 3306 which is the default port for a mysql server. To make sure this works, you’ll want to make sure the port is actually open for communication on port 3306, preferably restricted to just the IP address of your local machine for better security. If you’re using uncomplicated firewall, you can achieve this with the following command (replacing the example address with your own IP address):

sudo ufw allow from 198.51.100.11 to any port 3306

And now we’re ready to actually dump the database. Fill in your own credentials where appropriate:

mysqldump -P 3310 -h 127.0.0.1 -u mysql_user -p database_name table_name > backup_file.sql

You can leave off the name of the table if you want to dump the entire database. This will generate all the SQL statements needed to recreate the database and write them out to the file backup_file.sql.

Import the SQL Dump

Now on my local machine, I want to import the dump into a new database I’ve created:

create database website_bak;

Then I give my non-root user permissions to write to all of the tables in the new database:

grant all privileges on website_bak . * to 'user'@'localhost';

Now we can quit back out and run the command to import the dump file into the database:

mysql -u user -p website_bak < /path/to/backup_file.sql

Encrypting the backups

Sat, 14 Jan 2017 00:00:00 GMT

As of the time of writing, the website still looks crumby. But there has been progress! You just can’t enjoy it yet.

Considering I implemented the strictest level of encryption on my website before I even built a way for users to submit data, I obviously care about security. In fact, I probably approach security in way that could be construed as paranoia, but that’s just because I think such an approach offers the greatest about of fun.

Anyway, my latest addition to the website has been a custom user model which will eventually facilitate the ability for users to order prints of my photographs if they so choose. Unfortunately, this created some dependency issues when I attempted to apply the Django migrations to all the tables that had been built using the default wagtail user model. So rather than mess around editing the migration files for hours and probably just ruining things further, it probably makes more sense to just drop the database entirely and rebuild it from scratch using the custom user model. But wait! Then I’d lose all three of these masterfully crafted blog posts. Time for backups.

My fancy new workstation has a separate 1 TB HDD that would be ideal for such a backup. But all that fancy-shmancy web encryption would be wasted if I backed up all the data to an unencrypted drive. Which brings us to the exciting world of:

Full Disk Encryption!

Step one of course is to fire up gparted and delete the existing partitions. Then we create a new partition which uses up the whole drive, but we leave the filesystem unformatted. We’ll do that ourselves by following a guide such as this. We also want to zero out the drive before the encryption part so you’ll probably want to confirm the location of the disk and partition with something like:

fdisk -l

And throw something like this at it, where you replace sda1 with whichever disk partition you happen to be targeting:

sudo dd if=/dev/zero of=/dev/sda1 bs=1M

You’ll want to be extra careful when using a command like dd since a typo there could result in irreversible loss of data if you target the wrong disk or partition. So at this point if you haven’t ruined your computer by mistyping the above command and you’re zeroing out a large disk like my 1 TB HDD, you can go out and play with your friends because it will take about forever for the command to complete. Just be patient and don’t worry about how it seems to hang, dd just doesn’t output any info to describe its progress.

If you’ve been patient enough to wait for forever, you’re now ready to actually start doing the encryption stuff, like so:

sudo cryptsetup luksFormat /dev/sda1

You’ll want to make sure you’re using latest version of cryptsetup before running this command, since we’re using all the defaults here which will offer plenty of security as long as your using version 1.6.0+. If you’re interested, you can read a bit more about the tool here. Once you run the command, you’ll be prompted to supply a password that will be used to unlock the disk. The encryption won’t be worth much if you use a weak password, but you’ll also need to make sure you don’t lose it as there will be no way to recover the data on the disk if you misplace the password. As such, this might be a good time to start using a password manager like Keeper which makes it easy to create and use complicated passwords without having to worry about forgetting them later.

Now we need to mount and unlock the drive:

sudo cryptsetup luksOpen /dev/sda1 encrypted-hdd

This lets us give the drive a filesystem:

sudo mke2fs /dev/mapper/encrypted-hdd

And finally we want to transfer ownership of the drive to our unprivileged user so that user can actually write to the disk. It will look something like this, where the username in my case is myuser:

sudo chown -R myuser /media/myuser/ae722e82-1476-4374-acd8-a8a18ef8ae7a

You can check where your disk is mounted using:

df -h

Setting up a new work environment

Fri, 13 Jan 2017 00:00:00 GMT

As with most computer related things it seems, working on one problem always seems to expose a variety of new problems, some of which are blocking while others are just interesting optimizations. In the age of Squarespace and other such tools that allow one to erect aesthetically pleasing websites in a matter of minutes, part of the fun of this project is to tackle as many of the challenges I encounter head on. As such, I’ve spent the better part of my week working more on my new workstation than I have on the website itself.

It turns out that buying new hardware doesn’t necessarily mean that everything will run flawlessly when you stubbornly insist your development environment must be run under linux. After about four separate clean install attempts of various Debian configurations, I finally gave up and turned to Ubuntu 16.04 LTS instead. For the past few years I have preferred to use Debian because I considered Ubuntu to be linux with training wheels. And perhaps it is, but I have to say it sure was nice to finish the install and not have to wrestle with the innards of my computer to try and figure out why my touchpad, wifi-adapter, and xserver all weren’t running properly despite having seemingly downloaded the latest drivers from the stable-branch repository. Under Ubuntu, everything has pretty much just worked without my intervention (save for my GTX 960M), so now I can actually focus on the website itself.

Building the website

Mon, 02 Jan 2017 00:00:00 GMT

This website will serve to showcase my various projects and interests.

Various topics and categories will be explored:

Programming
Photography
Miscellaneous projects

For the foreseeable future, I will mainly be writing about programming in Python and various web-focused languages while I build out this website. The site is hosted by DigitalOcean* on an Ubuntu 16.04 x64 instance and powered by Django and Wagtail. It is served by the nginx web server. The source code can be found on GitHub here.

If you’re reading this close to the date of publication, you can clearly see that my focus thus far has been on function rather than form. I have spent the majority of my time thus far laying the basic framework for the rest of the website. Behind the scenes, I have used LetsEncrypt to ensure a high level of security if you choose to exchange any information with the website. At the time of writing, this website receives an A+ rating from SSL Labs.

Going forward

Now that the groundwork has been laid, I will be focusing on making the website a little prettier. I’ll also begin building out the photography section of the website since I think this is the area that will largely dictate how things the rest of the pages will look, stylistically speaking.