Installing Salt on my “farm”

•May 19, 2017 • Leave a Comment

Previously, I built myself a VirtualBox farm.

So I’ve wanted to try Ansible for some time, based solely on a vague description of what it does, the fact that it is open source, written in Python, and the lovely Ender series reference. It turns out, though, although it’s clean (you only install software on the master, the “clients” operate solely through ssh), it loses on performance to Salt, which is also open source and written in Python. So I decided to explore Salt. Sorry, Mr. Card.

The Salt tutorial recommended installing their demo system using Vagrant. I did. It was terrible. The machines that Vagrant made while compact, wouldn’t shut down, plus they used an old Ubuntu 14 image (turns out there is probably a reason, but I’ll get to that). Also, I wanted to understand how it works, so I decided to install it on my own machines.

I installed the Ubuntu minimum CD into my machines. I tried the bootstrap script as recommended by the SaltStack folks, but it broke. Turns out the bootstrap has been broken for a while. I wanted to use the latest Ubuntu (I’m probably making things harder for myself, but…). Then I read this that mentioned the apt-get packages. They may be a little out of date, but I ended up with 2016.11.2 instead of 2016.11.4, so perfectly workable. On my queen, I ran

sudo apt-get install salt-master

and on my drones I ran

sudo apt-get install salt-minion

and then updated my drones with the queen’s hostname as per this post.

I also set the service to start automatically

sudo update-rc.d salt-master defaults


sudo update-rc.d salt-minion defaults

I made one drone template and then cloned them. When I did that I had to update the hostname in /etc/hostname, and regenerate the salt keys. So I wrote a simple shell script to do it for me, and put it in the drone template so it got cloned to the rest. Here’s what it looked like

#! /bin/bash

echo $1 > /etc/hostname

service salt-minion stop
rm -v /etc/salt/pki/minion/minion.p*
cat /dev/null > /etc/salt/minion_id

shutdown -r now

Theoretically, I could do this without a restart, but a) changing the hostname immediately hasn’t been working for me, and it requires a reboot, and b) with the minimal Ubuntu, restarts are so fast I haven’t needed to worry about it. Now the hostname is changed, and the key with the new hostname will be regenerated when the service starts on boot.

This process resulted in a number of phantom keys, machines I never expected to see again. I found a way to delete all machine keys for machines not currently responding

salt-run manage.down removekeys=True

and the usual way of deleting keys


-d DELETE, --delete=DELETE
Delete the specified key. Globs are supported.

-D, --delete-all
Delete all keys.

so after I had all the drones up with new hostnames and new keys generated, everything was ready to be accepted


I ran

salt-key --accept-all


and everything looks happy. Time to test things out



This is the end of the first part of the official Salt tutorial.

Changing up Linux flavors

•May 12, 2017 • Leave a Comment

So just a note. I set up my “farm” scheme with FreeBSD. FreeBSD is my go to because it installs itself minimally, and I have experience with it (both historically, and my current NAS4FREE server uses it). But when I went to install Salt, I found that it doesn’t support FreeBSD (then later I figured out it does, but I’m going to keep my new choice). So now I am on the hunt for what I should be installing to make things as easy as possible in the future.

For a second,  I was thinking CentOS, since that’s what a lot of people in the VFX industry use. Jenkins doesn’t support it, though, so that’s out.

On a side note, it looks like Chef is open source (and supports Ubuntu/Debian) and Puppet is not, so I guess I know which one I’m going to learn about.

It’s looking like everything I’m interested in supports Ubuntu/Debian. I was leaning towards Debian, if only for the range of devices it supports. Ansible, however, doesn’t support it (nor does Puppet, though I don’t really care about a non-open source project). This isn’t really a deal breaker since I’m not sure I’m going to ever explore Ansible (despite the Orson Scott Card reference, it loses out in performance to Salt). But I think this is emblematic that most everyone supports Ubuntu right now. I think installing it will give the best chance of being future-proof.

On an additional note, I installed the command-line version of Ubuntu from the minimum install, and now my machines start and stop so much faster! We’ll see if that’s because it’s not loading things that are needed. But for now, I’m very happy

update: turns out it indeed is missing some things, but what I’ve noticed so far is commands aren’t installed, such as ‘ifconfig’ which is pretty basic. But, these things are a quick ‘apt-get’ away, so I’m still happy. I really don’t want dozens of services I don’t use starting up. Just the ones I need and add.

my VirtualBox machine farm

•May 5, 2017 • Leave a Comment

A little while ago I wanted to install Inkscape on OSX, because I wanted a free alternative to Illustrator (which I don’t know all that well, so I would be looking up tutorials, anyway). But the only way to do this is to run middleware, and it all sounded messy. So I thought “I’ll run a free VMware machine, and then when I’m done, just throw it away!” But (with plenty of caveats) you can run VMware machines for free on Windows, but not on Mac. So I re-discovered VirtualBox. It’s open source machine virtualization that can install on OSX. So I created a Redhat machine, installed Inkscape, and have yet to use it. Ah well.

But what I have been missing is access to a render farm. Not for rendering, but from the standpoint of having a whole bunch of machines to experiment with.

In my investigations into proper DevOps, I have learned about things like Salt and Ansible (for remote control), Puppet (for automated machine setup), and Jenkins (for keeping live code alive while updating, aka continuous deployment). Every time I read about one of these things, I’m annoyed I didn’t learn about it before because it would have made my life running a render farm SO much easier! I’ve really been wanting to experiment with these things, but I no longer have a render farm to try them out on.

So, with VirtualBox, and VBoxManage (the command-line interface [CLI] for VirtualBox), I’m going to setup my own “render farm LITE” to experiment with.

First, I made a base machine, and then a second machine that used the first machine’s hard drive. The idea was to create a lot of machines without filling up my storage with virtual hard drives and also to allow changing one machine to affect all of them. Although changes were nicely carried over, only one machine at a time could use the drive. So I decided to make a more traditional design. One machine will contain the network drive, and all the other machines will mount that.

In my experiments, I discovered the most FreeBSD-ish way of downloading a file (headless, of course) is using fetch. But downloading authenticated files (e.g. “https://*”) fails, so I had to use the –no-verify-peer option. I’m sure this exposes me security-wise, but it got the file downloaded.

So now I’m going to create separate machines. This will be a little messy, but honestly, creating multiple machines using a central drive didn’t really typify the real world, since it’s not possible to do that with discreet machines. And that would make things like Jenkins, Salt, Ansible, Chef, and Puppet moot, so now I can get some experience with those.

am going to link the machines. Which means only changes (not the entire OS, for example) will get stored. Combined with storing any large files on the network drive, this should mitigate any VM sprawl. I plan to also have these machines be volatile so that I can delete them at any time and get storage space back.

Lastly, my virtual farm is going to live inside of its own NAT Network, so that inbound connections don’t work, outbound connections (e.g. Google) work, and each host can talk to each other host. Each machine in my “farm” should only need to see the other machines in my “farm.” This will mitigate having to plan for IP leases on my existing network. It will also marginally increase security, but since I didn’t plan to open any ports on my router anyway, not by much.

Here are some notes for reference:

  • (FreeBSD) To change the hostname permanently, edit the first line of /etc/rc.conf. Reboot
  • (Ubuntu) To change the hostname permanently, ‘echo new_name > /etc/hostname’
  • All the farm machines can get started headless. When I figure out what can be used to determine what IPs they got I’ll post it here, but for now “the perfect is the enemy of the good”

My project

•April 28, 2017 • Leave a Comment

Just a quick note that I added more modules to my open source project, at

Some time ago I put some of my code up on Github. The intention was to make it available to me at a new VFX house, or really anyone, especially in the VFX industry. It wasn’t so much to show off my coding skills, as a lot of it is old, not especially elegant, and inefficient (but it’s largely stuff to interact with humans and do things once, so a little inefficiency is never noticed). But it works, which is really the point.

The only thing that has stopped me from dumping everything on there is I’m making a point to test everything well. Most of the code was written before I “discovered” unit testing (like so many great discoveries, it had been right there for a long time). I’ve been holding out on adding my modules until after I have written good unit tests for everything. With this most recent addition, my unit testing even uncovered a couple bugs that I estimate have been there for about 5 years!

The latest addition is really the reason I started the whole thing in the first place. The VFX industry does everything with sequences of files, each one representing a single frame of the movie. Not movie files, like .mov or .mp4, as many people think. Also making things more complex, every piece of software seems to have its own language for describing the sequences, even though they are all essentially describing the same thing. So a long time ago I decided this was a common enough concept I should create a module to handle it. I had it accept input in any of these formats, store it in simple terms, and then I could output to any format I wanted. It seemed like a simple concept, but the original module had over 1000 lines of code! I’ve paired that down now, but with the unit tests, it’s getting back up there.

So this module seemed like a good idea at one point. But what it became was so much more powerful than I imagined. It allowed me to develop scripts and utilities SO much faster than before. With more intuitive interfaces. And the unforeseen benefit: these manipulations worked, 99% of the time. I didn’t have to debug the code every time. And if I did come across a bug, I fixed it once for everything that used it (but that’s not a revelation, that’s basic computer science).

If even one person is helped by this module, I’ll consider it a win.

Git notes

•April 21, 2017 • Leave a Comment

I’m getting myself setup to develop on the SickRage project. I thought I would write down my git commands to get started.

Setup my project

First, inside github, I forked the desired project

Then, using the url of the new project in my profile

git clone /Sickrage

I then linked in my config files, so that I can wipe out the folder safely

ln -s config.ini /Sickrage/config.ini
ln -s sickbeard.db /Sickrage/sickbeard.db

now I add the original repository as a upstream source

git remote add upstream

Updating my fork

now, every time my fork is out of date, I go into the folder and

git checkout master
git fetch upstream
git merge upstream/master

and then to update my fork on GitHub

git push

which should normally be a fast-forward. Any development I am doing should be in its own topic branch. So now that my master is up-to-date, I can update my topic branch

git checkout [topic]
git merge master

and hopefully, there won’t be any conflicts. If this works, update my fork on GitHub again

git push

this won’t work if you aren’t already configured. You can run

git push origin HEAD

to push WHATEVER BRANCH YOU ARE ON or if you want the first command to work every time…

git push -u origin HEAD

which will make the changes permanent.

Deleting branches (but only if it’s one of my branches)

I created a branch to test out this process. Now, I want to delete it. So I go back to master, and…

git checkout master
git branch -d [topic]

and to delete the branch from GitHub

git push origin --delete [topic]

Submitting a Pull Request when I’m done working on my patch (branch)

 I haven’t yet made a Pull Request, so these may change in the future

For this particular example (SickRage) “develop” is the branch that you are expected to branch your work from.

If the develop branch doesn’t exist locally (e.g. this is a fresh clone)

git branch develop
git checkout develop
git pull

we are now on the develop branch. Update to the latest commit, and push this to your fork

git fetch upstream
git merge upstream/develop
git push

then make my topic branch based off that (not master)

git checkout -b [topic] develop

 and push it to GitHub

git push origin HEAD

and to malke that functlionality permanant

git push -u origin HEAD

or to do that explicitly

git push origin [topic]

pull requests for GitHub another article

go to your fork, go to the pull requests tab in the upper left, and initiate a pull request in the upper right. Select the main repo, and the develop branch, then your fork and your topic branch. You will see a diff representing your work on the branch.

I’m not sure how to submit the pull request after that. The only message I am getting is “able to merge” and “No commit comments for this range.” I have a feeling I’m working from an old instruction set for how to contribute to this project. I’ll update this past if I ever figure out what I’m doing wrong.

Your data available to the highest bidder

•April 14, 2017 • Leave a Comment

So I spent some time this morning learning more about the Republicans leaving us to the wolves and allowing your ISP to sell ANY information it can glean from your web activity to anyone that they deem worthy. This article was interesting. It pointed out that the regulations that congress struck down actually weren’t in effect yet. So we’ve always been exposed. Did you know that? I didn’t! More reason we need to be protected (or informed, but I don’t see anyone stepping up to do that anytime soon) The other thing it pointed out that I already knew, was that if you pay for a VPN service to encrypt your traffic, you need to trust the service won’t sell your history just like the ISPs.

What would be really good is to use a trusted VPN service to encrypt all your traffic. Over at Make: they have an article about using a Raspberry Pi to create just that, a VPN router that encrypts all data leaving your house. But its 100% wireless, so I imagine it’s only really useful somewhere you want to secure your wireless browsing, like a coffee house. I suppose you could use the wired connection in lieu of one of the wireless connections. But Netflix doesn’t like you using any of these services, so I guess I need something that can be selective.

For now, I’ve installed Opera. A little while ago Opera added a particular feature: free VPN. You just click it on (on Mac, Opera menu > Preferences > Privacy & Security > VPN > Enable VPN), and everything is encrypted and sent to Opera. Now, this has the same limitations of a VPN service, but I would trust Opera a lot more that some off-the-shelf VPN service. So the idea is I do most of my surfing with Opera, and Netflix with my old browser (Chrome).

My next step is flashing my old router with DD-WRT. I tried doing that a long time ago and it didn’t end so well, so I just stuck with the factory firmware. If I flash the firmware, get a VPN service, and get an aftermarket cable internet modem, I can replace the Comcast modem/router that is probably costing a rental charge, too. Also, things like Netflix wouldn’t like to work with it, but I’ve found at least one example of how to exclude Netflix from the VPN.

In the future, there are routers with open-source firmware already flashed that can run a VPN connection, but they are several hundred dollars. Also, I would imagine some of the mesh wireless solutions will have that kind of capability by the time I’m ready to fork over that kind of money.

So for now, just know you have no rights on the internet your ISP doesn’t want to give you. Between this and the loss of net neutrality, the internet feels like the Wild West more than ever.

Samba – going backward through the OpenVPN rabbit hole

•March 31, 2017 • Leave a Comment

In the past, I setup a Samba share on my Amazon Web Services (AWS) instance and secured it with OpenVPN. Now, I wanted to do some software development on my instance, and to do that, I wanted the instance to reach back through my VPN and access a file share.

I used these articles to figure most of this out.

First, I verified that the instance was able to talk back through the connection. The way I setup my VPN, the instance (server) was, and my machine (client) was So from the instance

$ ping
PING ( 56(84) bytes of data.
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=96.9 ms
64 bytes from icmp_seq=2 ttl=64 time=193 ms
64 bytes from icmp_seq=3 ttl=64 time=216 ms

then I created a shared folder to connect to. I created this on OSX, but any machine connected with my VPN simply needs to have the same share name, and it can be snapped right into place.

I created a folder on my desktop named ‘sickrage’. Then I went to System Preferences, then Sharing. Under file sharing, I clicked ‘+’ to add a shared folder, and then selected the new folder on my desktop. Then under Options selected my user, and entered my password.

then on the instance, I installed cifs

sudo yum install -y cifs-utils

then I created a mount folder

sudo mkdir /mnt/sickrage

then I mounted the share once to test it

sudo mount // /mnt/sickrage/ -t cifs -o uid=ec2-user,\

I then created a dedicated user to own the mount. This user will also run the server during development.

sudo useradd sickrage -d /home/sickrage -M -p sickrage
sudo cp -R /home/ec2-user /home/sickrage
sudo chmod 700 -R /home/sickrage
sudo chown -R sickrage:sickrage /home/sickrage/
sudo chsh -s /bin/bash sickrage

I then added the mount information to the /etc/fstab

//  /mnt/sickrage  cifs  uid=sickrage,gid=sickrage,\
credentials=/root/.smbcredentials,iocharset=utf8,sec=ntlmssp,nounix  \
0  0

and then created a .smbcredentials file to store the login information


and lock it down so that not just anyone can come along and read the file

sudo chmod 600 /root/.smbcredentials

This is still a plain-text file, and as I mentioned before, this isn’t the most secure way of doing things, but then, I didn’t write the system.

Now I have remote storage on my local machine, and hopefully, that will reduce the number of disk cycles my remote instance will go through