Getting more out of GitHub with Pubget’s Chrome Extension

At Pubget we host our code base on GitHub because of its simplicity, reliability, and great web interface. The extent to which we depend on GitHub is far more than as a convenience, but as the backbone of our engineering team.

Consequently, when GitHub doesn’t have all of the features we need to do our work as efficiently as possible, it’s not just an annoyance, but a source of friction for the entire organization. To grease the wheels of our daily workflow, we’ve begun building a suite of “GitHub Extras” in the form of a Chrome Extension (we've also used the API to add tags when we deploy!). Our focus thus far has been on the Issues tool provided by the web interface, upon which we have now added the ability to upload attachments or include a screenshot from any open tab.

What does this mean for our organization? When our product team completes a new feature spec, it’s a simple one-click to upload the Word doc to a new issue for the dev team. While testing internally and a page renders incorrectly on Internet Explorer, it’s a two-second task for the QA team to grab a snapshot of the offending site and create an issue for it.

As we say, “time saved is time spent coding,” and we love to code!

Download the extension here.

Learn more about our dev team workflow here.

Pubget's automated builds

As a small team, we don't have the luxury of dedicated release engineers and thus need to automate our build as much as possible. To do this we use git branches, CruiseControl.rb and some nifty shell scripts to keep new code flowing. Coupled with the automation, we also have a weekly build master rotation where one member of the dev team shields the rest from tracking down build failures.

Ok, a picture is worth a thousand words so here's our code flow (everything green is automated):

Developers will typically workout out of the DEVELOPMENT branch so this is where new code begins flowing through the system. We find that it's good practice to link submissions to GitHub issues for management & deploy tracking. The changes then get pulled into STAGING which is the continuous build branch for automated testing. We like CruiseControl.rb because it's light weight & low maintenance. If tests fail, the build master is notified and he can track down the right person to fix the failing test in staging. Once the build is green, the qualified code gets pushed to MASTER which is the production branch behind pubget.com. Since the master branch is considered the healthiest, it feeds down into development and any other project branches that may exist. At this point every branch now has that latest code change and the process begins again.

Too many words? Here's the short flow:

1. devs submit to development
2. changes are tested in staging
3. qualified changes get pushed to master
4. development & special project branches stay close to shore through master

It's definitely worth mentioning that when test failures happen in staging, the automation pauses. Nothing is pulled from development into staging and nothing is pushed from staging into master. This ensures that only healthy code makes its way into production and only the developer that broke the build needs to fix staging. Other developers can continue merging changes into development knowing they will be tested once staging is working again.

This process has helped in a bunch of ways:

• developers submit code and trust that it will be qualified, deployed & propagated to all branches
• developers stay productive even when the build is broken
• we have a production branch where we can quickly apply a hotfix if needed

How to notify github issues when you deploy

A common problem in any production application is that there is a natural delay between submitting a fix and the fix being deployed. Even with an awesome continuous build process that uses fast parallel testing, it can be hours or a day before the production site sees the fix.

This means the customer service folks either have to continously check the production site or the deployment logs and then know how to test or match the fix numbers with the deployments. This is a hard problem that either solved with labor or customers get notified something is fixed before they can test the fix.

At Pubget, we use github and rake for deployments (to passenger apache servers). So, we decided to solve this problem with code. Essentially, when a fix or number of fixes are deployed, we use the github API to comment on each issue when it is deployed. This allows the customer service folks who are participating in the issues to get automatically notified when their fixes make their way through the build process.


To do this, we call this from the deployment rake task:

This looks for fixes that have made its way to the master branch, through our build process and that have our deploy tag and update them with a comment when the fix was deployed.

Now the customer service folks know exactly when their fixes are on the production site and don't need to monitor or ask developers when they can connect with the customer to give them the good news that their bugs have been fixed.

How many coffees does it take to make Pubget?

About 50 business days ago we got a new coffee machine. It has a neat feature that it actually counts all the coffees it makes. The number today was 1483. This means, it takes about 30 coffees a day!

New hardware is installed and ready to go!

Now that the hardware is in place, our new Hadoop/HBase cluster is almost ready for the team to use!

Faster tests for Rails

We have a lot of tests. Everytime there is a problem on pubget.com, we try to turn this into a test. This has added up over the last 3 years to 2 hours of functional and unit tests (we have others but not for the pubget.com build process).

Recently we lost one of our testing servers due hardware failure. This in itself was not problem as all the tests and database configuration is easily reproduced. However, waiting 2 hours between starting a test and checking if it worked was too long to wait.

We also noticed the server was not taxed at all. The CPU was not busy, HDD hardly getting much use and the memory was plentiful. The problem was just everying was single threaded and not using the 4 cores available.

As a result, we looked into a multi process solution to this. Thanks to the good work at http://ruby-toolbox.com/, we started with
Parallel Tests and found it 100% compatible with our fork of CruiseControl.rb.

We had to manually create the pubget_test1, pubget_test2, etc databases because our rails user does not have database create permissions but the rest of the instructions in their README worked perfectly.

If you want to save time in your tests and have more than one core or processor - I would encourage parallel testing. The next step is probably multi-box testing but this was such an easy win to double performance it was irresistible to include when we rebuilt the server.

* We also now have a hot spare for the build server thanks to unison for making a hot spare very easy to maintain.

Installing ImageMagick on OS X Leopard

In the past, getting image magick to work in ruby on rails for OSX was near impossible.

Even if you went down the macports route, you would still run into problems. However, it looks like if you have the latest xcode and leopard, things are a lot easier.

cd /tmp
curl -O http://www.ijg.org/files/jpegsrc.v7.tar.gz
tar -zxf jpegsrc.v7.tar.gz
cd jpeg-7/
ln -s `which glibtool` ./libtool
./configure --enable-shared --prefix=/usr/local
make
sudo make install

then

cd /tmp
curl -O ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar -zxf ImageMagick.tar.gz
cd ImageMagick-*/
./configure --prefix=/usr/local
make
sudo make install

which then only leaves the gem

sudo gem install rmagick

Thanks to all that made this easy to install and this link for the code.