Pubget Tech Blog

Getting more out of GitHub with Pubget’s Chrome Extension

noreply@blogger.com (Mike Anderson) — Tue, 20 Dec 2011 20:59:00 +0000

At Pubget we host our code base on GitHub because of its simplicity, reliability, and great web interface. The extent to which we depend on GitHub is far more than as a convenience, but as the backbone of our engineering team.

Consequently, when GitHub doesn’t have all of the features we need to do our work as efficiently as possible, it’s not just an annoyance, but a source of friction for the entire organization. To grease the wheels of our daily workflow, we’ve begun building a suite of “GitHub Extras” in the form of a Chrome Extension (we've also used the API to add tags when we deploy!). Our focus thus far has been on the Issues tool provided by the web interface, upon which we have now added the ability to upload attachments or include a screenshot from any open tab.

What does this mean for our organization? When our product team completes a new feature spec, it’s a simple one-click to upload the Word doc to a new issue for the dev team. While testing internally and a page renders incorrectly on Internet Explorer, it’s a two-second task for the QA team to grab a snapshot of the offending site and create an issue for it.

As we say, “time saved is time spent coding,” and we love to code!

Download the extension here.

Learn more about our dev team workflow here.

Pubget's automated builds

noreply@blogger.com (Ian Connor) — Tue, 29 Nov 2011 21:10:00 +0000

As a small team, we don't have the luxury of dedicated release engineers and thus need to automate our build as much as possible. To do this we use git branches, CruiseControl.rb and some nifty shell scripts to keep new code flowing. Coupled with the automation, we also have a weekly build master rotation where one member of the dev team shields the rest from tracking down build failures.

Ok, a picture is worth a thousand words so here's our code flow (everything green is automated):

Developers will typically workout out of the DEVELOPMENT branch so this is where new code begins flowing through the system. We find that it's good practice to link submissions to GitHub issues for management & deploy tracking. The changes then get pulled into STAGING which is the continuous build branch for automated testing. We like CruiseControl.rb because it's light weight & low maintenance. If tests fail, the build master is notified and he can track down the right person to fix the failing test in staging. Once the build is green, the qualified code gets pushed to MASTER which is the production branch behind pubget.com. Since the master branch is considered the healthiest, it feeds down into development and any other project branches that may exist. At this point every branch now has that latest code change and the process begins again.

Too many words? Here's the short flow:

devs submit to development
changes are tested in staging
qualified changes get pushed to master
development & special project branches stay close to shore through master

It's definitely worth mentioning that when test failures happen in staging, the automation pauses. Nothing is pulled from development into staging and nothing is pushed from staging into master. This ensures that only healthy code makes its way into production and only the developer that broke the build needs to fix staging. Other developers can continue merging changes into development knowing they will be tested once staging is working again.

This process has helped in a bunch of ways:

developers submit code and trust that it will be qualified, deployed & propagated to all branches
developers stay productive even when the build is broken
we have a production branch where we can quickly apply a hotfix if needed

How to notify github issues when you deploy

noreply@blogger.com (Ian Connor) — Fri, 04 Nov 2011 20:06:00 +0000

A common problem in any production application is that there is a natural delay between submitting a fix and the fix being deployed. Even with an awesome continuous build process that uses fast parallel testing, it can be hours or a day before the production site sees the fix.

This means the customer service folks either have to continously check the production site or the deployment logs and then know how to test or match the fix numbers with the deployments. This is a hard problem that either solved with labor or customers get notified something is fixed before they can test the fix.

At Pubget, we use github and rake for deployments (to passenger apache servers). So, we decided to solve this problem with code. Essentially, when a fix or number of fixes are deployed, we use the github API to comment on each issue when it is deployed. This allows the customer service folks who are participating in the issues to get automatically notified when their fixes make their way through the build process.

To do this, we call this from the deployment rake task:

This looks for fixes that have made its way to the master branch, through our build process and that have our deploy tag and update them with a comment when the fix was deployed.

Now the customer service folks know exactly when their fixes are on the production site and don't need to monitor or ask developers when they can connect with the customer to give them the good news that their bugs have been fixed.

How many coffees does it take to make Pubget?

noreply@blogger.com (Ian Connor) — Thu, 20 Oct 2011 21:56:00 +0000

About 50 business days ago we got a new coffee machine. It has a neat feature that it actually counts all the coffees it makes. The number today was 1483. This means, it takes about 30 coffees a day!

New hardware is installed and ready to go!

noreply@blogger.com (Ian Connor) — Sat, 20 Aug 2011 16:39:00 +0000

Now that the hardware is in place, our new Hadoop/HBase cluster is almost ready for the team to use!

Faster tests for Rails

noreply@blogger.com (Ian Connor) — Thu, 21 Jul 2011 17:41:00 +0000

We have a lot of tests. Everytime there is a problem on pubget.com, we try to turn this into a test. This has added up over the last 3 years to 2 hours of functional and unit tests (we have others but not for the pubget.com build process).

Recently we lost one of our testing servers due hardware failure. This in itself was not problem as all the tests and database configuration is easily reproduced. However, waiting 2 hours between starting a test and checking if it worked was too long to wait.

We also noticed the server was not taxed at all. The CPU was not busy, HDD hardly getting much use and the memory was plentiful. The problem was just everying was single threaded and not using the 4 cores available.

As a result, we looked into a multi process solution to this. Thanks to the good work at http://ruby-toolbox.com/, we started with
Parallel Tests and found it 100% compatible with our fork of CruiseControl.rb.

We had to manually create the pubget_test1, pubget_test2, etc databases because our rails user does not have database create permissions but the rest of the instructions in their README worked perfectly.

If you want to save time in your tests and have more than one core or processor - I would encourage parallel testing. The next step is probably multi-box testing but this was such an easy win to double performance it was irresistible to include when we rebuilt the server.

* We also now have a hot spare for the build server thanks to unison for making a hot spare very easy to maintain.

Installing ImageMagick on OS X Leopard

noreply@blogger.com (Ian Connor) — Sat, 02 Jul 2011 17:09:00 +0000

In the past, getting image magick to work in ruby on rails for OSX was near impossible.

Even if you went down the macports route, you would still run into problems. However, it looks like if you have the latest xcode and leopard, things are a lot easier.

cd /tmp
curl -O http://www.ijg.org/files/jpegsrc.v7.tar.gz
tar -zxf jpegsrc.v7.tar.gz
cd jpeg-7/
ln -s `which glibtool` ./libtool
./configure --enable-shared --prefix=/usr/local
make
sudo make install

then

cd /tmp
curl -O ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar -zxf ImageMagick.tar.gz
cd ImageMagick-*/
./configure --prefix=/usr/local
make
sudo make install

which then only leaves the gem

sudo gem install rmagick

Thanks to all that made this easy to install and this link for the code.

Quick Ruby on Rails, MySQL gem on OSX tip

noreply@blogger.com (Ian Connor) — Sat, 11 Jun 2011 22:33:00 +0000

After having some new engineers join (we are still hiring) and having the same problem popup, I wanted to share a quick tip on getting your MySQL gem up and running on the latest OSX software/hardware.

Install MySQL and the mysql gem on OSX 10.6

First grab mysql-server 5.1. For Mac OS 10.6, make sure you get the 64-bit DMG.

Ensure you have xcode installed from the extra CD that comes in the box. Now install the mysql gem. You'll need to specify both that it's 64-bit and the location of the config file.


sudo env ARCHFLAGS="-arch x86_64" gem install mysql -v 2.7 -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

It should install then without any complaints. If you see native compilation errors it either means you have the 32bit version installed or MySQL 5.5.

Hello World

noreply@blogger.com (Ian Connor) — Sat, 04 Jun 2011 19:40:00 +0000

Welcome to the Pubget Tech blog.

We will be be posting links and commentary on various big data topics we run into everyday at Pubget.

So, please subscribe and keep a look out for technical information around Solr/Lucene, Ruby on Rails, MySQL, Hadoop/HBase and Cloud Computing.