Junglist Generation

Pennyworth Quick Start

2011-11-16T14:17:00+13:00

At Heavy Water operations we have been working on a Jenkins based continuous deployment system we've named Pennyworth. Should you choose to evaluate pennyworth, any project you desire can be automatically tested, packaged and included in an appropriate repository available for automated installation and deployment with APT (hopefully via Chef!)

This post will hopefully get you up and running in a virtual environment, building the example jobs and walking through each of the keys required to create a new job.

Requirements

Ruby versions: 1.9.3-p0, 1.9.2-p290, 1.8.7-p352
VirtualBox 4.1.6
VirtualBox 4.1.6 Extension pack
4GB RAM (tunable in Vagrantfile)

Clone the repository

I'm using /tmp in this example.

haxstation :: /tmp » hub clone heavywater/pennyworth
Cloning into pennyworth...
remote: Counting objects: 298, done.
remote: Compressing objects: 100% (179/179), done.
remote: Total 298 (delta 72), reused 295 (delta 69)
Receiving objects: 100% (298/298), 114.21 KiB, done.
Resolving deltas: 100% (72/72), done.

Installation of dependencies

Now that you have a clone you can begin installing the dependencies with Bundler:

haxstation :: /tmp/pennyworth ‹develop› » bundle install
Fetching source index for http://rubygems.org/
Installing archive-tar-minitar (0.5.2)
Installing erubis (2.7.0)
Installing ffi (1.0.10) with native extensions
Installing i18n (0.6.0)
Installing json (1.5.2) with native extensions
Installing net-ssh (2.1.4)
Installing net-scp (1.0.4)
Installing thor (0.14.6)
Installing virtualbox (0.9.2)
Installing vagrant (0.8.7)
Using bundler (1.0.21)
Your bundle is complete! Use `bundle show [gemname]` to see where a
bundled gem is installed.
haxstation :: /tmp/pennyworth ‹develop› »

Tune Vagrant

With the dependencies all installed, you're ready to kick off Vagrant. If you don't have 4GB of ram installed, open up the Vagrantfile and edit line #4, reducing the vm.memory_size value:

``` ruby

Vagrant::Config.run do |config|
  config.vm.define :pennyworth do |pennyworth|
  pennyworth.vm.customize do |vm|
    vm.memory_size = 4096

```

Vagrant time

You're ready to launch Pennyworth!

It takes 3000s+ on my very hefty development workstation:

Intel i7-2600k @ 4.9 GHz (HT/WC)
16GB RAM (Vagrant configured to use 8192MB)
Intel 510 SSD

Testing on EC2 has been much quicker due to bandwidth available provided you can afford the resource.

vagrant status

haxstation :: /tmp/pennyworth ‹develop› » bundle exec vagrant status
Current VM states

pennyworth               not created

The environment has not yet been created. Run `vagrant up` to
create the environment.

vagrant up

haxstation :: /tmp/pennyworth ‹develop› » bundle exec vagrant up
[pennyworth] Importing base box 'natty64'...
[pennyworth] Matching MAC address for NAT networking...
[pennyworth] Clearing any previously set forwarded ports...
[pennyworth] Forwarding ports...
[pennyworth] -- ssh: 22 => 2222 (adapter 1)
[pennyworth] -- web: 8080 => 8080 (adapter 1)
[pennyworth] Creating shared folders metadata...
[pennyworth] Running any VM customizations...
[pennyworth] Booting VM...
[pennyworth] Waiting for VM to boot. This can take a few minutes.
[pennyworth] VM booted and ready for use!
[pennyworth] Mounting shared folders...
[pennyworth] -- v-root: /vagrant
[pennyworth] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1
[pennyworth] -- v-csr-2: /tmp/vagrant-chef-1/chef-solo-2
[pennyworth] -- v-csdb-3: /tmp/vagrant-chef-1/chef-solo-3
[pennyworth] Running provisioner: Vagrant::Provisioners::ChefSolo...
[pennyworth] Generating chef JSON and uploading...
[pennyworth] Running chef-solo...
...

[pennyworth] [Wed, 16 Nov 2011 02:43:33 +0000] INFO: Chef Run complete
in 3482.654154 seconds
[pennyworth] [Wed, 16 Nov 2011 02:43:33 +0000] INFO: Running report
handlers
[pennyworth] [Wed, 16 Nov 2011 02:43:33 +0000] INFO: Report handlers
complete
haxstation :: /tmp/pennyworth ‹develop› »

3482 seconds = 58.0333333 minutes

You'll want to avoid doing a full vagrant destroy; vagrant up cycle very often, unless you have a very fast connection.

Jenkins Dashboard

Reprepro index

Adding a pennyworth job with data bags

Here's the ruby pennyworth job data bag included as an example:

(ruby.js) download

{ "id" : "ruby",
  "version" : {
    "major" : "1.9",
    "minor" : "3"
  },
  "project_type" : "package_from_git",
  "git_branch" : "ruby_1_9_3",
  "git_url" : "git://github.com/ruby/ruby.git",
  "test_commands" : [
    "autoconf",
    "./configure --prefix=/usr/local --disable-install-doc",
    "make test"
  ],
  "build_commands" : [
    "make install DESTDIR=\$DESTDIR"
  ],
  "package_commands": [
    "mkdir -p fpm",
    "cd fpm; fpm -s dir -t deb -n ruby1.9.3-full -v \$VERSION -C \$DESTDIR -p ruby1.9.3-full-VERSION_ARCH.deb --post-install /var/lib/jenkins/ldconfig usr/local/bin usr/local/include /usr/local/lib usr/local/share",
    "sudo /usr/bin/reprepro -Vb /srv/apt includedeb \$LSB_CODENAME \$DESTDIR/ruby1.9.3-full-\$VERSION_*.deb",
    "sudo /usr/local/bin/update_version.rb \$JOB_NAME \$VERSION"
  ],
  "child_projects" : [
      "install-dependency-packages"
  ],
  "remote_poll": true,
  "clean" : true,
  "wipeoutworkspace" : true
}

Let's look at each of the keys.

version (line 2 - 5)

This key contains a hash that Pennyworth uses to construct a version number automatically, available to shell scripts during the build process as $VERSION.

In this example, $VERSION would be 1.9.3.$BUILD_NUMBER

project_type (line 6)

Controls which type of project pennyworth thinks this data bag item actually is.

In this example, the project type is Package from Git, signifying that we will be continuously packaging a Git repository.

Other available project types are: test_from_git which tests without packaging and package which lacks the Git support -- we have used it to install legacy packages only available over HTTP.

This really only exists because a suitable abstraction hasn't been built around the jenkins_job LWRP yet, which we're planning on addressing shortly. Currently we pull all values out of the data bag and assign them (if available). It's pretty gross.

git_branch (line 7)

Used for the *_from_git project types; controls which branch Jenkins will track for changes.

git_url (line 8)

Used for the *_from_git project types; controls which remote Jenkins will track for changes.

test_commands (line 9 - 13)

An array of shell commands to run to start the testing off from a fresh checkout, so, potentially including autoconf and configure... Obviously hard to provide a catch all for this.

We have been considering an advanced build tool that can determine any and all appropriate commands required to kick this phase off.

build_commands (line 14 - 16)

An array of shell commands to run after testing to produce a compiled version of your project.

Bundler could be used during this phase to make pre-built Gem extensions, etc. available to the packaging phase. We've used this technique for Rails deployments in the past. Consequently, this is where build tools like Make, Rake and Leiningen have been used to produce compiled versions of many of our clients applications.

Our example uses the $DESTDIR variable, which is generated by Jenkins as $WORKSPACE/fpm; i.e. a sub directory in the workspace. This variable is available to all project types.

package_commands (line 17 - 23)

An array of shell commands to run after building a compiled version of the project.

Our example shows the steps required to package a directory with FPM and include them into the built in APT repository managed by Reprepro (previously known as mirrorer)

It also makes use of our update_version.rb script which uses the Chef::DataBagItem class to update an item in the package data-bag. We use this during deployment to figure out what versions of packages have been built by pennyworth.

This would be another great extension point for the build tool I hinted at earlier that could automatically determine what kind of files should be included into packages and manage the inclusion of the artifact in a repository.

We have used this section to launch notification and deployment tools that signal Chef to run over the network, including via SSH with the Knife Deploy plugin which should be available as a gem soon.

child_projects (line 24 - 26)

An array of job names that this job is a parent of. Used to create dependencies/triggers between jobs.

remote_poll (line 27)

Enable or disable remote polling of the Git repository.

Setting this to false means the job will not poll Git, but can be triggered by API or the Web UI.

clean (line 28)

Enable or disable the cleaning of the workspace.

wipeoutworkspace (line 29)

Enable or disable the "wiping out" of the workspace, total removal. Probably not required with clean.

Questions?

Contact AJ aj@junglist.gen.nz and Darrin darrin@heavywater.ca

Heavy Water operations has sponsored and requisitioned the development, testing and deployment of Pennyworth. Feel free to contact us to discuss more complicated requirements, or any problems. No warranty implied =)

MiniTest Cookbook 0.0.5 Released

2011-08-13T14:46:00+12:00

A few days ago I released version 0.0.5 of the MiniTest Cookbook which included a feature I've been working on that allows you to build MiniTest unit test cases programmaticaly throughout the Chef run and have them executed by a dedicated test runner.

``` ruby

minitest_unit_testcase "test something" do
  block do
    refute_equal true, false, "true is not false"
    assert_equal true, true, "true is not equal true
  end
end

```

I've also retained the ability to run via Vagrant, for ease of testing. The develop branch has some more recent enhancements, notably an in-line Sinatra server which I'm building example test cases around.

vagrant@vagrant:~$ sudo chef-solo -c /tmp/vagrant-chef-2/solo.rb -j
/tmp/vagrant-chef-2/dna.json
[Sat, 13 Aug 2011 03:03:58 +0000] INFO: *** Chef 0.10.2 ***
[Sat, 13 Aug 2011 03:03:58 +0000] INFO: Setting the run_list to
["recipe[minitest]", "recipe[minitest::examples]"] from JSON
[Sat, 13 Aug 2011 03:03:58 +0000] INFO: Run List is [recipe[minitest],
recipe[minitest::examples]]
[Sat, 13 Aug 2011 03:03:58 +0000] INFO: Run List expands to [minitest,
minitest::examples]
[Sat, 13 Aug 2011 03:03:58 +0000] INFO: Starting Chef Run for
vagrant.monkeybrains.net
[Sat, 13 Aug 2011 03:03:58 +0000] INFO: Processing
gem_package[minitest] action install (minitest::default line 19)
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: gem_package[minitest]
installed version ~> 2.3.1
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Processing
gem_package[net-tftp] action install (minitest::default line 24)
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Chef Handlers will be at:
/tmp/handlers
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Processing
remote_directory[/tmp/handlers] action create (chef_handler::default
line 23)
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Processing
cookbook_file[/tmp/handlers/README] action create (dynamically
defined)
[Sat, 13 Aug 2011 03:04:08 +0000] INFO:
remote_directory[/tmp/handlers] created
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Processing
cookbook_file[/tmp/handlers/chefminitest.rb] action create
(minitest::default line 28)
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Processing
chef_handler[ChefMiniTest::Handler] action enable (minitest::default
line 29)
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Enabling
chef_handler[ChefMiniTest::Handler] as a report handler
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Enabling
chef_handler[ChefMiniTest::Handler] as a exception handler
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Processing
gem_package[sinatra] action install (minitest::examples line 1)
[Sat, 13 Aug 2011 03:04:08 +0000] INFO: Processing
gem_package[minitest] action install (minitest::default line 19)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: gem_package[minitest]
installed version ~> 2.3.1
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
gem_package[net-tftp] action nothing (minitest::default line 24)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
remote_directory[/tmp/handlers] action nothing (chef_handler::default
line 23)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
cookbook_file[/tmp/handlers/chefminitest.rb] action create
(minitest::default line 28)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
chef_handler[ChefMiniTest::Handler] action nothing (minitest::default
line 29)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
gem_package[sinatra] action nothing (minitest::examples line 1)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
minitest_unit_testcase[test_truth] action create (minitest::examples
line 5)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
minitest_unit_testcase[test_http_port] action create
(minitest::examples line 55)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
minitest_unit_testcase[test_http_fitter_happier] action create
(minitest::examples line 55)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
minitest_unit_testcase[test_dns_resolution] action create
(minitest::examples line 55)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Processing
minitest_unit_testcase[test_tftp_binary_get] action create
(minitest::examples line 55)
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Chef Run complete in
10.741559696 seconds
[Sat, 13 Aug 2011 03:04:09 +0000] INFO: Running report handlers
Run options: -v --seed 59848

# Running tests:

[Sat, 13 Aug 2011 03:04:09 +0000] INFO: ChefMiniTestRunner starting
##test_truth = 0.00 s = .
##test_http_port = 0.00 s = E
##test_http_fitter_happier = 0.00 s = E
##test_dns_resolution = 0.03 s = E
##test_tftp_binary_get = 5.03 s = E
[Sat, 13 Aug 2011 03:04:14 +0000] INFO: ChefMiniTestRunner completed


Finished tests in 5.063267s, 0.9875 tests/s, 0.1975 assertions/s.

  1) Error:
test_http_port(#):
Errno::ECONNREFUSED: Connection refused - connect(2)
    /usr/local/lib/ruby/1.9.1/socket.rb:37:in `connect'
    /usr/local/lib/ruby/1.9.1/socket.rb:37:in `connect_internal'
    /usr/local/lib/ruby/1.9.1/socket.rb:86:in `connect'
    /usr/local/lib/ruby/1.9.1/socket.rb:247:in `block in tcp'
    /usr/local/lib/ruby/1.9.1/socket.rb:157:in `each'
    /usr/local/lib/ruby/1.9.1/socket.rb:157:in `foreach'
    /usr/local/lib/ruby/1.9.1/socket.rb:239:in `tcp'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:18:in
    `block in from_file'

  2) Error:
test_http_fitter_happier(#):
Errno::ECONNREFUSED: Connection refused - connect(2)
    /usr/local/lib/ruby/1.9.1/net/http.rb:644:in `initialize'
    /usr/local/lib/ruby/1.9.1/net/http.rb:644:in `open'
    /usr/local/lib/ruby/1.9.1/net/http.rb:644:in `block in connect'
    /usr/local/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
    /usr/local/lib/ruby/1.9.1/timeout.rb:89:in `timeout'
    /usr/local/lib/ruby/1.9.1/net/http.rb:644:in `connect'
    /usr/local/lib/ruby/1.9.1/net/http.rb:637:in `do_start'
    /usr/local/lib/ruby/1.9.1/net/http.rb:626:in `start'
    /usr/local/lib/ruby/1.9.1/net/http.rb:1168:in `request'
    /usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.2/lib/chef/rest/rest_request.rb:84:in
    `block in call'
    /usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.2/lib/chef/rest/rest_request.rb:99:in
    `hide_net_http_bug'
    /usr/local/lib/ruby/gems/1.9.1/gems/chef-0.10.2/lib/chef/rest/rest_request.rb:83:in
    `call'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:30:in
    `block (3 levels) in from_file'
    /usr/local/lib/ruby/1.9.1/timeout.rb:58:in `timeout'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:29:in
    `block (2 levels) in from_file'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:26:in
    `each'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:26:in
    `block in from_file'

  3) Error:
test_dns_resolution(#):
Errno::ECONNREFUSED: Connection refused - recvfrom(2)
    /usr/local/lib/ruby/1.9.1/resolv.rb:747:in `recv'
    /usr/local/lib/ruby/1.9.1/resolv.rb:747:in `recv_reply'
    /usr/local/lib/ruby/1.9.1/resolv.rb:641:in `request'
    /usr/local/lib/ruby/1.9.1/resolv.rb:506:in `block in
    each_resource'
    /usr/local/lib/ruby/1.9.1/resolv.rb:1000:in `block (3 levels) in
    resolv'
    /usr/local/lib/ruby/1.9.1/resolv.rb:998:in `each'
    /usr/local/lib/ruby/1.9.1/resolv.rb:998:in `block (2 levels) in
    resolv'
    /usr/local/lib/ruby/1.9.1/resolv.rb:997:in `each'
    /usr/local/lib/ruby/1.9.1/resolv.rb:997:in `block in resolv'
    /usr/local/lib/ruby/1.9.1/resolv.rb:995:in `each'
    /usr/local/lib/ruby/1.9.1/resolv.rb:995:in `resolv'
    /usr/local/lib/ruby/1.9.1/resolv.rb:498:in `each_resource'
    /usr/local/lib/ruby/1.9.1/resolv.rb:391:in `each_address'
    /usr/local/lib/ruby/1.9.1/resolv.rb:367:in `getaddress'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:38:in
    `block in from_file'

  4) Error:
test_tftp_binary_get(#):
Net::TFTPTimeout: execution expired
    /usr/local/lib/ruby/gems/1.9.1/gems/net-tftp-0.1.0/lib/net/tftp.rb:150:in
    `recvfrom'
    /usr/local/lib/ruby/gems/1.9.1/gems/net-tftp-0.1.0/lib/net/tftp.rb:150:in
    `block (2 levels) in getbinary'
    /usr/local/lib/ruby/gems/1.9.1/gems/net-tftp-0.1.0/lib/net/tftp.rb:149:in
    `loop'
    /usr/local/lib/ruby/gems/1.9.1/gems/net-tftp-0.1.0/lib/net/tftp.rb:149:in
    `block in getbinary'
    /usr/local/lib/ruby/gems/1.9.1/gems/net-tftp-0.1.0/lib/net/tftp.rb:148:in
    `getbinary'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:49:in
    `block (3 levels) in from_file'
    /usr/local/lib/ruby/1.9.1/timeout.rb:58:in `timeout'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:48:in
    `block (2 levels) in from_file'
    /usr/local/lib/ruby/1.9.1/tempfile.rb:320:in `open'
    /tmp/vagrant-chef-2/chef-solo-0/minitest/recipes/examples.rb:47:in
    `block in from_file'

5 tests, 1 assertions, 0 failures, 4 errors, 0 skips
[Sat, 13 Aug 2011 03:04:14 +0000] INFO: Report handlers complete
vagrant@vagrant:~$

Test all the fucking time.

Time for Octopress

2011-08-13T14:29:00+12:00

I've installed octopress (which uses Jekyll under the hood) in a terrible attempt to start blogging again. Here goes nothing.