In case you haven’t read it, Orphans of the Sky is a novel by Robert A. Heinlein about a generation ship floating through space where the people on board have forgotten that they are on a space ship. It’s pretty good, but more importantly the name of the first of the two novellas that make up the novel is titled “Universe”. The title of the second part, “Common Sense”, became relevant after I really dug into making this release for reasons that will soon become clear.

While it ended up being a bigger deal than I expected it to be, I am pleased to announce the release of goiardi version 0.7.0.

I took what I thought would be a brief detour from my current work on the serf integration and “shovey” stuff for goiardi to implement Chef RFC 014 real quick like, which I had been considering doing even before the RFC came along. I had been planning on making this a small release numbered 0.6.1, but then after I proposed a mechanism for caching the universe endpoint it was suggested in the RFC discussion that I see how goiardi performed with the full load of cookbooks in the supermarket before jumping the gun and adding caching before it was necessarily needed.

It turned out that will over 6,000 cookbooks the universe endpoint ran like a dog. Ultimately the problem boiled down to using gob to encode the complex data structures in cookbooks, nodes, etc. in the database. Generally gob encoding in golang is faster than using JSON, but in this particular case JSON is actually faster. The “common sense” thing to do was to change how those complex structures are stored in the database, even though it’s a bit of a brutal change. Sometimes, though, you have to bite the bullet and do the thing that sucks. It’s ended up working nicely, however; the postgres implementation is now able to make use of the json type and the json functions built into Postgres. Before switching from gob to json, serving /universe with over 6200 cookbooks took over 3 seconds with Postgres or MySQL, but afterwards Postgres can serve it in roughly 325 milliseconds. MySQL isn’t able to take advantage of the Postgres-specific json functionality, but it still went down to 1 second, so that’s a marked improvement. Using the in-memory (with or without file backing) takes about 1.2 seconds. I’m willing to take this; if you have that many cookbooks and can’t wait a second for /universe, you may want to use Postgres for your data store.

From the CHANGELOG:

• Add /universe API endpoint, per https://github.com/opscode/chef-rfc/blob/master/rfc014-universe-endpoint.md.
• Make file uploading a little more forgiving.
• Make validating some cookbook metadata more forgiving, to bring goiardi’s validations in line with erchef.
• Added some functions to make listing all cookbooks and recipes on the server faster and move the logic into the cookbook package.

• Breaking DB change: with both MySQL and Postgres, the way data structures for cookbooks, nodes, etc. has changed from gob encoding to using JSON. This obviously breaks existing items in the database, so the following steps must be followed by users using either SQL backend for data storage:

  • Export their goiardi server’s data with the -x flag.
  • Either revert all changes to the db with sqitch, then redeploy, or drop the database manually and recreate it from either the sqitch patches or the full table dump of the release (provided starting with 0.7.0)
  • Reload the goiardi data with the -m flag. See the README or the godocs for more information.

More documentation on the universe endpoint in goiardi is available in the README. Specific information on upgrading is also in the README in the UPGRADING section.

Back to the serf/shovey stuff, then.