Goiardi version 0.7.2 is a minor yak shaving release that there isn’t a lot to say about. The CHANGELOG pretty much sums it up:

• Remove a newline in a debug statement, courtesy of @spheromak.
• Also per @spheromak’s suggestion, fixed some possible race conditions revealed by building goiardi with the -race flag and running chef-pedant against it.
• Edit doc.go slightly to make godocs more attractive.

Elsewhere, shovey work continues and comes closer to completion.

The media refuses to report on this scourge afflicting the Florida coast, which is why you never hear about it.

“Shovey” is chugging along on a fast pace, but in the meantime a couple of issues came up that I thought deserved attention. Goiardi version 0.7.1 is a minor release, addressing two relatively small issues.

First, this release adds options for configuring the db pool size and max number of connections. It was happening automagically before, but I was always a bit uneasy about that. Until last night, though, I kept missing how to set that in the documentation for database/sql.

Second, for cookbook files stored locally (which is currently all of them), goiardi will now set the URL of the resource to use the current configured hostname. This was brought to my attention because someone had uploaded a bunch of cookbooks, changed the hostname and port and turned on SSL, and had the URLs break. It takes a specific chain of events to see the problem, which is why it had gone unnoticed up to this point.

From the CHANGELOG:

• Add –db-pool-size and –max-connections options for configuring the number of idle db connections kept around and the maximum number of db connections to make to the server. It isn’t particularly useful if you’re not using one of the SQL backends.
• For locally stored cookbook files (which is currently all of them), goiardi now generates the URL to the resource from the currently configured hostname. This fixes an issue where if you uploaded a cookbook and then changed the goairdi server’s hostname, the URLs to download cookbooks would break.

Other than that, there’s nothing else new in this release.

In case you haven’t read it, Orphans of the Sky is a novel by Robert A. Heinlein about a generation ship floating through space where the people on board have forgotten that they are on a space ship. It’s pretty good, but more importantly the name of the first of the two novellas that make up the novel is titled “Universe”. The title of the second part, “Common Sense”, became relevant after I really dug into making this release for reasons that will soon become clear.

While it ended up being a bigger deal than I expected it to be, I am pleased to announce the release of goiardi version 0.7.0.

I took what I thought would be a brief detour from my current work on the serf integration and “shovey” stuff for goiardi to implement Chef RFC 014 real quick like, which I had been considering doing even before the RFC came along. I had been planning on making this a small release numbered 0.6.1, but then after I proposed a mechanism for caching the universe endpoint it was suggested in the RFC discussion that I see how goiardi performed with the full load of cookbooks in the supermarket before jumping the gun and adding caching before it was necessarily needed.

It turned out that will over 6,000 cookbooks the universe endpoint ran like a dog. Ultimately the problem boiled down to using gob to encode the complex data structures in cookbooks, nodes, etc. in the database. Generally gob encoding in golang is faster than using JSON, but in this particular case JSON is actually faster. The “common sense” thing to do was to change how those complex structures are stored in the database, even though it’s a bit of a brutal change. Sometimes, though, you have to bite the bullet and do the thing that sucks. It’s ended up working nicely, however; the postgres implementation is now able to make use of the json type and the json functions built into Postgres. Before switching from gob to json, serving /universe with over 6200 cookbooks took over 3 seconds with Postgres or MySQL, but afterwards Postgres can serve it in roughly 325 milliseconds. MySQL isn’t able to take advantage of the Postgres-specific json functionality, but it still went down to 1 second, so that’s a marked improvement. Using the in-memory (with or without file backing) takes about 1.2 seconds. I’m willing to take this; if you have that many cookbooks and can’t wait a second for /universe, you may want to use Postgres for your data store.

From the CHANGELOG:

• Add /universe API endpoint, per https://github.com/opscode/chef-rfc/blob/master/rfc014-universe-endpoint.md.
• Make file uploading a little more forgiving.
• Make validating some cookbook metadata more forgiving, to bring goiardi’s validations in line with erchef.
• Added some functions to make listing all cookbooks and recipes on the server faster and move the logic into the cookbook package.

• Breaking DB change: with both MySQL and Postgres, the way data structures for cookbooks, nodes, etc. has changed from gob encoding to using JSON. This obviously breaks existing items in the database, so the following steps must be followed by users using either SQL backend for data storage:

  • Export their goiardi server’s data with the -x flag.
  • Either revert all changes to the db with sqitch, then redeploy, or drop the database manually and recreate it from either the sqitch patches or the full table dump of the release (provided starting with 0.7.0)
  • Reload the goiardi data with the -m flag. See the README or the godocs for more information.

More documentation on the universe endpoint in goiardi is available in the README. Specific information on upgrading is also in the README in the UPGRADING section.

Back to the serf/shovey stuff, then.

The Order of the Elephant is a Danish order awarded mostly to foreign heads of state and royals. Its elephant-themed insignia is appropriate for this latest release of goiardi, version 0.6.0.

Along with some bug fixes and smaller improvements, this release introduces Postgres as a supported database backend. The schema is not compatible with erchef’s schema, but is close to the goiardi MySQL schema (with some differences of course). This is the first project I’ve done with Postgres, so comments on how goiardi is using it are welcome.

Full CHANGELOG:

• Postgres support.
• Fix rebuilding indexes with an SQL backend.
• Fix a bug where in MySQL mode events were being logged twice.
• Fix an annoying chef-pedant error with data bags.
• Event logging methods that are not allowed now return Method Not Allowed rather than Bad Request.
• Switch the logger to a fork that can be built and used with Windows that excludes syslog when building on Windows.
• Add basic syslog support.
• Authentication protocol version 1.2 now supported.
• Add a ‘status’ param to reporting, so a list of reports return by ‘knife runs’ can be narrowed by the status of the chef run (started, success, and failure).
• Fix an action at a distance problem with in-memory mode objects. If this behavior is still desirable (it seems to be slightly faster than the new way), it can be turned back on with the –use-unsafe-mem-store flag. This change DEFINITELY breaks in-mem data file compatibility. If upgrading, export your data, upgrade goiardi, and reload your data.
• Add several new searchable parameters for logged events.
• Add organization_id to all MySQL tables that might need it someday. Orgs are not used at all, so only the default value of 1 currently makes it to the database.
• Finally ran ‘go fmt’ on goiardi. It didn’t even mess up the long comment blocks, which was what I was afraid it would do. I also ran golint against goiardi and took its recommendations where it made sense, which was most areas except for some involving generated parser code, comments on GobEncode/Decode, commenting a bunch of identical functions on an interface in search, and a couple of cases involving make and slices. All in all, though, the reformatting, linting, and light refactoring has done it good.

To reiterate, this update breaks saved data file compatibility. You’ll need to export and re-import your data if you’re using the persistent in-memory data store.

A selection of precompiled binaries are provided on the release page, including Windows and Solaris builds. There are also knife plugins for goiardi’s reporting and event logging capabilities at knife-goiardi-reporting and knife-goiardi-event-log. The reporting plugin was forked from the official Chef knife-reporting plugin to add support for the “status” parameter. Both plugins are also available on rubygems.

Finally, after someone asked how goiardi would handle running ~500 nodes, I decided to take a look and see. I whipped up a little test script to create a thousand nodes and clients to see how goiardi (and more importantly the search) would handle it. I wasn’t particularly worried about the non-search portions of goiardi (1000 rows on a table isn’t very much at all, after all), but I hadn’t tested search extensively with large amounts of data. After creating those 1,000 nodes and clients, goiardi was using about 1.40 GB of RAM. An earlier test with 5,500 nodes (but no clients) used 6.45GB RAM. The numbers show that there’s definitely room for improvement with the search, in that the capability for indexing nodes and data bags is limited by RAM, but performance was good even at 5,500 nodes. More rigorous testing is needed, however. Search is an area I intend to revisit shortly to improve these problems.

The next issues to work on with goiardi are currently improving search to work even better with large amounts of data and use raft (or something similar) to allow multiple goiardi instances to keep their search indexes consistent, reworking the permissions system, being able to upload files to s3 or similar storage providers, and the serf based pushy type thing.

With the recent release of Go 1.3, Go now has experimental support for Solaris and its various derivatives. This means that goiardi will build and run on Solaris, although in an experimental way.

Unfortunately to get Go on Solaris/illumos/etc. you have to build it yourself from source, and at least when I was trying it out with OmniOS in vagrant it did not come with gcc by default. Getting go up so you can try out goiardi (if you don’t want to get the binary is pretty straightforward.

• First, install gcc. These instructions set you through installing gcc. It wasn’t clear to me at first, but you do have to run all the pkg install commands mentioned. First install the appropriate version of gcc, then install those other packages.

Once that’s done, make sure your PATH is set per the instructions on that page.

• Install git with pkg install developer/versioning/git

• Next, download go 1.3: wget http://golang.org/dl/go1.3.src.tar.gz and extract it somewhere. This will be your GOROOT. Set the environment variable with export GOROOT=<go's location> (assuming you’re using bash or similar).

The next steps come from *Installing Go from source:

• In this new GOROOT, cd src/, then run ./all.bash. It will build go for you. If it complains about being unable to find unistd.h, you were like me and forgot the second step on that Installing GCC page.

• You want to actually be able to use go, so add it to your path: export PATH=$PATH:$GOROOT/bin.

• Now you need a workspace. In your home directory, run:

mkdir gocode
mkdir gocode/src
mkdir gocode/bin
mkdir gocode/pkg
export GOPATH=$HOME/gocode
export PATH=$PATH:$GOPATH/bin

• Almost there. All that’s left is running

go get -t github.com/ctdk/goiardi
go install github.com/ctdk/goiardi

Assuming all went well, you can run goiardi at your prompt with whatever options you want, and it will happily run. After this point using goiardi is like using it on any other platform.

Between the previous goiardi release and this one, my kids discovered Minecraft. This means that I started playing Minecraft, which ended up slowing goiardi development down for a while.

This blocky adversary has been overcome, however, and goiardi version 0.5.2 is out.

Originally the next goiardi release was going to add Postgres support. This was about halfway done, and will be resumed shortly, but I decided that it was more important to add the ability to import and export data before any of the goiardi structs changed.

Changes in this release, from the CHANGELOG:

• Add import/export of goiardi data through a JSON dump.
• Add configuration options to specify the max sizes for objects uploaded to the filestore and for JSON requests from the client.

Importing and exporting is explained in the README and in the godocs. Basically you add the -x/--export flag to your usual goiardi command to export the data to a JSON file, and run the -m/--import flags with your new desired goiardi set of flags to import the data. The administrator is responsible for removing any existing data (on-disk or SQL) and index files. This can help with backups or moving between the different storage engines goiardi supports, keeping data between upgrades, and should make testing easier in the future.

Also, per the README, there are new options to specify the maximum size of an object that can be uploaded to the filestore, and the maximum size of a JSON request from a client. The -Q/--obj-max-size flag gets the filestore object max size, while -j/--json-req-max-size sets the JSON request max size limit.

Completely unrelated to any of my efforts, but while I was preparing this release Go 1.3 was released. Goiardi and its dependencies build just fine with Go 1.3 with no changes. This also means that goiardi should be able run on some of the stranger platforms out there, like Solaris, the illumos based distributions, plan9, and apparently Dragonfly BSD. So far goiardi has only been build on OmniOS r151008j, but it built fine.

Finally, as an added convenience: on the goiardi 0.5.2 release page, precompiled binaries are being provided for download, if you don’t feel like setting up go and compiling them yourself. As of this writing binaries are provided for Linux (built on Debian wheezy), MacOS X (built on 10.9 Mavericks), FreeBSD (built on FreeBSD 9.2), and illumos (built on OmniOS r151008j). The Linux, MacOS X, and FreeBSD binaries are x86_64, the illumos binary i386 update: I misinterpreted the output of psrinfo - it was compiled for 64 bit. It appears that there’s no reason that goiardi can’t easily be cross compiled though, so a more sensible set of precompiled binaries will be coming soon. If you don’t see binaries for your platform, try grabbing a binary that’s built on a similar platform to yours, or compile it yourself.

Coming down the pipeline is resuming and re-merging the Postgres work, making some data structures safer for in-memory mode, some extra features for event logging and reporting, improving tests and moving away from depending on chef-pedant, and an idea I’ve been kicking around for a serf-based pushy type thing. For those curious about the Postgres goiardi schema, you can see it in the ‘postgres’ tree in goiardi-schema. It’s firmed up, but still subject to change before getting merged back in.

There’s now a knife plugin to use with goiardi’s new event logging capabilities: knife-goiardi-event-log 0.1.1.

Usage info from the README:

There are four goiardi event log commands: knife goiardi gel list, knife goiardi gel show, knife goiardi gel delete and knife goiardi gel purge.

knife goiardi gel list returns a list of logged events. A limit may be specified with --limit; the default limit is 15. An offset may also be specified; the default offset is 0.

knife goiardi gel show [id] gives a detailed listing of a particular event log item.

knife goiardi gel delete [id] deletes the given event, and prints out a detailed display of the deleted event.

knife goiardi gel purge [id] purges all event log items with an id less than the given id. After purging the events, it displays the number of events that were purged.

Being able to list events of a certain kind of action, type of object, time range, etc. would be nice and are on the roadmap for this plugin. Adding that capability depends on changes to goairdi before that can happen though. This plugin makes use of all the current event log functionality however.

The sky darkens. Blotting out the sun, Rodan appears to destroy beloved landmarks and stomp on small models of tanks. Goiardi version 0.5.1 has been released.

This is an incremental release, made up of bug fixes and relatively minor new features. The most interesting new features are event logging and reporting, although the log level changes are also nice.

From the CHANGELOG:

• Add log levels (from debug to critical). This makes -V/–verbose useful.
• Add an easier option in the config file to specify log levels by name.
• ipv6 already worked, but accidentally. Now it works in a more deliberate fashion, preventing mishaps with addresses, colons, and port numbers.
• Authentication protocol version 1.1 now supported.
• Remove a sort on run lists that was there for some reason. I have no idea what it was put there for, but it was wrong.
• Add an event log to log changes to objects like nodes, clients, etc. See the README or godocs for details.
• Add support for reporting (http://docs.opscode.com/reporting.html)

This release should not break old save files, if you’re using that. If you’re using MySQL, you’ll need to deploy the sqitch changes.

The next big things to start working on are the long promised Postgres support, and etcd/serf/consul etc. integration. Groupcache has also been asked for and looks like a worthwhile thing to pick up. As always, the issues page is a good place to look at what’s being worked on.

Goiardi is also starting to grow beyond using chef-pedant for most of its testing; it’s about time to start more native golang tests for goiardi and not use chef-pedant for the main tests. It should always work with chef-pedant, of course, but goiardi’s starting to get more features that chef-pedant doesn’t check.

Stormily agitated, a new goiardi release appears! Goiardi version 0.5.0 has been released.

Aside from a few bug fixes, this release brings MySQL support. Now, in addition to running entirely in memory, goiardi can use MySQL for storing data. The README and godocs explain how to set that up.

From the CHANGELOG:

• MySQL support added
• No longer redirect /environments/NAME/roles/NAME to /roles/NAME/environments/NAME
• Update documentation, reformat godocs
• Split actors apart into separate user and client types, made new Actor interface that encompasses both users and clients.

Note: This release contains one breaking change. Saved data files from previous goiardi releases will not work with 0.5.0, so if you’re in that situation either don’t upgrade or remove your saved index and data files before starting up. This will regenerate your admin, validator, and webui keys, and of course nuke anything you had in place before.

Postgres support has already been requested by a few people, so that should be coming fairly soon. The issues page has been filled in with some enhancements that would be nice for goiardi to have. Feel free to add your own if there’s something you’d like to see.

It seems like only yesterday that there was a goiardi release, but really it was a few days ago. Goiardi version 0.4.2 is hot off the presses and ready for your perusal.

This is a bugfix release. The reason this is the “Spinal Tap” release was because I noticed these problems while I took the day off after getting a spinal tap and was bored, so it seemed fitting. The bug fixes, from the CHANGELOG, are:

• Perm tweak for nodes updating themselves.
• Small change with validating role descriptions when creating or updating from JSON.
• Fix issue with saving complicated indexed objects to disk where improperly flattened indexable objects were making the gob encoder puke all over itself when encoding the tries in the index docs.
• Fixed a possible regression with synchronizing cookbooks that did not show up in testing, but only in real use.
• An absolutely bonkers fix for listing cookbook files with webui. Webui wants all of the cookbook top level attributes sent over with a request to /cookbooks/<name>/<version>, but this is the exact opposite of the behavior chef-pedant wants, where empty definitions, attributes, etc. are not sent over. Knife also seems quite content with this, so the fix for now, since the two cases are mutually exclusionary, is to only send the empty hashes for those top level attributes with a GET if the request is coming from the webui. Bizarre, but it seems to be what’s necessary.

These were all problems that evaded notice during testing, but did pop up when I was using goiardi to set up a vagrant VM to start working on letting goiardi run a real database (optionally, of course) instead of the in-memory data store system it’s using now. The in-memory mode is great and all for testing, and being able to freeze the data to disk for persistence is very handy, but being able to use a real database would be nice for some workloads. Supporting both the current in-memory storage and a database backend is a top priority.

Which database to use is still a bit up in the air, but I’m leaning towards throwing caution to the wind and providing support for both MySQL and Postgres, with SQLite as a distinct possibility down the line. MySQL is likely to come first, but that is not set in stone.

Hot on the heels of the 0.4.0 release comes goiardi version 0.4.1. This release adds a –disable-webui option to disable using chef-webui with goiardi, fixes a bug with parsing configuration file options and rearranges how some of the config struct items are set, fixes a typo in the sample config file, and makes some small documentation tweaks.

Other than that, all the changes announced in the 0.4.0 release announcement and previous releases remain relevant.

I am pleased to announce another goiardi release.

This release, version 0.3.3 of goiardi, doesn’t add much in the way of new functionality. The data store and indexer have been tweaked slightly, and go tests have been added for some of the goiardi components. The go test coverage isn’t complete, and most goiardi testing is handled with chef-pedant, but these golang tests will be handy for testing internal functionality that could break, yet not show up in the chef-pedant testing.

While there isn’t much in this release, I decided to make one here because it’s a good place to make one before starting to tackle authentication. Enjoy.

Update: It’s a little embarrassing, but there was a problem merging branches, and a necessary change got backed out or overlooked, so version 0.3.1 has been superseded by version 0.3.3. On the off chance anyone got the previous version, I apologize.

Pushed up a change in the latest goiardi master last night that fixes an issue with saving the persistent index and data store files where, if the temp directory was not on the same partition, renaming the file after saving would cause an error. I was clued into this by a tweet in Japanese that seemed to say something about that. Ooops.

As of the most recent commit, goiardi can now rebuild its search index in case it accidentally got deleted or corrupted somehow.

To rebuild, just run knife index rebuild (or knife index rebuild -y if you don’t want to be bothered to press “y” to confirm that you really want to do it). You’ll get a message from knife that ‘knife index rebuild’ has been removed for Chef 11+ servers. This can be ignored; while goiardi identifies itself as a Chef 11+ server, this command will work because they’ve kept it in for older versions of Chef.

Originally posted on time.to.pullthepl.ug.

I am pleased to announce another goiardi (a Chef server written in go) release.

Version 0.3.0 of goiardi adds the ability to save the in-memory data store and the search index to disk, rather than losing everything everytime the server restarts. If the options are set, goiardi will save when it receives a SIGTERM or SIGINT signal, as well as save in the background periodically. The interval defaults to five minutes, but can be specified as an option as well. In the interest of safety and data integrity, goiardi will first save to a temporary file before renaming it to the save file name.

Of course, goiardi will also still happily run entirely in memory as before.

NB: Bear in mind that even if the data has been saved to disk goiardi still keeps the data store and search index in memory while it’s running. This could cause problems if you try to shove too much data into it. I expect most use cases would have no problems (famous last words), but the appropriate caution is warranted.

It still does no authentication or permission checking, and while it does its best to avoid data corruption and data loss it is not a real database and should not be relied on as such. It may be useful for situations where you do not want or need a full real Chef server, but could use persistent data which you don’t get with chef-zero.

Next tasks coming up: provide a way to reindex the data, improve existing docs as needed and write more, make actual go tests, fork chef-pedant and make a “goiardi” mode to simplify testing against chef-pedant.

Anyway, that’s what this release has. Have fun with it if you’re so inclined, and thank you if you do.

Originally posted on time.to.pullthepl.ug.

A while back, I decided that I would like to learn Go, which is a pretty neat language all around (in my humble opinion). I had to come up with some sort of project that would be ambitious enough to teach myself a lot of the language features, but not so huge that I’d give up in frustration. Then I thought of a name, and realized what I had to do.

After many evenings spent coding, I’m proud to announce the first offical release of Goiardi, an in-memory Chef server written in Go (the README has more information). (I started on it before I found out about Chef-Zero, but I decided to keep at it anyway. Chef-Zero was also a huge help in figuring out what the Chef server behavior ought to be.) At this time it has pretty much all the features Chef-Zero does, but sorting searches isn’t working yet. If anyone has real examples of how those sorts are specified, I’d love to hear them.

Goiardi does not support authenication or validation at all, so it isn’t very secure. Of course, it also doesn’t persist its contents, so you probably shouldn’t rely on it for your infrastructure provisioning needs.

Aside from the authentication and permission tests, goiardi performs pretty well against chef-pedant. There are a few minor tests that fail because of slight JSON formatting differences or error messages that don’t agree with chef-pedant, but otherwise it does quite well. Improving its performance against chef-pedant is, of course, an ongoing concern. It should do fine as a stand-in for chef-zero though, or some other situation where you need a chef server running but don’t care if it loses all its information when it quits.

Goiardi is, like many Chef things, open source and licensed under version 2.0 of the Apache License.

The Future

Where to go from here? Goiardi certainly has a lot of room for improvement. The first things I intend to do with it are allowing you to use a config file to store options, and allowing freezing the in-memory data structures to disk so they persist when goiardi is shut down. Those aren’t in this release because if I kept adding “just one more thing” before making a formal release, I’d never actually get it out.

Further in the future, handling real authenication and permissions is pretty important. A real Chef Server mode, where goiardi uses Solr and a real database, is on the roadmap down the road as well. SSL, better documentation, and making the Go code more idiomatic where needed are also on the TODO list.

Anyway, there is is. Feel free to check it out, bang on it, see where it breaks (and where it works). Suggestions, code, and comments are always welcome. Hopefully the comments are nice, but this is the Internet.

Update: At least part of the future is now. Goiardi has config file support now with 0.2.1, huzzah! This release also fixes a problem building goiardi with more recent versions of go-flags.

Goiardi has been tested to compile with the native go compiler and run on Mac OS X (10.7 and 10.8), Debian ‘wheezy’, and Arch Linux. It also builds and runs with gccgo (using the -compiler gccgo option to go), tested on Arch Linux. Efforts are ongoing to get it to build with gccgo without the go command on OmniOS, but so far they’ve been unsuccessful.

The Name

If you’re wondering about the name, Ettore Boiardi was the real Chef Boyardee. Wakka wakka.