It’s been running behind, but goiardi v0.11.10 - Tomato Catchup has finally gotten its act together enough to emerge.

This announcement encompasses two releases, because I dragged my feet for quite a while announcing the 0.11.9 release. As is pretty common with the 0.11.x releases for a while now, the changes here are pretty small. They’re a little more important than some of these other small bugfixes, at least, since they allow newer versions of the Chef tools to work with goiardi.

From the CHANGELOG(s):

0.11.10
-------
* It was brought to my attention by @joerg that 'create_key' is now a valid JSON
  hash key for when creating clients with newer chef tools. Added it to the
  whitelist of valid elements for creating clients. Thanks again for bringing
  it to my attention.

0.11.9
------
* Follow chef RFC041. Newer chef-clients and knifes (knives?) were breaking
  goiardi wasn't following this RFC by reporting the supported api versions.
  (Thanks julian7 for the PR and bringing it to my attention.)
* Remove dependency on golang.org/x/exp/utf8string - it's failing some tests on
  the Debian build servers, and goiardi's liable to get kicked out of testing
  shortly if it isn't addressed. Since goiardi wasn't actually using utf8string
  for anything real complicated, it was easy enough to tear that out and whip
  up a replacement with built-in functions. (Thanks jordi for bringing this
  situation to my attention as well.)

It’s been lurking in the ocean for a long time, waiting on both me finding the time to bring it all together, to make the release once it was ready, and then ironing out the transition to using circleci 2.0 to build the packages, but goiardi 0.11.8 - Coelacanth is now available.

There’s not a ton new in this release, but it does bring some depsolver tweaks in that bring goiardi’s depsolver closer to Chef Server’s behavior. The depsolver changes don’t change the actual behavor of cookbook dependency resolution to any great degree, but it’s a bit of an improvement regardless. Along with that, goiardi now purges old sandboxes once they’re done and have been sitting around for a while. This will at least free up a little space.

Packages can be found at the usual places at packagecloud.io and in the homebrew tap, along with the Debian unstable (as of this writing, it should be in testing soon) and Ubuntu universe distro repositories.

From the CHANGELOG:

* Made some small tweaks and updates to the depsolver to make that a little
  better. NB: Down the road, there may be some further changes to the depsolver,
  especially where 'most constrained' cookbooks are concerned.
* Update the circleci config to use the version 2.0 syntax.
* Add purging old sandboxes that have been hanging around for too long.

This will also be the last release that has support for Debian wheezy – LTS support for wheezy has been dropped, so it seems like a good time to drop it from the goiardi build process. The goiardi SysV init script will be kept available somewhere though in case it’s needed.

Chef 12+ support’s had some progress recently. As has been the case for a long time, the hold up is dealing with all the weird corner cases with RBAC permissions. Once that actually works, very basic Chef 12+ support should fall into place. Unfortunately it’s taken such a long time that there will need to be more time spent getting it back up to speed with more recent developments.

Die Wunde! Die Wunde!

Another bugfix release that took way too long to get done, but various sorts of Real Life™ things kept popping up. The goiardi 0.11.7 - Amfortas release has some pretty serious memory usage improvements, along with some search fixes, more control over logging type information that gets stored, and more.

Packages are up on packagecloud.io and in the homebrew tap, and will be moving into Debian sid and testing shortly. New docker images on Docker Hub (along with some other goodies that have been languishing) should be coming soon as well.

From the CHANGELOG:

* Allow access to /debug/pprof with a whitelist of IP addresses
* Properly index arrays of hashes, arrays of arrays, etc. in object attributes.
* Pretty serious memory usage improvements with search (both the in-memory
  and postgres searches).
* Fix reconnecting to serf if the connection is somehow interrupted.
* Fix negated range queries (it turns out they *do* have a use after all), and 
  refactor how NOT queries are handled generally.
* Add options to purge old reports and node statuses.
* Add option to skip logging extended object information in the event log.
* A handful of other bugfixes.
* Bump up to using golang 1.9.3 for builds.
* Minor changes to the documentation.

On a frequently said note, I need to get a move on with Chef 12+ support - recent Chef clients no longer work with Chef 11 compatible servers. Eeeek!

Spookay!

This is another release in the ever increasing series of small bugfix releases in the 0.11.x series. The only changes in goiardi 0.11.5 - Spookay! are some fixes for certain somewhat obscure search query cases. Hopefully now I’ll get moving more on the long promised 1.0.0.

From the CHANGELOG, such as it is:

* Several search fixes:
  - With postgres search:
    * Fixed reindexing after it broke with the previous update that eliminated
    a lot of unneeded extra rows in the database.
    * Fixed basic queries with NOT statements.
    * Separately, fixed using NOT with subqueries. On a somewhat complicated
      note, but in a way that appears to match standard Solr behavior, when
      doing a query like "name:chef* AND NOT (admin:true OR admin:bleh)" it
      works as is, but when a negated subquery is followed by another basic
      query statement, it needs to have extra parentheses around the NOT +
      subquery, like "name:chef* AND (NOT (admin:true OR admin:bleh)) AND
      public_key:*". A convoluted and unlikely scenario, but it could happen.
  - With in-memory search:
    * NOT + subqueries was also broken with the in-mem search. The fixes for
      the pg-search partially fixed it for in-mem in that it no longer made the
      server panic, but it was returning incorrect results. Additional work
      ended up being needed for in-mem search.

The most just and noble of all temperature scales, only approached in excellence by the quiet majesty of Rankine.

The new goiardi 0.11.4 - Fahrenheit Forever considers 23º to be absolutely freezing, and 70º to be a nice warm day.

Another release that doesn’t change a whole lot that’s very obvious to the end user, but is mostly aimed at the future. The only change likely to matter a great deal to most people is the fact that goiardi now supports Chef auth v1.3, in addition to 1.0, 1.1, and 1.2 (which never really got off the ground for whatever reason). The thing with HEAD requests could be very handy, once there’s client support; the way goiardi implements it, the HEAD request makes much lighter database queries than the GET does, so doing things like checking for the existence of a particular node could be done without consuming as many resources.

One more item of note: since goiardi is using contexts now, it does require golang 1.7 or greater. As noted in the CHANGELOG, goiardi already technically only supported 1.7+, but it would probably build with older versions. Now, that condition is explicit.

Packages are in the usual places, and 0.11.4 is already in sid.

From the CHANGELOG:

* Implement Chef authentication version 1.3.
* Move the custom goiardi error type out of util and into its own module.
  Wrappers around the new module are in util still for convenience, and
  because the functions and interface are used all over the place.
* Many endpoints now handle HEAD requests where appropriate. With some
  endpoints this is not especially useful, but with others it's a lightweight
  way to see what resources exist and so forth. Implements Chef RFC 090.
* Start using contexts with requests. This does mean that goiardi will require
  at least go 1.7. (As of 0.11.3 goiardi only supported go 1.7+, but it was
  likely to build with somewhat older versions anyway.)
* Minor bugfixes - deal with a possible race condition with the in-mem search
  index, change some logging statements from Info to Debug that didn't need to
  be Info level and removed a test log statement that was no longer necessary,
  updated copyright dates.
* Add the Chef API version header to responses.
* Change behavior if the data file and use-(mysql|postgresql) are specified
  together; formerly it was a fatal error, but now it'll just emit a warning
  in the error log and ignore the data file setting.

This is a goiardi 0.11.3 release. All programmerdwarfship is of, um, some sort of quality. It is encircled with bands of cruft. It is made from raw golang source files. This object menaces with spikes of yak hair. On the item is an image of a goiardi 0.11.3 release. On the item is an image of a programmer in sphalerite. The programmer is making a plaintive gesture.

The goiardi 0.11.3 - Menaces with Spikes has come! A housekeeping and bugfix release, mostly. It has command line options. Beware its documentation!

There are bugfixes in this release, one of which is fairly important. If an object being indexed for search had arrays with duplicate items, the in-memory trie index would crash goiardi when trying to index it. (This not only didn’t come up hardly ever, but it was really hard to track down the actual source of the crash.)

There’s new command line flags, so you can configure MySQL and PostgreSQL connections without having to use the config file. Another new development which might be interesting is that now you can configure many of those command line flags with environment variables. This makes running goiardi in Docker easier, since you can just pass in values that way rather than having to mount your own config file to the image, built a new image of your own from scratch, or such and such.

The optional index value length trimming that’s new with this release should be able to save some memory, if your roles, nodes, environments, or data bags have lots of very long values in them. Right now it’s disabled by default, but in the next major (or minor - basically whatever ends up not being 0.11.x; the question is if that will be 0.12.0 or 1.0.0) release the trimming will be on by default. Testing has shown that trimming those indexed values to 100 characters works well, but 50 characters is too short. At 50 characters, some chef-pedant search tests were breaking.

Aside from all that, most of the rest is documentation housekeeping and small changes to packaging. As always, binaries for a variety of platforms are available on the release page, and packages for Debian wheezy, jessie, and stretch, Ubuntu trusty, xenial, and yakkety, and CentOS 6 & 7 are available for various combinations of amd64, arm, and s390x architectures at the goiardi packagecloud.io repo.

From the CHANGELOG:

* Add an option to trim values in search indexes. Currently not enabled by
  default, but will be in the next minor goiardi release (so, either 0.12.0 or
  1.0.0, depending on which ends up being next). Existing indexes ought to be
  reindexed upon upgrading, but they should still work if this is skipped.
* Fix a bug where duplicated items in slices in objects being indexed with the
  in-memory trie based index would cause goiardi to crash. For good measure,
  even though it isn't necessary to prevent a crash remove those same
  duplicate items from objects being indexed with the postgres index.
* Mark --use-unsafe-mem-store as deprecated. In the unlikely event someone's
  using that option, a warning will print in the log. This option may be
  removed at any time.
* Allow setting configuration options via environment variables. (See
  the documentation for the details.)
* Finally allow configuring MySQL or PostgreSQL connection options with
  command line flags (or, now, environment variables).
* Fixed format issues and wording in a few places in the documentation, along
  with updating the docs for the current version.
* Add a hidden flag to generate a simple man page.
* Add that simple man page, along with the html docs, to the packagecloud.io
  packages.
* Add a Dockerfile to allow running the local goiardi source in docker.
* Add Debian "stretch" and Ubuntu "yakkety yak" to the distro versions we have
  in the package repository.

Flying on homemade wings (but not doing anything unseemly for revenge), goiardi version 0.11.2 - Wieland der Schmied is out.

Not much here this time. There was a PR with a fix for escaped characters in certain searches that I wanted to get in before the stretch window closes and that totally freezes, and while I was doing that I decided to take the opportunity to add a build tag to allow a package maintainer to optionally not compile goiardi with support for storing secrets in vault.

Packages are in the usual place on packagecloud.io and in the homebrew tap, and as per the general case some binaries are also on the release page.

From the CHANGELOG:

* Fix a bug with escaped characters in certain searches (thanks ickymettle).
  Does require rebuilding the search index.
* Allow using 'novault' as a build tag to avoid having to have the vault api
  present when building goiardi. Not relevant to most people.

Photo by DiBgd.

Remember, dimetrodons are not dinosaurs. They may be majestic and awesome, but as synapsids they’re actually more closely related to you than to any reptile, dinosaur or not.

The new goiardi 0.11.1 - Dimetrodon release is a relatively minor one, but it has some features I wanted to get out before stretch freezes for real.

Performance-wise, the most important change is a refactoring of the reindexing process with search. Now, it’s broken up into smaller chunks to make it more manageable, and only one reindexing job can run at a time. This avoids the problem where multiple reindexing jobs could be issued in quick succession, and it could happen that none of them would end up complete and the index would be in a bad state.

This release also lets you store secrets like public keys in an external service. Currently, the only supported service is vault. As of this writing client and user public keys, the shovey signing key, and user password hashes can be stored in vault instead of the database. The glaring omission right now is that the SSL certificate and keys for when goiardi itself is using TLS for HTTP connections are still stored as files on disk. That will be dealt with eventually, although having nginx sitting in front of goiardi and handling the TLS related duties is worth considering anyway.

From the CHANGELOG:

* Allow storing secrets (client & user public keys, shovey signing private
  keys, and user password hashes) in an external service. Currently only vault
  is supported.
* Rework reindexing to break it into smaller chunks and ensure that only one
  reindexing job can run at a time.
* Package goiardi for RHEL 7 and Debian jessie for s390x. Rather experimental,
  of course.

After years of extensive dwarven research, scientists have concluded that no, pig iron is not related to pigs in any way, and is therefore impossible to craft into ☼steel bacon☼.

It’s still not orgs and whatnot, but after a long simmering process goiardi 0.11.0 - Steel Bacon is now available. The fixes in this release were important enough to release before 1.0.0 is ready.

From the CHANGELOG:

* Ability to upload cookbooks to S3.
* Add script to upload local files to S3 to migrate.
* Change how items are indexed with the postgres indexer, to reduce the number of rows in the search_items table substantially (at the cost of possible differences in search results in a few weird corner cases).
* Search parser no longer chokes on Unicode. Unfortunately Postgres' ltree module does not accept all Unicode alphanumeric characters as valid still.
* Use vendoring.
* Rejigger the package building process a bit - changing how the different packages are built and how version numbers are determined.
* Fix a long-standing annoyance where the log file would get truncated when goiardi started or restarted.
* Allow passing environment variables to goiardi through the config file.
* Fix in-memory indexer to work with go 1.7.
* Add packages for CentOS 6 and 7. Also use a gox fork pulling in someone's PR with better ARM support until that gets merged upstream eventually.
* Change the postgres columns using the 'json' data type to use 'jsonb' instead. This is generally better, but does mean that goiardi now requires PostgreSQL 9.4 or later.

Packages are available at https://packagecloud.io/ct/goiardi, and individual binaries for a variety of architectures can be found on the release page.

The most important parts of this release are being able to upload files to AWS S3 (or compatible services), which you can read more about in the s3 docs, and the improvements with the Postgres search. These search changes reduce the number of rows in the search table by roughly half, at the expense of slightly more complex SQL queries behind the scenes. Upgrades to 0.11.0 from earlier versions of goiardi using the Postgres search will need to reindex the search tables by running knife index rebuild -y.

In any circumstance, earlier versions of goiardi using a version of Postgres older than 9.4 will need to upgrade postgres before updating goiardi and applying the sqitch patches.

This release also brings goiardi binaries for Linux on s390x (a.k.a z/Arch) (because go 1.7 will compile binaries for s390x now). Said binaries appear to work, so the next minor version of goiardi will include Debian and RHEL based packages for that architecture. This way, goiardi will obviously be Ready For The Enterprise™. Binaries for VMS will have to wait until such time as goiardi builds again with gccgo, sadly.

¡Atencíon! 0, 10, 4.

It’s been a while, but there’s a goiardi 0.10.4 release out. Binaries and packages are at the usual places.

The big thing in this release is adding statsd metrication to goiardi. Otherwise, there’s a bug fix, and exporting pprof info over HTTP. Also, goiardi’s been confirmed to work with go 1.6.

The statsd thing is pretty interesting though, because you can use it to send information about goiardi to graphite (or somesuch), and then visualize that information with grafana (or somesuch, if you don’t appreciate shameless plugs for one’s employer). There’s something on the backburner to make using grafana with goiardi even more interesting; more on that later.

From the CHANGELOG:

* Export pprof info over HTTP, but only accept connections from localhost for that information.

* Add statsd metrics for things like chef-client run timings (requires reporting) and started/succeeded/failed, number of nodes, API endpoint timings, various pieces of runtime info like GC pauses, RAM used, and number of resources updated & total resources for client runs.

* Fix JSON decoding issue where very large numbers would suddenly turn into floats.

In other news, the languishing 1.0.0-dev branch has been synced back up with master as of 0.10.3 and is slowly coming along again. There is likely to be at least an 0.11.0 release though, with some features I’d like to get in before 1.0.0.

This release ended up sitting for a while waiting for me to have time to get it all pulled together, but since there ended up being some extra issues to fix it ended up working out. Goiardi version 0.10.3 “The Fatal Mistake” is now available.

This was supposed to be 0.10.2, but after I had cut the release I found while trying to build a Windows binary with gox that the new version of the tideland logging library, while it was supposed to let OSes without syslog still be able to use the library, was causing gox to blow up. While it should have just worked, I decided to take matters into my own hands and use build tags in goiardi itself to deal with the matter. Of course getting 0.10.3 into shape itself proved to be a bit of a hassle, but it’s ironed out now. In any case, 0.10.2 was skipped.

Binaries are available on the release page, and packages for Debian wheezy and jessie, Ubuntu 14.04, and Raspbian are at the goiardi packagecloud repo.

From the CHANGELOG for both 0.10.2 and 0.10.3:

0.10.3
------
* Handle someone trying to use syslog on Windows ourselves, rather than
  letting the logging library do it (it was causing trouble with gox).

0.10.2
------
* Fix up packaging and deploy scripts a bit
* Add sql schemas to the deb
* Fixed a logic error when configuring the address to listen on where the
  value specified in the config file was always ignored, and only an address
  specified on the command line worked. (Thanks to jordi and DQEbert here for
  bringing this to my attention.)
* Added options to specify proxy hostname and port different than what goairdi
  itself is listening on. (Thanks to jordi and DQEbert here as well.)
* CoC
* Added Debian wheezy to the list of distros we generate packages for.
* The logging library goiardi used moved. It had been forked, but since the
  dependencies of said fork also moved, goiardi switched to the new version of
  that library. Happily the logger library had added logging to syslog as an
  option, so we just went back to using upstream at the new location.  (Thanks
  to theckman for providing a fix for this.)
* In concert with the above, add a "fatal" log level.
* Terraform removed the depgraph module, so that's been vendored into goiardi
  along with its digraph dependency.

Ah, Tosca, you will pay for this release most dearly!

The new goiardi version 0.10.1 is just another bugfix release, but there’s a lot of stuff in here. It should all lead to an all around smoother goiardi experience.

Binaries are available on the release page, and packages for Debian wheezy, Ubuntu 14.04, and Raspbian are at the goiardi packagecloud repo.

From the CHANGELOG:

* Fix some tests
* Scripts, configuration files for more efficient packaging
* circleci integration
* Bomb on importing data if public keys don't validate. (thanks jordi and 
  DQEbert for bringing this to my attention.)
* Validate older PKCS#1 keys -- golang's stdlib pukes on them without some
  massaging. (thanks jordi and DQEbert for bringing this to my attention.)
* Fix reindexing - databags were not being reindexed with the postgres search,
  and the SaveItem calls were moved to goroutines; otherwise, the request
  from knife would time out and knife would restart the reindex.
* Allow '.' in cookbook names; despite what an error chef-pedant is looking
  for, those are allowed. (thanks jordi and DQEbert for bringing this to my 
  attention.)
* Make the authentication lib more general (thanks theckman)
* Output the version of golang used to build a particular goiardi binary
  (again, thanks theckman)
* The changed hostname in URLs to download bug didn't get fixed in 0.7.1 quite
  all the way after all. It is now. (Thanks to oker1 for bringing that to my
  attention.)
* Fixed search tests to pass when run using more than one processor. (Brought
  to my attention by theckman.)
* Fixed a deadlock that could happen when saving an in-mem index to disk at
  the exact moment an object was being indexed. Seems to be specific to
  go1.5.1 (or at least it never happened before that I saw), but needs fixed
  anyway. (Also brought to my attention by theckman.)
* Fixed broken pipe errors with too large requests when running chef-pedant
  against goiardi built with go 1.5.1.
* Update some docs.

Next week is Chef Summit. If you’re reading this and are attending, say “hi”.

“Taihun” was the Gothic word for “ten”. For whatever reason all of the surviving Germanic languages have lost that ‘h’ sound in the middle of the word “ten”, but Gothic preserved it. You can also see some of the usual Germanic sound changes in that word, where a ’d’ sound became a ’t’, and at least some ‘k’ sounds became an ‘h’ sound. Compare Latin decem (bear in mind that the ‘c’ there had a ‘k’ sound originally). You see that sound change as well with “hundred” (compare Latin centum (same thing with the ‘c’)). It also shows a lack of satemization - compare the Russian де́сять (desyat’).

I am pleased to present goiardi version 0.10.0 - Taihun. This release brings the long discussed PostgreSQL based search, along with a search bug fix and some behind-the-scenes changes that won’t impact anyone directly.

From the CHANGELOG:

* Search architecture changed so different search backends can be used (thanks
  oker1 for your work on that).
* Postgres search is here at last! If you're using Postgres, instead of using
  the ersatz solr search, you can instead use Postgres to power your searches.
* Add a mutex for the original goiardi search - multiple simple queries
  executing simultaneously are not a problem, but multiple complex queries can 
  eat up all the RAM on the machine and cause goiardi to crash. This mitigates
  that situation.
* Be a little more forgiving with reporting protocol versions - allow
  specifying the protocol version as a query param instead of only as a
  header. This is to make showing reports with the webui a little easier.
* Bump the Chef Server version we claim to be from 11.1.6 to 11.1.7.

The Postgres search has come a long way since it went into preview. Performance with 10,000 fauxhai generated nodes is a lot more reasonable now, and it appears to be stable and behaving well. If it isn’t, of course, I’d like to hear about it. There is more detailed documentation about the new postgres search available, if you’re interested. The postgres search preview post also has useful information that’s still relevant, if you haven’t seen it yet.

For the future, once 1.0.0 is finished, I also want to make a standalone version of this search à la universe that could work with Chef Server. This would require some changes in erchef to populate the tables, though.

As mentioned in the release notes:

No matter what, whether you’re using the new postgres search or not, if you’re upgrading to this release you’ll need to rebuild your search index. If you’re using the old search, delete your index file first. Then, whether you’re using the old trie index or the new postgres based search, run knife index rebuild -y.

At last, the little issues that kept presenting themselves should all be dealt with. Development on 1.0.0, bringing orgs+RBAC to goiardi, can restart and hopefully be finished sooner rather than later.

This is a small bugfix release before something really cool comes along. This goiardi 0.9.2 - Lithobrake release addresses an issue that @julsevern mentioned he noticed on Twitter, where importing goiardi JSON dumps with reports was breaking because it was exporting reports with a nodeName tag, but import expected a node_name tag.

It turned out that when I brought goiardi in line with golang coding conventions with variable names I inadvertently changed the JSON tag for node names in reports from node_name to nodeName and never noticed. This release fixes that issue. Goiardi will import JSON dumps with either version of the node name tag.

Photo by Sailko

Arthropleura was a eight and a half foot long millipede that dwelled in the Carboniferous period 340 to 280 million years ago, as well as in your dreams tonight.

It’s been a few months, but goiardi 0.9.1 - Arthropleura is out now. As it happens ChefConf is going on right now. This release wasn’t actually timed to come out at the same time as ChefConf, and I wasn’t able to make it there this year, but that’s how it worked out.

This is yet another interim release addressing some issues that were brought to my attention over January and February that then got put on the back burner for a bit while I was on vacation, busy with real work, and just taking a bit of a goiardi break after I was getting kind of burned out from cranking on it too hard for too long. It slowed down goiardi development for a while, but I was getting pretty useless for a while there anyway.

From the CHANGELOG:

* Fix error where requests for zero byte cookbook files would crash.
* Authentication docs improvments (thanks oker1!)
* Rewritten and more robust cookbook depsolver.
* Fix for client creation with cheffish (thanks whiteley!)
* Fix for search where searching for something like "foo:bar AND NOT foo:bar"
  was returning incorrect results. (brought to my attention and test provided
  by brimstone, thanks!)
* Fixed a bug where clients could be created with the same name as a user (or
  vice versa) in in-memory mode.

This release has quite a few contributions from others. Thanks go out to oker1, whiteley, and brimstone on github for their pull requests and bringing issues to my attention.

Coming up, I’m planning on at least one more release before 1.0.0 comes out, which will bring more search improvements to finally deal with search performance when there are thousands of nodes. The current ersatz solr implementation works fine for smaller installations, but once there’s a lot of content to be indexed the RAM and speed become problems still. After that these new changes will need to be merged into 1.0.0 and then that branch will be wrapped up. When I left off that branch’s development it was coming along nicely, but still had areas that needed tightened up to fit standard Chef server behavior more closely.

The Spaceship-and-Sun, of course, is the sign of the Galactic Empire, and was once the symbol of order and prosperity from one end of the galaxy to the other. In other news, I re-read the original Foundation trilogy this past week.

This is another interim release while work chugs along on the 1.0.0-dev branch to get goiardi up to speed with Chef 12. That’s coming along, but I decided to address some issues that had been bothering me for a long time before finishing with that, so I’m happy to release goiardi 0.9.0 - Spaceship and Sun.

From the CHANGELOG:

* Validate IP address supplied on the command line or in the config file.
* Compress index docs to reduce memory usage with the search index.
* Ordering searches works now.
* Index and datastore files now only write to disk if there have been changes
  since the last time they were saved.
* In tandem with the previous change, freeze interval default has been changed
  from 300 seconds to 10 seconds.
* Bump Chef Server version we claim to be from 11.1.3 to 11.1.6.

Normally a release like this would not require a minor version bump, but it does change the format of the index file. Upgrading that is extremely easy, though: before starting the upgraded goiardi up, delete the index file. After goiardi has started, rebuild the index file with knife index rebuild. The new index format uses much less RAM - testing has shown the new index using between a quarter to a third as much RAM as before. Search results are also now ordered in a sensible fashion.

As mentioned in the release notes, if you are upgrading goiardi from source you will need to run go get -u github.com/ctdk/goiardi - the go-trie package has been updated. Goiardi now requires at least go 1.3 to build, and go 1.4 is recommended.

This is a minor release fixing a couple of small issues that popped up that seemed important enough that they probably shouldn’t wait until 1.0.0 comes out.

Goiardi version 0.8.2 fixes a couple of typos, one with checking errors when checking for the existence of a client in SQL mode, and one for the option to use postgres in the sample config file. The use-postgresql option was documented correctly elsewhere, but was listed incorrectly as use-postgres in the sample config file. This release also adds some checks to make sure the directory given for the local filestore exists, and is a directory.

Binaries are up on the release page, as usual.

In ancient Mesopotamia, diseases were often referred to as being the “hand” of some god. The hand of Nergal was a plague of some kind, while the hand of Ishtar was some kind of STD. Ishtar is generally identified with the Greek Aphrodite and Roman Venus as a goddess of love (although she was by no means identical to Aphrodite and Venus), so the name’s quite appropriate. The word “venereal” ultimately derives from “Venus”, after all.

These are two minor releases. Goiardi version 0.8.1 moves the bulk of the documentation from the README to readthedocs, where it should be much easier to read and work on, fixes a small issue with the postgres SQL dump file, and disables SSLv3. Schob 0.1.1 adds a fix to allow building schob on 32 bit platforms and on Windows, and updates links to the goiardi shovey documentation.

In honor of the upcoming Chef Summit, there are new releases from goiardi and friends. This officially brings out shovey, the goiardi push jobs facility, to the world.

Because of the relative complexity of shovey, there are actually three releases here:

• goiardi v0.8.0 - Hojotoho! Hojotoho!
• schob v0.1.0 - Heiaha! Heiaha!
• knife-shove v0.6.0 - Helmwige! Hier!

Astute readers may recognize the releases’ codenames as the opening lines of Act III of Die Walküre.

Shovey’s included with goiardi. Schob is the shovey client that gets installed on the nodes that will run jobs, and knife-shove is a knife plugin forked from knife-push to manage shovey jobs. The goiardi shovey docs, the schob README, and the knife-shove README have a bunch of documentation on how to use shovey.

Nice, functional cookbooks for shovey and goiardi are also coming. There is a shovey-jobs cookbook already, but it only supports Debian. Other platforms are being added, and will be ready soon.

Also coming reasonably soon are organizations and RBAC (it’s finally time), and revamping the documentation. The README has gotten too long and unweildy. It will be split apart, reorganized, and expanded where needed, while the godocs for goiardi will stop being a mirror of the README and focus more on development and internals.

See you at the Summit!

Goiardi version 0.7.2 is a minor yak shaving release that there isn’t a lot to say about. The CHANGELOG pretty much sums it up:

• Remove a newline in a debug statement, courtesy of @spheromak.
• Also per @spheromak’s suggestion, fixed some possible race conditions revealed by building goiardi with the -race flag and running chef-pedant against it.
• Edit doc.go slightly to make godocs more attractive.

Elsewhere, shovey work continues and comes closer to completion.

The media refuses to report on this scourge afflicting the Florida coast, which is why you never hear about it.

“Shovey” is chugging along on a fast pace, but in the meantime a couple of issues came up that I thought deserved attention. Goiardi version 0.7.1 is a minor release, addressing two relatively small issues.

First, this release adds options for configuring the db pool size and max number of connections. It was happening automagically before, but I was always a bit uneasy about that. Until last night, though, I kept missing how to set that in the documentation for database/sql.

Second, for cookbook files stored locally (which is currently all of them), goiardi will now set the URL of the resource to use the current configured hostname. This was brought to my attention because someone had uploaded a bunch of cookbooks, changed the hostname and port and turned on SSL, and had the URLs break. It takes a specific chain of events to see the problem, which is why it had gone unnoticed up to this point.

From the CHANGELOG:

• Add –db-pool-size and –max-connections options for configuring the number of idle db connections kept around and the maximum number of db connections to make to the server. It isn’t particularly useful if you’re not using one of the SQL backends.
• For locally stored cookbook files (which is currently all of them), goiardi now generates the URL to the resource from the currently configured hostname. This fixes an issue where if you uploaded a cookbook and then changed the goairdi server’s hostname, the URLs to download cookbooks would break.

Other than that, there’s nothing else new in this release.

In case you haven’t read it, Orphans of the Sky is a novel by Robert A. Heinlein about a generation ship floating through space where the people on board have forgotten that they are on a space ship. It’s pretty good, but more importantly the name of the first of the two novellas that make up the novel is titled “Universe”. The title of the second part, “Common Sense”, became relevant after I really dug into making this release for reasons that will soon become clear.

While it ended up being a bigger deal than I expected it to be, I am pleased to announce the release of goiardi version 0.7.0.

I took what I thought would be a brief detour from my current work on the serf integration and “shovey” stuff for goiardi to implement Chef RFC 014 real quick like, which I had been considering doing even before the RFC came along. I had been planning on making this a small release numbered 0.6.1, but then after I proposed a mechanism for caching the universe endpoint it was suggested in the RFC discussion that I see how goiardi performed with the full load of cookbooks in the supermarket before jumping the gun and adding caching before it was necessarily needed.

It turned out that will over 6,000 cookbooks the universe endpoint ran like a dog. Ultimately the problem boiled down to using gob to encode the complex data structures in cookbooks, nodes, etc. in the database. Generally gob encoding in golang is faster than using JSON, but in this particular case JSON is actually faster. The “common sense” thing to do was to change how those complex structures are stored in the database, even though it’s a bit of a brutal change. Sometimes, though, you have to bite the bullet and do the thing that sucks. It’s ended up working nicely, however; the postgres implementation is now able to make use of the json type and the json functions built into Postgres. Before switching from gob to json, serving /universe with over 6200 cookbooks took over 3 seconds with Postgres or MySQL, but afterwards Postgres can serve it in roughly 325 milliseconds. MySQL isn’t able to take advantage of the Postgres-specific json functionality, but it still went down to 1 second, so that’s a marked improvement. Using the in-memory (with or without file backing) takes about 1.2 seconds. I’m willing to take this; if you have that many cookbooks and can’t wait a second for /universe, you may want to use Postgres for your data store.

From the CHANGELOG:

• Add /universe API endpoint, per https://github.com/opscode/chef-rfc/blob/master/rfc014-universe-endpoint.md.
• Make file uploading a little more forgiving.
• Make validating some cookbook metadata more forgiving, to bring goiardi’s validations in line with erchef.
• Added some functions to make listing all cookbooks and recipes on the server faster and move the logic into the cookbook package.

• Breaking DB change: with both MySQL and Postgres, the way data structures for cookbooks, nodes, etc. has changed from gob encoding to using JSON. This obviously breaks existing items in the database, so the following steps must be followed by users using either SQL backend for data storage:

  • Export their goiardi server’s data with the -x flag.
  • Either revert all changes to the db with sqitch, then redeploy, or drop the database manually and recreate it from either the sqitch patches or the full table dump of the release (provided starting with 0.7.0)
  • Reload the goiardi data with the -m flag. See the README or the godocs for more information.

More documentation on the universe endpoint in goiardi is available in the README. Specific information on upgrading is also in the README in the UPGRADING section.

Back to the serf/shovey stuff, then.

The Order of the Elephant is a Danish order awarded mostly to foreign heads of state and royals. Its elephant-themed insignia is appropriate for this latest release of goiardi, version 0.6.0.

Along with some bug fixes and smaller improvements, this release introduces Postgres as a supported database backend. The schema is not compatible with erchef’s schema, but is close to the goiardi MySQL schema (with some differences of course). This is the first project I’ve done with Postgres, so comments on how goiardi is using it are welcome.

Full CHANGELOG:

• Postgres support.
• Fix rebuilding indexes with an SQL backend.
• Fix a bug where in MySQL mode events were being logged twice.
• Fix an annoying chef-pedant error with data bags.
• Event logging methods that are not allowed now return Method Not Allowed rather than Bad Request.
• Switch the logger to a fork that can be built and used with Windows that excludes syslog when building on Windows.
• Add basic syslog support.
• Authentication protocol version 1.2 now supported.
• Add a ‘status’ param to reporting, so a list of reports return by ‘knife runs’ can be narrowed by the status of the chef run (started, success, and failure).
• Fix an action at a distance problem with in-memory mode objects. If this behavior is still desirable (it seems to be slightly faster than the new way), it can be turned back on with the –use-unsafe-mem-store flag. This change DEFINITELY breaks in-mem data file compatibility. If upgrading, export your data, upgrade goiardi, and reload your data.
• Add several new searchable parameters for logged events.
• Add organization_id to all MySQL tables that might need it someday. Orgs are not used at all, so only the default value of 1 currently makes it to the database.
• Finally ran ‘go fmt’ on goiardi. It didn’t even mess up the long comment blocks, which was what I was afraid it would do. I also ran golint against goiardi and took its recommendations where it made sense, which was most areas except for some involving generated parser code, comments on GobEncode/Decode, commenting a bunch of identical functions on an interface in search, and a couple of cases involving make and slices. All in all, though, the reformatting, linting, and light refactoring has done it good.

To reiterate, this update breaks saved data file compatibility. You’ll need to export and re-import your data if you’re using the persistent in-memory data store.

A selection of precompiled binaries are provided on the release page, including Windows and Solaris builds. There are also knife plugins for goiardi’s reporting and event logging capabilities at knife-goiardi-reporting and knife-goiardi-event-log. The reporting plugin was forked from the official Chef knife-reporting plugin to add support for the “status” parameter. Both plugins are also available on rubygems.

Finally, after someone asked how goiardi would handle running ~500 nodes, I decided to take a look and see. I whipped up a little test script to create a thousand nodes and clients to see how goiardi (and more importantly the search) would handle it. I wasn’t particularly worried about the non-search portions of goiardi (1000 rows on a table isn’t very much at all, after all), but I hadn’t tested search extensively with large amounts of data. After creating those 1,000 nodes and clients, goiardi was using about 1.40 GB of RAM. An earlier test with 5,500 nodes (but no clients) used 6.45GB RAM. The numbers show that there’s definitely room for improvement with the search, in that the capability for indexing nodes and data bags is limited by RAM, but performance was good even at 5,500 nodes. More rigorous testing is needed, however. Search is an area I intend to revisit shortly to improve these problems.

The next issues to work on with goiardi are currently improving search to work even better with large amounts of data and use raft (or something similar) to allow multiple goiardi instances to keep their search indexes consistent, reworking the permissions system, being able to upload files to s3 or similar storage providers, and the serf based pushy type thing.

Between the previous goiardi release and this one, my kids discovered Minecraft. This means that I started playing Minecraft, which ended up slowing goiardi development down for a while.

This blocky adversary has been overcome, however, and goiardi version 0.5.2 is out.

Originally the next goiardi release was going to add Postgres support. This was about halfway done, and will be resumed shortly, but I decided that it was more important to add the ability to import and export data before any of the goiardi structs changed.

Changes in this release, from the CHANGELOG:

• Add import/export of goiardi data through a JSON dump.
• Add configuration options to specify the max sizes for objects uploaded to the filestore and for JSON requests from the client.

Importing and exporting is explained in the README and in the godocs. Basically you add the -x/--export flag to your usual goiardi command to export the data to a JSON file, and run the -m/--import flags with your new desired goiardi set of flags to import the data. The administrator is responsible for removing any existing data (on-disk or SQL) and index files. This can help with backups or moving between the different storage engines goiardi supports, keeping data between upgrades, and should make testing easier in the future.

Also, per the README, there are new options to specify the maximum size of an object that can be uploaded to the filestore, and the maximum size of a JSON request from a client. The -Q/--obj-max-size flag gets the filestore object max size, while -j/--json-req-max-size sets the JSON request max size limit.

Completely unrelated to any of my efforts, but while I was preparing this release Go 1.3 was released. Goiardi and its dependencies build just fine with Go 1.3 with no changes. This also means that goiardi should be able run on some of the stranger platforms out there, like Solaris, the illumos based distributions, plan9, and apparently Dragonfly BSD. So far goiardi has only been build on OmniOS r151008j, but it built fine.

Finally, as an added convenience: on the goiardi 0.5.2 release page, precompiled binaries are being provided for download, if you don’t feel like setting up go and compiling them yourself. As of this writing binaries are provided for Linux (built on Debian wheezy), MacOS X (built on 10.9 Mavericks), FreeBSD (built on FreeBSD 9.2), and illumos (built on OmniOS r151008j). The Linux, MacOS X, and FreeBSD binaries are x86_64, the illumos binary i386 update: I misinterpreted the output of psrinfo - it was compiled for 64 bit. It appears that there’s no reason that goiardi can’t easily be cross compiled though, so a more sensible set of precompiled binaries will be coming soon. If you don’t see binaries for your platform, try grabbing a binary that’s built on a similar platform to yours, or compile it yourself.

Coming down the pipeline is resuming and re-merging the Postgres work, making some data structures safer for in-memory mode, some extra features for event logging and reporting, improving tests and moving away from depending on chef-pedant, and an idea I’ve been kicking around for a serf-based pushy type thing. For those curious about the Postgres goiardi schema, you can see it in the ‘postgres’ tree in goiardi-schema. It’s firmed up, but still subject to change before getting merged back in.

The sky darkens. Blotting out the sun, Rodan appears to destroy beloved landmarks and stomp on small models of tanks. Goiardi version 0.5.1 has been released.

This is an incremental release, made up of bug fixes and relatively minor new features. The most interesting new features are event logging and reporting, although the log level changes are also nice.

From the CHANGELOG:

• Add log levels (from debug to critical). This makes -V/–verbose useful.
• Add an easier option in the config file to specify log levels by name.
• ipv6 already worked, but accidentally. Now it works in a more deliberate fashion, preventing mishaps with addresses, colons, and port numbers.
• Authentication protocol version 1.1 now supported.
• Remove a sort on run lists that was there for some reason. I have no idea what it was put there for, but it was wrong.
• Add an event log to log changes to objects like nodes, clients, etc. See the README or godocs for details.
• Add support for reporting (http://docs.opscode.com/reporting.html)

This release should not break old save files, if you’re using that. If you’re using MySQL, you’ll need to deploy the sqitch changes.

The next big things to start working on are the long promised Postgres support, and etcd/serf/consul etc. integration. Groupcache has also been asked for and looks like a worthwhile thing to pick up. As always, the issues page is a good place to look at what’s being worked on.

Goiardi is also starting to grow beyond using chef-pedant for most of its testing; it’s about time to start more native golang tests for goiardi and not use chef-pedant for the main tests. It should always work with chef-pedant, of course, but goiardi’s starting to get more features that chef-pedant doesn’t check.

Stormily agitated, a new goiardi release appears! Goiardi version 0.5.0 has been released.

Aside from a few bug fixes, this release brings MySQL support. Now, in addition to running entirely in memory, goiardi can use MySQL for storing data. The README and godocs explain how to set that up.

From the CHANGELOG:

• MySQL support added
• No longer redirect /environments/NAME/roles/NAME to /roles/NAME/environments/NAME
• Update documentation, reformat godocs
• Split actors apart into separate user and client types, made new Actor interface that encompasses both users and clients.

Note: This release contains one breaking change. Saved data files from previous goiardi releases will not work with 0.5.0, so if you’re in that situation either don’t upgrade or remove your saved index and data files before starting up. This will regenerate your admin, validator, and webui keys, and of course nuke anything you had in place before.

Postgres support has already been requested by a few people, so that should be coming fairly soon. The issues page has been filled in with some enhancements that would be nice for goiardi to have. Feel free to add your own if there’s something you’d like to see.

It seems like only yesterday that there was a goiardi release, but really it was a few days ago. Goiardi version 0.4.2 is hot off the presses and ready for your perusal.

This is a bugfix release. The reason this is the “Spinal Tap” release was because I noticed these problems while I took the day off after getting a spinal tap and was bored, so it seemed fitting. The bug fixes, from the CHANGELOG, are:

• Perm tweak for nodes updating themselves.
• Small change with validating role descriptions when creating or updating from JSON.
• Fix issue with saving complicated indexed objects to disk where improperly flattened indexable objects were making the gob encoder puke all over itself when encoding the tries in the index docs.
• Fixed a possible regression with synchronizing cookbooks that did not show up in testing, but only in real use.
• An absolutely bonkers fix for listing cookbook files with webui. Webui wants all of the cookbook top level attributes sent over with a request to /cookbooks/<name>/<version>, but this is the exact opposite of the behavior chef-pedant wants, where empty definitions, attributes, etc. are not sent over. Knife also seems quite content with this, so the fix for now, since the two cases are mutually exclusionary, is to only send the empty hashes for those top level attributes with a GET if the request is coming from the webui. Bizarre, but it seems to be what’s necessary.

These were all problems that evaded notice during testing, but did pop up when I was using goiardi to set up a vagrant VM to start working on letting goiardi run a real database (optionally, of course) instead of the in-memory data store system it’s using now. The in-memory mode is great and all for testing, and being able to freeze the data to disk for persistence is very handy, but being able to use a real database would be nice for some workloads. Supporting both the current in-memory storage and a database backend is a top priority.

Which database to use is still a bit up in the air, but I’m leaning towards throwing caution to the wind and providing support for both MySQL and Postgres, with SQLite as a distinct possibility down the line. MySQL is likely to come first, but that is not set in stone.

Hot on the heels of the 0.4.0 release comes goiardi version 0.4.1. This release adds a –disable-webui option to disable using chef-webui with goiardi, fixes a bug with parsing configuration file options and rearranges how some of the config struct items are set, fixes a typo in the sample config file, and makes some small documentation tweaks.

Other than that, all the changes announced in the 0.4.0 release announcement and previous releases remain relevant.

A new version of goiardi, version 0.4.0, as been released.

This version fixes a bug with pessimistic matching, adds more tests, improves the documentation with an expanded README and more godocs, and introduces client/user authentication and authorization and SSL for connections as optional settings. It will also run and authenticate properly with the chef webui, if you’re so inclined. The webui is a separate install, however, and does not ship with goiardi.

There is also a fork of chef-pedant customized for goiardi now, to handle some tests with authentication where goiardi and the chef-server make slightly different decisions.

Goiardi still works fine in no-auth mode. The authentication and authorization parts are completely optional, but they’re there if you want or need authentication. It’s recommended that if you do use authentication that you should set the –index-file and –data-file options for persistent data and index storage, but it’s not strictly necessary. See the README for more on how to set up authentication.

SSL is pretty straightforward, and also explained in the README. You’ll need to generate a certificate/key pair for it to work, but that isn’t too hard. This does assume that you’d use a self-signed certificate for this; if you’re going to use a real cert I assume you know what you’re doing (and am a little surprised, to be honest).

That’s it for this release. The next major goiardi task coming up, barring any unexpected changes, is allowing it to use a database for the backend. This will be optional, of course; I plan on supporting the in-memory, no-auth mode a la chef-zero for goiardi indefinitely. Further out, there may be a smoother installation process, and someday even perhaps goiardi-specific features.

I am pleased to announce another goiardi release.

This release, version 0.3.3 of goiardi, doesn’t add much in the way of new functionality. The data store and indexer have been tweaked slightly, and go tests have been added for some of the goiardi components. The go test coverage isn’t complete, and most goiardi testing is handled with chef-pedant, but these golang tests will be handy for testing internal functionality that could break, yet not show up in the chef-pedant testing.

While there isn’t much in this release, I decided to make one here because it’s a good place to make one before starting to tackle authentication. Enjoy.

Update: It’s a little embarrassing, but there was a problem merging branches, and a necessary change got backed out or overlooked, so version 0.3.1 has been superseded by version 0.3.3. On the off chance anyone got the previous version, I apologize.