12 Jul 2017, 07:20

goiardi is in Debian stable

Following the natural progression of this post announcing goiardi being in sid, I’m pleased to note that as of a few weeks ago when Debian stretch was released as stable, packages for goiardi are now available from the stable repo. It was also automatically imported into ubuntu, and can now be found in universe for “The Zesty Zapus” (whatever that is) and “The Artful Aardvark” (which I do know what that is, although I’m not sure how artful they usually are).

Newer versions, of course, are available in testing and unstable, from the goiardi packagecloud repository, or by building from source, but it’s not a bad thing to

15 Jun 2017, 18:28

goiardi version 0.11.5 - Spookay!

Spookay!

This is another release in the ever increasing series of small bugfix releases in the 0.11.x series. The only changes in goiardi 0.11.5 - Spookay! are some fixes for certain somewhat obscure search query cases. Hopefully now I’ll get moving more on the long promised 1.0.0.

From the CHANGELOG, such as it is:

* Several search fixes:
  - With postgres search:
    * Fixed reindexing after it broke with the previous update that eliminated
    a lot of unneeded extra rows in the database.
    * Fixed basic queries with NOT statements.
    * Separately, fixed using NOT with subqueries. On a somewhat complicated
      note, but in a way that appears to match standard Solr behavior, when
      doing a query like "name:chef* AND NOT (admin:true OR admin:bleh)" it
      works as is, but when a negated subquery is followed by another basic
      query statement, it needs to have extra parentheses around the NOT +
      subquery, like "name:chef* AND (NOT (admin:true OR admin:bleh)) AND
      public_key:*". A convoluted and unlikely scenario, but it could happen.
  - With in-memory search:
    * NOT + subqueries was also broken with the in-mem search. The fixes for
      the pg-search partially fixed it for in-mem in that it no longer made the
      server panic, but it was returning incorrect results. Additional work
      ended up being needed for in-mem search.

16 May 2017, 06:55

goiardi version 0.11.4 - Fahrenheit Forever

The most just and noble of all temperature scales, only approached in excellence by the quiet majesty of Rankine.

The new goiardi 0.11.4 - Fahrenheit Forever considers 23º to be absolutely freezing, and 70º to be a nice warm day.

Another release that doesn’t change a whole lot that’s very obvious to the end user, but is mostly aimed at the future. The only change likely to matter a great deal to most people is the fact that goiardi now supports Chef auth v1.3, in addition to 1.0, 1.1, and 1.2 (which never really got off the ground for whatever reason). The thing with HEAD requests could be very handy, once there’s client support; the way goiardi implements it, the HEAD request makes much lighter database queries than the GET does, so doing things like checking for the existence of a particular node could be done without consuming as many resources.

One more item of note: since goiardi is using contexts now, it does require golang 1.7 or greater. As noted in the CHANGELOG, goiardi already technically only supported 1.7+, but it would probably build with older versions. Now, that condition is explicit.

Packages are in the usual places, and 0.11.4 is already in sid.

From the CHANGELOG:

* Implement Chef authentication version 1.3.
* Move the custom goiardi error type out of util and into its own module.
  Wrappers around the new module are in util still for convenience, and
  because the functions and interface are used all over the place.
* Many endpoints now handle HEAD requests where appropriate. With some
  endpoints this is not especially useful, but with others it's a lightweight
  way to see what resources exist and so forth. Implements Chef RFC 090.
* Start using contexts with requests. This does mean that goiardi will require
  at least go 1.7. (As of 0.11.3 goiardi only supported go 1.7+, but it was
  likely to build with somewhat older versions anyway.)
* Minor bugfixes - deal with a possible race condition with the in-mem search
  index, change some logging statements from Info to Debug that didn't need to
  be Info level and removed a test log statement that was no longer necessary,
  updated copyright dates.
* Add the Chef API version header to responses.
* Change behavior if the data file and use-(mysql|postgresql) are specified
  together; formerly it was a fatal error, but now it'll just emit a warning
  in the error log and ignore the data file setting.

24 Mar 2017, 14:25

goiardi version 0.11.3 - Menaces with Spikes
This is a goiardi 0.11.3 release. All programmerdwarfship is of, um, some sort of quality. It is encircled with bands of cruft. It is made from raw golang source files. This object menaces with spikes of yak hair. On the item is an image of a goiardi 0.11.3 release. On the item is an image of a programmer in sphalerite. The programmer is making a plaintive gesture.

The goiardi 0.11.3 - Menaces with Spikes has come! A housekeeping and bugfix release, mostly. It has command line options. Beware its documentation!

There are bugfixes in this release, one of which is fairly important. If an object being indexed for search had arrays with duplicate items, the in-memory trie index would crash goiardi when trying to index it. (This not only didn’t come up hardly ever, but it was really hard to track down the actual source of the crash.)

There’s new command line flags, so you can configure MySQL and PostgreSQL connections without having to use the config file. Another new development which might be interesting is that now you can configure many of those command line flags with environment variables. This makes running goiardi in Docker easier, since you can just pass in values that way rather than having to mount your own config file to the image, built a new image of your own from scratch, or such and such.

The optional index value length trimming that’s new with this release should be able to save some memory, if your roles, nodes, environments, or data bags have lots of very long values in them. Right now it’s disabled by default, but in the next major (or minor - basically whatever ends up not being 0.11.x; the question is if that will be 0.12.0 or 1.0.0) release the trimming will be on by default. Testing has shown that trimming those indexed values to 100 characters works well, but 50 characters is too short. At 50 characters, some chef-pedant search tests were breaking.

Aside from all that, most of the rest is documentation housekeeping and small changes to packaging. As always, binaries for a variety of platforms are available on the release page, and packages for Debian wheezy, jessie, and stretch, Ubuntu trusty, xenial, and yakkety, and CentOS 6 & 7 are available for various combinations of amd64, arm, and s390x architectures at the goiardi packagecloud.io repo.

From the CHANGELOG:

* Add an option to trim values in search indexes. Currently not enabled by
  default, but will be in the next minor goiardi release (so, either 0.12.0 or
  1.0.0, depending on which ends up being next). Existing indexes ought to be
  reindexed upon upgrading, but they should still work if this is skipped.
* Fix a bug where duplicated items in slices in objects being indexed with the
  in-memory trie based index would cause goiardi to crash. For good measure,
  even though it isn't necessary to prevent a crash remove those same
  duplicate items from objects being indexed with the postgres index.
* Mark --use-unsafe-mem-store as deprecated. In the unlikely event someone's
  using that option, a warning will print in the log. This option may be
  removed at any time.
* Allow setting configuration options via environment variables. (See
  the documentation for the details.)
* Finally allow configuring MySQL or PostgreSQL connection options with
  command line flags (or, now, environment variables).
* Fixed format issues and wording in a few places in the documentation, along
  with updating the docs for the current version.
* Add a hidden flag to generate a simple man page.
* Add that simple man page, along with the html docs, to the packagecloud.io
  packages.
* Add a Dockerfile to allow running the local goiardi source in docker.
* Add Debian "stretch" and Ubuntu "yakkety yak" to the distro versions we have
  in the package repository.

16 Jan 2017, 22:57

goiardi version 0.11.2 - Wieland der Schmied

Flying on homemade wings (but not doing anything unseemly for revenge), goiardi version 0.11.2 - Wieland der Schmied is out.

Not much here this time. There was a PR with a fix for escaped characters in certain searches that I wanted to get in before the stretch window closes and that totally freezes, and while I was doing that I decided to take the opportunity to add a build tag to allow a package maintainer to optionally not compile goiardi with support for storing secrets in vault.

Packages are in the usual place on packagecloud.io and in the homebrew tap, and as per the general case some binaries are also on the release page.

From the CHANGELOG:

* Fix a bug with escaped characters in certain searches (thanks ickymettle).
  Does require rebuilding the search index.
* Allow using 'novault' as a build tag to avoid having to have the vault api
  present when building goiardi. Not relevant to most people.

24 Nov 2016, 10:31

goiardi version 0.11.1 - Dimetrodon

A bunch of dimetrodons

Photo by DiBgd.

Remember, dimetrodons are not dinosaurs. They may be majestic and awesome, but as synapsids they’re actually more closely related to you than to any reptile, dinosaur or not.

The new goiardi 0.11.1 - Dimetrodon release is a relatively minor one, but it has some features I wanted to get out before stretch freezes for real.

Performance-wise, the most important change is a refactoring of the reindexing process with search. Now, it’s broken up into smaller chunks to make it more manageable, and only one reindexing job can run at a time. This avoids the problem where multiple reindexing jobs could be issued in quick succession, and it could happen that none of them would end up complete and the index would be in a bad state.

This release also lets you store secrets like public keys in an external service. Currently, the only supported service is vault. As of this writing client and user public keys, the shovey signing key, and user password hashes can be stored in vault instead of the database. The glaring omission right now is that the SSL certificate and keys for when goiardi itself is using TLS for HTTP connections are still stored as files on disk. That will be dealt with eventually, although having nginx sitting in front of goiardi and handling the TLS related duties is worth considering anyway.

From the CHANGELOG:

* Allow storing secrets (client & user public keys, shovey signing private
  keys, and user password hashes) in an external service. Currently only vault
  is supported.
* Rework reindexing to break it into smaller chunks and ensure that only one
  reindexing job can run at a time.
* Package goiardi for RHEL 7 and Debian jessie for s390x. Rather experimental,
  of course.

17 Nov 2016, 10:30

goiardi is in Debian sid now

In what I consider to be pretty cool news, due to the efforts of @JordiMallach goiardi is now in Debian sid. One assumes that ultimately it will get into the next version of Debian, unless something goes terribly wrong.

The packages at https://packagecloud.io/ct/goiardi will still be available, of course, if you choose to use them. The packages there will still be automatically built and uploaded when a new release is cut, but they’re built with fpm rather than having the more standard debian build files all nicely specified. The packagecloud packages also build packages for older versions of debian, in addition to the packages for ubuntu and CentOS, so those packages still have a place.

Whatever version’s in the debian repositories should, at least, be assured of being quite stable, so that’s a point in it’s favor. If, however, it starts feeling fossilized, then the packagecloud package stream will be available.

11 Nov 2016, 12:58

goiardi version 0.11.0 - Steel Bacon

After years of extensive dwarven research, scientists have concluded that no, pig iron is not related to pigs in any way, and is therefore impossible to craft into ☼steel bacon☼.

It’s still not orgs and whatnot, but after a long simmering process goiardi 0.11.0 - Steel Bacon is now available. The fixes in this release were important enough to release before 1.0.0 is ready.

From the CHANGELOG:

* Ability to upload cookbooks to S3.
* Add script to upload local files to S3 to migrate.
* Change how items are indexed with the postgres indexer, to reduce the number of rows in the search_items table substantially (at the cost of possible differences in search results in a few weird corner cases).
* Search parser no longer chokes on Unicode. Unfortunately Postgres' ltree module does not accept all Unicode alphanumeric characters as valid still.
* Use vendoring.
* Rejigger the package building process a bit - changing how the different packages are built and how version numbers are determined.
* Fix a long-standing annoyance where the log file would get truncated when goiardi started or restarted.
* Allow passing environment variables to goiardi through the config file.
* Fix in-memory indexer to work with go 1.7.
* Add packages for CentOS 6 and 7. Also use a gox fork pulling in someone's PR with better ARM support until that gets merged upstream eventually.
* Change the postgres columns using the 'json' data type to use 'jsonb' instead. This is generally better, but does mean that goiardi now requires PostgreSQL 9.4 or later.

Packages are available at https://packagecloud.io/ct/goiardi, and individual binaries for a variety of architectures can be found on the release page.

The most important parts of this release are being able to upload files to AWS S3 (or compatible services), which you can read more about in the s3 docs, and the improvements with the Postgres search. These search changes reduce the number of rows in the search table by roughly half, at the expense of slightly more complex SQL queries behind the scenes. Upgrades to 0.11.0 from earlier versions of goiardi using the Postgres search will need to reindex the search tables by running knife index rebuild -y.

In any circumstance, earlier versions of goiardi using a version of Postgres older than 9.4 will need to upgrade postgres before updating goiardi and applying the sqitch patches.

This release also brings goiardi binaries for Linux on s390x (a.k.a z/Arch) (because go 1.7 will compile binaries for s390x now). Said binaries appear to work, so the next minor version of goiardi will include Debian and RHEL based packages for that architecture. This way, goiardi will obviously be Ready For The Enterprise™. Binaries for VMS will have to wait until such time as goiardi builds again with gccgo, sadly.

30 Mar 2016, 18:50

goiardi version 0.10.4 - Numbers Station

¡Atencíon! 0, 10, 4.

It’s been a while, but there’s a goiardi 0.10.4 release out. Binaries and packages are at the usual places.

The big thing in this release is adding statsd metrication to goiardi. Otherwise, there’s a bug fix, and exporting pprof info over HTTP. Also, goiardi’s been confirmed to work with go 1.6.

The statsd thing is pretty interesting though, because you can use it to send information about goiardi to graphite (or somesuch), and then visualize that information with grafana (or somesuch, if you don’t appreciate shameless plugs for one’s employer). There’s something on the backburner to make using grafana with goiardi even more interesting; more on that later.

From the CHANGELOG:

* Export pprof info over HTTP, but only accept connections from localhost for that information.

* Add statsd metrics for things like chef-client run timings (requires reporting) and started/succeeded/failed, number of nodes, API endpoint timings, various pieces of runtime info like GC pauses, RAM used, and number of resources updated & total resources for client runs.

* Fix JSON decoding issue where very large numbers would suddenly turn into floats.

In other news, the languishing 1.0.0-dev branch has been synced back up with master as of 0.10.3 and is slowly coming along again. There is likely to be at least an 0.11.0 release though, with some features I’d like to get in before 1.0.0.

17 Nov 2015, 13:30

goiardi version 0.10.3 - The Fatal Mistake

This release ended up sitting for a while waiting for me to have time to get it all pulled together, but since there ended up being some extra issues to fix it ended up working out. Goiardi version 0.10.3 “The Fatal Mistake” is now available.

This was supposed to be 0.10.2, but after I had cut the release I found while trying to build a Windows binary with gox that the new version of the tideland logging library, while it was supposed to let OSes without syslog still be able to use the library, was causing gox to blow up. While it should have just worked, I decided to take matters into my own hands and use build tags in goiardi itself to deal with the matter. Of course getting 0.10.3 into shape itself proved to be a bit of a hassle, but it’s ironed out now. In any case, 0.10.2 was skipped.

Binaries are available on the release page, and packages for Debian wheezy and jessie, Ubuntu 14.04, and Raspbian are at the goiardi packagecloud repo.

From the CHANGELOG for both 0.10.2 and 0.10.3:

0.10.3
------
* Handle someone trying to use syslog on Windows ourselves, rather than
  letting the logging library do it (it was causing trouble with gox).

0.10.2
------
* Fix up packaging and deploy scripts a bit
* Add sql schemas to the deb
* Fixed a logic error when configuring the address to listen on where the
  value specified in the config file was always ignored, and only an address
  specified on the command line worked. (Thanks to jordi and DQEbert here for
  bringing this to my attention.)
* Added options to specify proxy hostname and port different than what goairdi
  itself is listening on. (Thanks to jordi and DQEbert here as well.)
* CoC
* Added Debian wheezy to the list of distros we generate packages for.
* The logging library goiardi used moved. It had been forked, but since the
  dependencies of said fork also moved, goiardi switched to the new version of
  that library. Happily the logger library had added logging to syslog as an
  option, so we just went back to using upstream at the new location.  (Thanks
  to theckman for providing a fix for this.)
* In concert with the above, add a "fatal" log level.
* Terraform removed the depgraph module, so that's been vendored into goiardi
  along with its digraph dependency.

07 Oct 2015, 10:36

goiardi version 0.10.1 - Presto, su! Mario!

Ah, Tosca, you will pay for this release most dearly!

The new goiardi version 0.10.1 is just another bugfix release, but there’s a lot of stuff in here. It should all lead to an all around smoother goiardi experience.

Binaries are available on the release page, and packages for Debian wheezy, Ubuntu 14.04, and Raspbian are at the goiardi packagecloud repo.

From the CHANGELOG:

* Fix some tests
* Scripts, configuration files for more efficient packaging
* circleci integration
* Bomb on importing data if public keys don't validate. (thanks jordi and 
  DQEbert for bringing this to my attention.)
* Validate older PKCS#1 keys -- golang's stdlib pukes on them without some
  massaging. (thanks jordi and DQEbert for bringing this to my attention.)
* Fix reindexing - databags were not being reindexed with the postgres search,
  and the SaveItem calls were moved to goroutines; otherwise, the request
  from knife would time out and knife would restart the reindex.
* Allow '.' in cookbook names; despite what an error chef-pedant is looking
  for, those are allowed. (thanks jordi and DQEbert for bringing this to my 
  attention.)
* Make the authentication lib more general (thanks theckman)
* Output the version of golang used to build a particular goiardi binary
  (again, thanks theckman)
* The changed hostname in URLs to download bug didn't get fixed in 0.7.1 quite
  all the way after all. It is now. (Thanks to oker1 for bringing that to my
  attention.)
* Fixed search tests to pass when run using more than one processor. (Brought
  to my attention by theckman.)
* Fixed a deadlock that could happen when saving an in-mem index to disk at
  the exact moment an object was being indexed. Seems to be specific to
  go1.5.1 (or at least it never happened before that I saw), but needs fixed
  anyway. (Also brought to my attention by theckman.)
* Fixed broken pipe errors with too large requests when running chef-pedant
  against goiardi built with go 1.5.1.
* Update some docs.

Next week is Chef Summit. If you’re reading this and are attending, say “hi”.

24 Jul 2015, 23:11

goiardi version 0.10.0 - Taihun

“Taihun” was the Gothic word for “ten”. For whatever reason all of the surviving Germanic languages have lost that ‘h’ sound in the middle of the word “ten”, but Gothic preserved it. You can also see some of the usual Germanic sound changes in that word, where a ’d’ sound became a ’t’, and at least some ‘k’ sounds became an ‘h’ sound. Compare Latin decem (bear in mind that the ‘c’ there had a ‘k’ sound originally). You see that sound change as well with “hundred” (compare Latin centum (same thing with the ‘c’)). It also shows a lack of satemization - compare the Russian де́сять (desyat’).

I am pleased to present goiardi version 0.10.0 - Taihun. This release brings the long discussed PostgreSQL based search, along with a search bug fix and some behind-the-scenes changes that won’t impact anyone directly.

From the CHANGELOG:

* Search architecture changed so different search backends can be used (thanks
  oker1 for your work on that).
* Postgres search is here at last! If you're using Postgres, instead of using
  the ersatz solr search, you can instead use Postgres to power your searches.
* Add a mutex for the original goiardi search - multiple simple queries
  executing simultaneously are not a problem, but multiple complex queries can 
  eat up all the RAM on the machine and cause goiardi to crash. This mitigates
  that situation.
* Be a little more forgiving with reporting protocol versions - allow
  specifying the protocol version as a query param instead of only as a
  header. This is to make showing reports with the webui a little easier.
* Bump the Chef Server version we claim to be from 11.1.6 to 11.1.7.

The Postgres search has come a long way since it went into preview. Performance with 10,000 fauxhai generated nodes is a lot more reasonable now, and it appears to be stable and behaving well. If it isn’t, of course, I’d like to hear about it. There is more detailed documentation about the new postgres search available, if you’re interested. The postgres search preview post also has useful information that’s still relevant, if you haven’t seen it yet.

For the future, once 1.0.0 is finished, I also want to make a standalone version of this search à la universe that could work with Chef Server. This would require some changes in erchef to populate the tables, though.

As mentioned in the release notes:

No matter what, whether you’re using the new postgres search or not, if you’re upgrading to this release you’ll need to rebuild your search index. If you’re using the old search, delete your index file first. Then, whether you’re using the old trie index or the new postgres based search, run knife index rebuild -y.

At last, the little issues that kept presenting themselves should all be dealt with. Development on 1.0.0, bringing orgs+RBAC to goiardi, can restart and hopefully be finished sooner rather than later.

22 Jun 2015, 13:02

Postgres search in preview

It’s been stewing for way too long, moving in fits and starts over the last several months, as I’ve been busy with work stuff, real life stuff, and switching jobs stuff, but I’m pleased to announce that a new search backend is available for goiardi. Instead of just using the ersatz Solr trie-based search that goiardi’s had for a while, you can use a Postgres based search backend for Chef searches.

Since this is a pretty big thing, I’m giving it some time to stew in testing before merging it into master and making a release. Right now, it’s in the goiardi 0.10.0-dev branch.

Motivation

Goiardi’s search is fine for smaller workloads, but once you had a few hundred or so nodes in place it began to start chugging and getting very slow and the memory usage would begin getting too high. Solr, on the other hand, while a fine product, never seemed like it was quite the right tool for the job here. I thought that having search be done in Postgres was the way forward.

Testing it out

The postgres search is not yet in master (obviously), so you’ll need to install goiardi from source. Follow the instructions on the installation notes in the goardi documenation. After you run go get, you’ll need to cd into the goiardi directory (so something like cd $HOME/go/src/github.com/ctdk/goiardi), checkout the 0.10.0-dev branch with git checkout 0.10.0-dev, and then run go install github.com/ctdk/goiardi.

Once goiardi is installed, you’ll need to make the configuration file. Use the sample file in etc/ in the goiardi repository. The important settings are the postgres options (obviously) and the “Postgres and advanced search” options in that file. You’ll need to set the local-filestore-dir, and comment out or remove the data-file and index-file options.

To set up the database properly, install sqitch and checkout the pg-search branch of goiardi-schema (or use the not-finalized sqitch postgres bundle inside the goiardi source) as described in goiardi’s postgres documentation.

Once it’s up and running and you can connect to the goiardi instance, it’s time to start playing around. You could slowly create a bunch of nodes, roles, and what have you, but an even easier way to go about it is to use this node builder ruby script. It requires the chef-api (version 0.5.0) and fauxhai gems.

Take that script, fill in your hostname and path to your admin client’s key (making sure you can read it, of course), customize how many nodes you want to create on line 31, and let ‘er rip. It will run for a while, but once it’s done you can start throwing searches at it to see how it performs.

It’s also totally possible to just use it normally as you’d use a chef server, of course.

Caveats

  • Obviously this requires Postgres. The in-mem/file and MySQL data stores have to use the old search.
  • This should be able to handle all normal use cases for Chef search, and almost all weird cases. It’s totally possible to create a situation where you get weird results back, however. The big ones are that fuzzy and distance searches will not behave like you may have hoped, and that because ltree indices in Postgres cannot accept arbitrary characters, but only alphanumeric (plus ‘_‘) characters with ‘.’ as a path separator, goiardi has to convert attributes and search fields with forbidden characters to an acceptable alternative. This shouldn’t be a problem, but it’s true that if you were to have attributes named both “/dev/xvda1” and “dev_xvda1” you might not get the search results you expect. My advice at this moment is “don’t do that”.
  • The goiardi postgres search has been tested with up to 10,000 nodes generated with fauxhai without problems, but it’s very new. It’s quite likely that there are situations where the traditional Chef search with Solr is the better choice. Right now you need to use erchef for the Solr search, but now that it’s possible to add arbitrary search backends to goiardi real solr support may come someday.
  • Ideally, though, the pg-search should be able to handle any search query you toss at it (excepting the issues above with key names, and fuzzy and distance searches). If you find a bug in the Solr query parser where it chokes on a totally legitimate query, please report it as a bug.

Future

This search needn’t be limited to just goiardi. Per the proof of concept seen in the standalone goiardi universe server, this goiardi postgres search could also be carved out and run as a standalone service. Unfortunately that won’t really work until it gets integrated back into the 1.0.0 branch and development there restarts, but it’s an exciting concept. For multi-organization implementation, each organization will probably have their own search schema, rather than keeping them all together. It may or may not ever be a good fit for, say, Hosted Chef, but might work well in a self hosted installation.

Shoutouts

Major props are due for both @oker1, for providing the initial impetus to finally get this done and providing code to split the indexer and search apart from the backends, and @coderanger for giving me ideas and directing me towards using the ltree index instead of jsonb for searching.

04 Jun 2015, 14:17

goiardi version 0.9.2 - Lithobrake

This is a small bugfix release before something really cool comes along. This goiardi 0.9.2 - Lithobrake release addresses an issue that @julsevern mentioned he noticed on Twitter, where importing goiardi JSON dumps with reports was breaking because it was exporting reports with a nodeName tag, but import expected a node_name tag.

It turned out that when I brought goiardi in line with golang coding conventions with variable names I inadvertently changed the JSON tag for node names in reports from node_name to nodeName and never noticed. This release fixes that issue. Goiardi will import JSON dumps with either version of the node name tag.

12 May 2015, 22:15

Now with repos and a cookbook

It took an awfully long time, but I’ve set up a packagecloud.io repository for goiardi finally. The repo can be found at https://packagecloud.io/ct/goiardi, and currently has debs for debian wheezy and ubuntu 14.04 on amd64. RHEL/CentOS and other debian/ubuntu packages will be forthcoming.

On a related note, while I didn’t write this, there’s a goiardi cookbook now, too. I had started on a real cookbook for goiardi, but ironically never finished it because I kept trying to do too much and have it do everything. So, this is pretty cool (and handy).

31 Mar 2015, 07:14

goiardi version 0.9.1 - Arthropleura

Photo by Sailko

Arthropleura was a eight and a half foot long millipede that dwelled in the Carboniferous period 340 to 280 million years ago, as well as in your dreams tonight.

It’s been a few months, but goiardi 0.9.1 - Arthropleura is out now. As it happens ChefConf is going on right now. This release wasn’t actually timed to come out at the same time as ChefConf, and I wasn’t able to make it there this year, but that’s how it worked out.

This is yet another interim release addressing some issues that were brought to my attention over January and February that then got put on the back burner for a bit while I was on vacation, busy with real work, and just taking a bit of a goiardi break after I was getting kind of burned out from cranking on it too hard for too long. It slowed down goiardi development for a while, but I was getting pretty useless for a while there anyway.

From the CHANGELOG:

* Fix error where requests for zero byte cookbook files would crash.
* Authentication docs improvments (thanks oker1!)
* Rewritten and more robust cookbook depsolver.
* Fix for client creation with cheffish (thanks whiteley!)
* Fix for search where searching for something like "foo:bar AND NOT foo:bar"
  was returning incorrect results. (brought to my attention and test provided
  by brimstone, thanks!)
* Fixed a bug where clients could be created with the same name as a user (or
  vice versa) in in-memory mode.

This release has quite a few contributions from others. Thanks go out to oker1, whiteley, and brimstone on github for their pull requests and bringing issues to my attention.

Coming up, I’m planning on at least one more release before 1.0.0 comes out, which will bring more search improvements to finally deal with search performance when there are thousands of nodes. The current ersatz solr implementation works fine for smaller installations, but once there’s a lot of content to be indexed the RAM and speed become problems still. After that these new changes will need to be merged into 1.0.0 and then that branch will be wrapped up. When I left off that branch’s development it was coming along nicely, but still had areas that needed tightened up to fit standard Chef server behavior more closely.

27 Dec 2014, 12:11

goiardi version 0.9.0 - Spaceship and Sun

The Spaceship-and-Sun, of course, is the sign of the Galactic Empire, and was once the symbol of order and prosperity from one end of the galaxy to the other. In other news, I re-read the original Foundation trilogy this past week.

This is another interim release while work chugs along on the 1.0.0-dev branch to get goiardi up to speed with Chef 12. That’s coming along, but I decided to address some issues that had been bothering me for a long time before finishing with that, so I’m happy to release goiardi 0.9.0 - Spaceship and Sun.

From the CHANGELOG:

* Validate IP address supplied on the command line or in the config file.
* Compress index docs to reduce memory usage with the search index.
* Ordering searches works now.
* Index and datastore files now only write to disk if there have been changes
  since the last time they were saved.
* In tandem with the previous change, freeze interval default has been changed
  from 300 seconds to 10 seconds.
* Bump Chef Server version we claim to be from 11.1.3 to 11.1.6.

Normally a release like this would not require a minor version bump, but it does change the format of the index file. Upgrading that is extremely easy, though: before starting the upgraded goiardi up, delete the index file. After goiardi has started, rebuild the index file with knife index rebuild. The new index format uses much less RAM - testing has shown the new index using between a quarter to a third as much RAM as before. Search results are also now ordered in a sensible fashion.

As mentioned in the release notes, if you are upgrading goiardi from source you will need to run go get -u github.com/ctdk/goiardi - the go-trie package has been updated. Goiardi now requires at least go 1.3 to build, and go 1.4 is recommended.

19 Dec 2014, 01:23

goiardi 0.9.0 in prerelease

Work continues in the 1.0.0-dev branch to get RBAC all working with goiardi. It’s coming along, but it’s a pretty big undertaking. I decided to take a break from that to address some issues I wanted to fix sooner than when 1.0.0 is ready.

Goiardi 0.9.0-pre1 is available for testing now. Per the release notes, this release validates IP addresses supplied on the command line or config file, if any, and brings improvements to search. The search indexes get compressed in-memory now to reduce their memory footprint while goiardi’s running, and sorting search results actually works now. This release is getting a minor version bump only because it requires deleting and rebuilding the search index. See the notes for this prerelease for more information.

Depending on how testing of this prerelease goes, a formal release should be coming reasonably soon. Super huge new features being implemented before 1.0.0 isn’t an impossibility, but it’s sort of unlikely.

12 Dec 2014, 09:28

goiardi has a new home

It’s true, goiardi’s home is now here. The Github Pages goiardi site will stay up indefinitely, since it’s not hurting anything by being here, but future goiardi updates will be found here at http://goiardi.gl. There’s also a new (but probably temporary) logo for goiardi.

I switched the goiardi site from using Octopress to Hugo, a static site generator written in Go. It’s pretty nice to use, and it’s fast.

I’m going to spend some time sprucing the new site up some and adding content where appropriate, but it’s at a good place to roll out.

23 Nov 2014, 11:29

Goiardi version 0.8.2 - Can-shaped Cranberry Sauce

This is a minor release fixing a couple of small issues that popped up that seemed important enough that they probably shouldn’t wait until 1.0.0 comes out.

Goiardi version 0.8.2 fixes a couple of typos, one with checking errors when checking for the existence of a client in SQL mode, and one for the option to use postgres in the sample config file. The use-postgresql option was documented correctly elsewhere, but was listed incorrectly as use-postgres in the sample config file. This release also adds some checks to make sure the directory given for the local filestore exists, and is a directory.

Binaries are up on the release page, as usual.

15 Oct 2014, 13:17

goiardi version 0.8.1 and schob version 0.1.1 - Hand of Ishtar, Hand of Nergal

In ancient Mesopotamia, diseases were often referred to as being the “hand” of some god. The hand of Nergal was a plague of some kind, while the hand of Ishtar was some kind of STD. Ishtar is generally identified with the Greek Aphrodite and Roman Venus as a goddess of love (although she was by no means identical to Aphrodite and Venus), so the name’s quite appropriate. The word “venereal” ultimately derives from “Venus”, after all.

These are two minor releases. Goiardi version 0.8.1 moves the bulk of the documentation from the README to readthedocs, where it should be much easier to read and work on, fixes a small issue with the postgres SQL dump file, and disables SSLv3. Schob 0.1.1 adds a fix to allow building schob on 32 bit platforms and on Windows, and updates links to the goiardi shovey documentation.

29 Sep 2014, 23:25

goiardi version 0.8.0 (plus shovey) - Hojotoho! Hojotoho!

In honor of the upcoming Chef Summit, there are new releases from goiardi and friends. This officially brings out shovey, the goiardi push jobs facility, to the world.

Because of the relative complexity of shovey, there are actually three releases here:

Astute readers may recognize the releases’ codenames as the opening lines of Act III of Die Walküre.

Shovey’s included with goiardi. Schob is the shovey client that gets installed on the nodes that will run jobs, and knife-shove is a knife plugin forked from knife-push to manage shovey jobs. The goiardi shovey docs, the schob README, and the knife-shove README have a bunch of documentation on how to use shovey.

Nice, functional cookbooks for shovey and goiardi are also coming. There is a shovey-jobs cookbook already, but it only supports Debian. Other platforms are being added, and will be ready soon.

Also coming reasonably soon are organizations and RBAC (it’s finally time), and revamping the documentation. The README has gotten too long and unweildy. It will be split apart, reorganized, and expanded where needed, while the godocs for goiardi will stop being a mirror of the README and focus more on development and internals.

See you at the Summit!

24 Sep 2014, 21:38

goiardi + shovey in final prerelease stage

Goiardi 0.8.0 is in the final stages of release engineering before its official release. This, of course, is the release that introduces shovey, the goiardi facility for running jobs on client nodes without having to do a full chef-client run.

0.8.0-pre3 should be the last prerelease version of goiardi 0.8.0. It goes hand in hand with schob 0.1.0pre2, now with updated documentation, and knife-shove 0.6.0-pre1. The shovey-jobs cookbook for setting up shovey on a node is important too, although it currently only works on Debian systems.

Documentation for all the pieces of shovey are available. The shovey docs in goiardi give an overview, and there’s some shovey API documentation as well. There’s knife-shove documentation and schob (the shovey client) documentation available in their respective repositories as well.

Barring any complications, a formal release should be coming in the next couple of days.

05 Sep 2014, 19:58

Another Shovey Preview: The Shovey-Jobs Cookbook

Shovey still isn’t finished, but it’s come a long ways. To make it easier to play with and find potential issues, I’ve released a shovey-jobs cookbook to install and configure shovey on a node.

It’s a preliminary cookbook, but since shovey’s pretty preliminary still I’m OK with that. To use the shovey-jobs cookbook, you’ll need to set goiardi and the knife-shove plugin up as explained in this previous post on shovey first. Goiardi still only works with shovey in in-memory mode; the SQL support for shovey hasn’t been finished yet. After goiardi and serf on the goiardi server are set up, spin up a node and run the shovey-jobs cookbook on it, following the instructions in the shovey-jobs README. This cookbook has only been tested on Debian, but it may work with Ubuntu as well. At the moment it’s unlikely to work with RHEL and its derivatives or other operating systems, mostly because the init script is Debian specific.

This cookbook depends on having the serf and golang cookbooks uploaded and in the node’s run list. Serf may need some extra configuration to work correctly; I had to add this to the test node’s attributes for it to work right.

"serf" => {
    "agent" => { 
      "node_name" => "goiardi-test.local",
      "start_join" => [ "10.250.55.108" ],
      "advertise" => "10.34.10.15"
    }
  }

The main serf points are that the serf node’s name needs to be the same as the chef client name on the node, it needs to join the same serf cluster as goiardi’s serf agent is running on, and it needs to advertise the correct address. The last one came up in testing for me because goiardi was running on my dev box, while the shovey node’s serf was advertising the internal vagrant address rather than the shared network address. That should be enough to get you going running commands with shovey. See the knife-shove docs for possible commands there.

On another note, there’s some documentation now for the shovey API. The new thing there is how the shovey client now streams job output back to the server, and the server can in turn stream it to a client. Unfortunately the knife-shove plugin does not yet allow you to watch the job output stream by, but it’s on the list of things to do before the formal release.

24 Aug 2014, 22:07

A preview of shovey

The long promised “shovey” jobs for goiardi, a serf-based way to push commands out to nodes, is finally ready for a preview. There’s still some loose ends to pull together, like saving job status on the client so if it’s interrupted it can report that fact back to the server and making the goiardi end work with MySQL and Postgres, but it’s ready to play with.

To try playing with it, you’ll need schob, the shovey client, and knife-shove, the knife plugin to manage shovey jobs.

As explained in the schob README:

  1. Checkout the “serfing” branch from goiardi
  2. Create an RSA keypair for signing shovey requests
  3. Install serf and run it as “serf agent”
  4. Run goiardi in in-memory mode (at this writing, MySQL and Postgres in goiardi don’t support shovey yet) with these extra options: --use-serf --use-shovey --sign-priv-key=/path/to/private.key --sign-pub-key=/path/to/public.key
  5. Install knife-shove from https://github.com/ctdk/knife-shove
  6. Set up the node in goiardi you want to test shovey on. I’ve been using the computer I do my usual goiardi dev work for most of it.
  7. Install schob the usual go way
  8. Note where the testing whitelist file is (probably somewhere like ~/go/src/github.com/ctdk/schob/test/whitelist.json).
  9. Run schob like so: schob -VVVV -e http://chef-server.local:4545 -n node-name.local -k /path/to/node.key -w /path/to/schob/test/whitelist.json -p /path/to/public.key --serf-addr=127.0.0.1:7373

Run schob with -h to see all current options.

Cookbooks for setting this all up and better documentation are coming, including documenting the HTTP and serf APIs for communicating with the server and client. The HTTP API is broadly similar to the Chef Pushy API, but not identical. Everything about shovey is still subject to change until it’s officially released, but it’s ready to play with now. Suggestions and comments are, of course, welcome.

09 Aug 2014, 16:01

Goiardi Version 0.7.2 - The Yak of Seville

Goiardi version 0.7.2 is a minor yak shaving release that there isn’t a lot to say about. The CHANGELOG pretty much sums it up:

  • Remove a newline in a debug statement, courtesy of @spheromak.
  • Also per @spheromak’s suggestion, fixed some possible race conditions revealed by building goiardi with the -race flag and running chef-pedant against it.
  • Edit doc.go slightly to make godocs more attractive.

Elsewhere, shovey work continues and comes closer to completion.

29 Jul 2014, 22:32

goiardi version 0.7.1 - Constant Manatee Attacks

The media refuses to report on this scourge afflicting the Florida coast, which is why you never hear about it.

“Shovey” is chugging along on a fast pace, but in the meantime a couple of issues came up that I thought deserved attention. Goiardi version 0.7.1 is a minor release, addressing two relatively small issues.

First, this release adds options for configuring the db pool size and max number of connections. It was happening automagically before, but I was always a bit uneasy about that. Until last night, though, I kept missing how to set that in the documentation for database/sql.

Second, for cookbook files stored locally (which is currently all of them), goiardi will now set the URL of the resource to use the current configured hostname. This was brought to my attention because someone had uploaded a bunch of cookbooks, changed the hostname and port and turned on SSL, and had the URLs break. It takes a specific chain of events to see the problem, which is why it had gone unnoticed up to this point.

From the CHANGELOG:

  • Add –db-pool-size and –max-connections options for configuring the number of idle db connections kept around and the maximum number of db connections to make to the server. It isn’t particularly useful if you’re not using one of the SQL backends.
  • For locally stored cookbook files (which is currently all of them), goiardi now generates the URL to the resource from the currently configured hostname. This fixes an issue where if you uploaded a cookbook and then changed the goairdi server’s hostname, the URLs to download cookbooks would break.

Other than that, there’s nothing else new in this release.

21 Jul 2014, 22:40

goiardi version 0.7.0 - Orphans of the Sky

In case you haven’t read it, Orphans of the Sky is a novel by Robert A. Heinlein about a generation ship floating through space where the people on board have forgotten that they are on a space ship. It’s pretty good, but more importantly the name of the first of the two novellas that make up the novel is titled “Universe”. The title of the second part, “Common Sense”, became relevant after I really dug into making this release for reasons that will soon become clear.

While it ended up being a bigger deal than I expected it to be, I am pleased to announce the release of goiardi version 0.7.0.

I took what I thought would be a brief detour from my current work on the serf integration and “shovey” stuff for goiardi to implement Chef RFC 014 real quick like, which I had been considering doing even before the RFC came along. I had been planning on making this a small release numbered 0.6.1, but then after I proposed a mechanism for caching the universe endpoint it was suggested in the RFC discussion that I see how goiardi performed with the full load of cookbooks in the supermarket before jumping the gun and adding caching before it was necessarily needed.

It turned out that will over 6,000 cookbooks the universe endpoint ran like a dog. Ultimately the problem boiled down to using gob to encode the complex data structures in cookbooks, nodes, etc. in the database. Generally gob encoding in golang is faster than using JSON, but in this particular case JSON is actually faster. The “common sense” thing to do was to change how those complex structures are stored in the database, even though it’s a bit of a brutal change. Sometimes, though, you have to bite the bullet and do the thing that sucks. It’s ended up working nicely, however; the postgres implementation is now able to make use of the json type and the json functions built into Postgres. Before switching from gob to json, serving /universe with over 6200 cookbooks took over 3 seconds with Postgres or MySQL, but afterwards Postgres can serve it in roughly 325 milliseconds. MySQL isn’t able to take advantage of the Postgres-specific json functionality, but it still went down to 1 second, so that’s a marked improvement. Using the in-memory (with or without file backing) takes about 1.2 seconds. I’m willing to take this; if you have that many cookbooks and can’t wait a second for /universe, you may want to use Postgres for your data store.

From the CHANGELOG:

  • Add /universe API endpoint, per https://github.com/opscode/chef-rfc/blob/master/rfc014-universe-endpoint.md.
  • Make file uploading a little more forgiving.
  • Make validating some cookbook metadata more forgiving, to bring goiardi’s validations in line with erchef.
  • Added some functions to make listing all cookbooks and recipes on the server faster and move the logic into the cookbook package.
  • Breaking DB change: with both MySQL and Postgres, the way data structures for cookbooks, nodes, etc. has changed from gob encoding to using JSON. This obviously breaks existing items in the database, so the following steps must be followed by users using either SQL backend for data storage:

    • Export their goiardi server’s data with the -x flag.
    • Either revert all changes to the db with sqitch, then redeploy, or drop the database manually and recreate it from either the sqitch patches or the full table dump of the release (provided starting with 0.7.0)
    • Reload the goiardi data with the -m flag. See the README or the godocs for more information.

More documentation on the universe endpoint in goiardi is available in the README. Specific information on upgrading is also in the README in the UPGRADING section.

Back to the serf/shovey stuff, then.

01 Jul 2014, 21:36

Goiardi version 0.6.0 - Order of the Elephant

The Order of the Elephant is a Danish order awarded mostly to foreign heads of state and royals. Its elephant-themed insignia is appropriate for this latest release of goiardi, version 0.6.0.

Along with some bug fixes and smaller improvements, this release introduces Postgres as a supported database backend. The schema is not compatible with erchef’s schema, but is close to the goiardi MySQL schema (with some differences of course). This is the first project I’ve done with Postgres, so comments on how goiardi is using it are welcome.

Full CHANGELOG:

  • Postgres support.
  • Fix rebuilding indexes with an SQL backend.
  • Fix a bug where in MySQL mode events were being logged twice.
  • Fix an annoying chef-pedant error with data bags.
  • Event logging methods that are not allowed now return Method Not Allowed rather than Bad Request.
  • Switch the logger to a fork that can be built and used with Windows that excludes syslog when building on Windows.
  • Add basic syslog support.
  • Authentication protocol version 1.2 now supported.
  • Add a ‘status’ param to reporting, so a list of reports return by ‘knife runs’ can be narrowed by the status of the chef run (started, success, and failure).
  • Fix an action at a distance problem with in-memory mode objects. If this behavior is still desirable (it seems to be slightly faster than the new way), it can be turned back on with the –use-unsafe-mem-store flag. This change DEFINITELY breaks in-mem data file compatibility. If upgrading, export your data, upgrade goiardi, and reload your data.
  • Add several new searchable parameters for logged events.
  • Add organization_id to all MySQL tables that might need it someday. Orgs are not used at all, so only the default value of 1 currently makes it to the database.
  • Finally ran ‘go fmt’ on goiardi. It didn’t even mess up the long comment blocks, which was what I was afraid it would do. I also ran golint against goiardi and took its recommendations where it made sense, which was most areas except for some involving generated parser code, comments on GobEncode/Decode, commenting a bunch of identical functions on an interface in search, and a couple of cases involving make and slices. All in all, though, the reformatting, linting, and light refactoring has done it good.

To reiterate, this update breaks saved data file compatibility. You’ll need to export and re-import your data if you’re using the persistent in-memory data store.

A selection of precompiled binaries are provided on the release page, including Windows and Solaris builds. There are also knife plugins for goiardi’s reporting and event logging capabilities at knife-goiardi-reporting and knife-goiardi-event-log. The reporting plugin was forked from the official Chef knife-reporting plugin to add support for the “status” parameter. Both plugins are also available on rubygems.

Finally, after someone asked how goiardi would handle running ~500 nodes, I decided to take a look and see. I whipped up a little test script to create a thousand nodes and clients to see how goiardi (and more importantly the search) would handle it. I wasn’t particularly worried about the non-search portions of goiardi (1000 rows on a table isn’t very much at all, after all), but I hadn’t tested search extensively with large amounts of data. After creating those 1,000 nodes and clients, goiardi was using about 1.40 GB of RAM. An earlier test with 5,500 nodes (but no clients) used 6.45GB RAM. The numbers show that there’s definitely room for improvement with the search, in that the capability for indexing nodes and data bags is limited by RAM, but performance was good even at 5,500 nodes. More rigorous testing is needed, however. Search is an area I intend to revisit shortly to improve these problems.

The next issues to work on with goiardi are currently improving search to work even better with large amounts of data and use raft (or something similar) to allow multiple goiardi instances to keep their search indexes consistent, reworking the permissions system, being able to upload files to s3 or similar storage providers, and the serf based pushy type thing.

19 Jun 2014, 21:19

Building goiardi on Solaris/illumos/etc.

With the recent release of Go 1.3, Go now has experimental support for Solaris and its various derivatives. This means that goiardi will build and run on Solaris, although in an experimental way.

Unfortunately to get Go on Solaris/illumos/etc. you have to build it yourself from source, and at least when I was trying it out with OmniOS in vagrant it did not come with gcc by default. Getting go up so you can try out goiardi (if you don’t want to get the binary is pretty straightforward.

  • First, install gcc. These instructions set you through installing gcc. It wasn’t clear to me at first, but you do have to run all the pkg install commands mentioned. First install the appropriate version of gcc, then install those other packages.

Once that’s done, make sure your PATH is set per the instructions on that page.

  • Install git with pkg install developer/versioning/git

  • Next, download go 1.3: wget http://golang.org/dl/go1.3.src.tar.gz and extract it somewhere. This will be your GOROOT. Set the environment variable with export GOROOT=<go's location> (assuming you’re using bash or similar).

The next steps come from *Installing Go from source:

  • In this new GOROOT, cd src/, then run ./all.bash. It will build go for you. If it complains about being unable to find unistd.h, you were like me and forgot the second step on that Installing GCC page.

  • You want to actually be able to use go, so add it to your path: export PATH=$PATH:$GOROOT/bin.

  • Now you need a workspace. In your home directory, run:

mkdir gocode
mkdir gocode/src
mkdir gocode/bin
mkdir gocode/pkg
export GOPATH=$HOME/gocode
export PATH=$PATH:$GOPATH/bin
  • Almost there. All that’s left is running
go get -t github.com/ctdk/goiardi
go install github.com/ctdk/goiardi

Assuming all went well, you can run goiardi at your prompt with whatever options you want, and it will happily run. After this point using goiardi is like using it on any other platform.

19 Jun 2014, 12:36

Goiardi Version 0.5.2 - Block of Dirt

Between the previous goiardi release and this one, my kids discovered Minecraft. This means that I started playing Minecraft, which ended up slowing goiardi development down for a while.

This blocky adversary has been overcome, however, and goiardi version 0.5.2 is out.

Originally the next goiardi release was going to add Postgres support. This was about halfway done, and will be resumed shortly, but I decided that it was more important to add the ability to import and export data before any of the goiardi structs changed.

Changes in this release, from the CHANGELOG:

  • Add import/export of goiardi data through a JSON dump.
  • Add configuration options to specify the max sizes for objects uploaded to the filestore and for JSON requests from the client.

Importing and exporting is explained in the README and in the godocs. Basically you add the -x/--export flag to your usual goiardi command to export the data to a JSON file, and run the -m/--import flags with your new desired goiardi set of flags to import the data. The administrator is responsible for removing any existing data (on-disk or SQL) and index files. This can help with backups or moving between the different storage engines goiardi supports, keeping data between upgrades, and should make testing easier in the future.

Also, per the README, there are new options to specify the maximum size of an object that can be uploaded to the filestore, and the maximum size of a JSON request from a client. The -Q/--obj-max-size flag gets the filestore object max size, while -j/--json-req-max-size sets the JSON request max size limit.

Completely unrelated to any of my efforts, but while I was preparing this release Go 1.3 was released. Goiardi and its dependencies build just fine with Go 1.3 with no changes. This also means that goiardi should be able run on some of the stranger platforms out there, like Solaris, the illumos based distributions, plan9, and apparently Dragonfly BSD. So far goiardi has only been build on OmniOS r151008j, but it built fine.

Finally, as an added convenience: on the goiardi 0.5.2 release page, precompiled binaries are being provided for download, if you don’t feel like setting up go and compiling them yourself. As of this writing binaries are provided for Linux (built on Debian wheezy), MacOS X (built on 10.9 Mavericks), FreeBSD (built on FreeBSD 9.2), and illumos (built on OmniOS r151008j). The Linux, MacOS X, and FreeBSD binaries are x86_64, the illumos binary i386 update: I misinterpreted the output of psrinfo - it was compiled for 64 bit. It appears that there’s no reason that goiardi can’t easily be cross compiled though, so a more sensible set of precompiled binaries will be coming soon. If you don’t see binaries for your platform, try grabbing a binary that’s built on a similar platform to yours, or compile it yourself.

Coming down the pipeline is resuming and re-merging the Postgres work, making some data structures safer for in-memory mode, some extra features for event logging and reporting, improving tests and moving away from depending on chef-pedant, and an idea I’ve been kicking around for a serf-based pushy type thing. For those curious about the Postgres goiardi schema, you can see it in the ‘postgres’ tree in goiardi-schema. It’s firmed up, but still subject to change before getting merged back in.

28 May 2014, 20:16

Knife plugin for the goiardi event log

There’s now a knife plugin to use with goiardi’s new event logging capabilities: knife-goiardi-event-log 0.1.1.

Usage info from the README:

There are four goiardi event log commands: knife goiardi gel list, knife goiardi gel show, knife goiardi gel delete and knife goiardi gel purge.

knife goiardi gel list returns a list of logged events. A limit may be specified with --limit; the default limit is 15. An offset may also be specified; the default offset is 0.

knife goiardi gel show [id] gives a detailed listing of a particular event log item.

knife goiardi gel delete [id] deletes the given event, and prints out a detailed display of the deleted event.

knife goiardi gel purge [id] purges all event log items with an id less than the given id. After purging the events, it displays the number of events that were purged.


Being able to list events of a certain kind of action, type of object, time range, etc. would be nice and are on the roadmap for this plugin. Adding that capability depends on changes to goairdi before that can happen though. This plugin makes use of all the current event log functionality however.

26 May 2014, 11:04

Goiardi Version 0.5.1 - Rodan Overhead

The sky darkens. Blotting out the sun, Rodan appears to destroy beloved landmarks and stomp on small models of tanks. Goiardi version 0.5.1 has been released.

This is an incremental release, made up of bug fixes and relatively minor new features. The most interesting new features are event logging and reporting, although the log level changes are also nice.

From the CHANGELOG:

  • Add log levels (from debug to critical). This makes -V/–verbose useful.
  • Add an easier option in the config file to specify log levels by name.
  • ipv6 already worked, but accidentally. Now it works in a more deliberate fashion, preventing mishaps with addresses, colons, and port numbers.
  • Authentication protocol version 1.1 now supported.
  • Remove a sort on run lists that was there for some reason. I have no idea what it was put there for, but it was wrong.
  • Add an event log to log changes to objects like nodes, clients, etc. See the README or godocs for details.
  • Add support for reporting (http://docs.opscode.com/reporting.html)

This release should not break old save files, if you’re using that. If you’re using MySQL, you’ll need to deploy the sqitch changes.

The next big things to start working on are the long promised Postgres support, and etcd/serf/consul etc. integration. Groupcache has also been asked for and looks like a worthwhile thing to pick up. As always, the issues page is a good place to look at what’s being worked on.

Goiardi is also starting to grow beyond using chef-pedant for most of its testing; it’s about time to start more native golang tests for goiardi and not use chef-pedant for the main tests. It should always work with chef-pedant, of course, but goiardi’s starting to get more features that chef-pedant doesn’t check.

02 May 2014, 10:10

Goiardi Version 0.5.0 - Stürmisch bewegt - now with MySQL

Stormily agitated, a new goiardi release appears! Goiardi version 0.5.0 has been released.

Aside from a few bug fixes, this release brings MySQL support. Now, in addition to running entirely in memory, goiardi can use MySQL for storing data. The README and godocs explain how to set that up.

From the CHANGELOG:

  • MySQL support added
  • No longer redirect /environments/NAME/roles/NAME to /roles/NAME/environments/NAME
  • Update documentation, reformat godocs
  • Split actors apart into separate user and client types, made new Actor interface that encompasses both users and clients.

Note: This release contains one breaking change. Saved data files from previous goiardi releases will not work with 0.5.0, so if you’re in that situation either don’t upgrade or remove your saved index and data files before starting up. This will regenerate your admin, validator, and webui keys, and of course nuke anything you had in place before.

Postgres support has already been requested by a few people, so that should be coming fairly soon. The issues page has been filled in with some enhancements that would be nice for goiardi to have. Feel free to add your own if there’s something you’d like to see.

19 Mar 2014, 23:04

goiardi version 0.4.2 - This is Spinal Tap

It seems like only yesterday that there was a goiardi release, but really it was a few days ago. Goiardi version 0.4.2 is hot off the presses and ready for your perusal.

This is a bugfix release. The reason this is the “Spinal Tap” release was because I noticed these problems while I took the day off after getting a spinal tap and was bored, so it seemed fitting. The bug fixes, from the CHANGELOG, are:

  • Perm tweak for nodes updating themselves.
  • Small change with validating role descriptions when creating or updating from JSON.
  • Fix issue with saving complicated indexed objects to disk where improperly flattened indexable objects were making the gob encoder puke all over itself when encoding the tries in the index docs.
  • Fixed a possible regression with synchronizing cookbooks that did not show up in testing, but only in real use.
  • An absolutely bonkers fix for listing cookbook files with webui. Webui wants all of the cookbook top level attributes sent over with a request to /cookbooks/<name>/<version>, but this is the exact opposite of the behavior chef-pedant wants, where empty definitions, attributes, etc. are not sent over. Knife also seems quite content with this, so the fix for now, since the two cases are mutually exclusionary, is to only send the empty hashes for those top level attributes with a GET if the request is coming from the webui. Bizarre, but it seems to be what’s necessary.

These were all problems that evaded notice during testing, but did pop up when I was using goiardi to set up a vagrant VM to start working on letting goiardi run a real database (optionally, of course) instead of the in-memory data store system it’s using now. The in-memory mode is great and all for testing, and being able to freeze the data to disk for persistence is very handy, but being able to use a real database would be nice for some workloads. Supporting both the current in-memory storage and a database backend is a top priority.

Which database to use is still a bit up in the air, but I’m leaning towards throwing caution to the wind and providing support for both MySQL and Postgres, with SQLite as a distinct possibility down the line. MySQL is likely to come first, but that is not set in stone.

16 Mar 2014, 14:32

goiardi version 0.4.1 release

Hot on the heels of the 0.4.0 release comes goiardi version 0.4.1. This release adds a –disable-webui option to disable using chef-webui with goiardi, fixes a bug with parsing configuration file options and rearranges how some of the config struct items are set, fixes a typo in the sample config file, and makes some small documentation tweaks.

Other than that, all the changes announced in the 0.4.0 release announcement and previous releases remain relevant.

16 Mar 2014, 10:28

Version 0.4.0 of goiardi released. Now, with authentication!

A new version of goiardi, version 0.4.0, as been released.

This version fixes a bug with pessimistic matching, adds more tests, improves the documentation with an expanded README and more godocs, and introduces client/user authentication and authorization and SSL for connections as optional settings. It will also run and authenticate properly with the chef webui, if you’re so inclined. The webui is a separate install, however, and does not ship with goiardi.

There is also a fork of chef-pedant customized for goiardi now, to handle some tests with authentication where goiardi and the chef-server make slightly different decisions.

Goiardi still works fine in no-auth mode. The authentication and authorization parts are completely optional, but they’re there if you want or need authentication. It’s recommended that if you do use authentication that you should set the –index-file and –data-file options for persistent data and index storage, but it’s not strictly necessary. See the README for more on how to set up authentication.

SSL is pretty straightforward, and also explained in the README. You’ll need to generate a certificate/key pair for it to work, but that isn’t too hard. This does assume that you’d use a self-signed certificate for this; if you’re going to use a real cert I assume you know what you’re doing (and am a little surprised, to be honest).

That’s it for this release. The next major goiardi task coming up, barring any unexpected changes, is allowing it to use a database for the backend. This will be optional, of course; I plan on supporting the in-memory, no-auth mode a la chef-zero for goiardi indefinitely. Further out, there may be a smoother installation process, and someday even perhaps goiardi-specific features.

13 Mar 2014, 10:03

Meanwhile, in the auth branch...

While there is still some work remaining with finishing some needed authorization checks, I’m happy to report that in the goiardi auth branch, authentication for chef clients and webui is working and passing the chef-pedant tests.

Once those authorization issues are ironed out, I’ll start work on updating the documentation and making a new formal release. If, in the meantime, anyone felt like testing it, it’s pretty easy. Start goiardi with the -A or --use-auth flags, and set --conf-root to whereever you’d like the chef-validator, chef-webui, and admin certificates to be stored. You’d probably also want to specify index and data files for persistence, but you don’t have to. Goiardi works with chef-webui fine as well, but at the moment you’ll have to set a password for the admin user with knife.

As of this writing goiardi authentication is still a work in progress, but it’s far enough along to start banging on to look for problems. The chef-pedant tests are pretty happy, but that doesn’t always catch every real world issue.

16 Feb 2014, 11:46

Version 0.3.3 of goiardi released

I am pleased to announce another goiardi release.

This release, version 0.3.3 of goiardi, doesn’t add much in the way of new functionality. The data store and indexer have been tweaked slightly, and go tests have been added for some of the goiardi components. The go test coverage isn’t complete, and most goiardi testing is handled with chef-pedant, but these golang tests will be handy for testing internal functionality that could break, yet not show up in the chef-pedant testing.

While there isn’t much in this release, I decided to make one here because it’s a good place to make one before starting to tackle authentication. Enjoy.

Update: It’s a little embarrassing, but there was a problem merging branches, and a necessary change got backed out or overlooked, so version 0.3.1 has been superseded by version 0.3.3. On the off chance anyone got the previous version, I apologize.

13 Feb 2014, 09:28

Go test coverage progressing

It’s in progress, but goiardi’s getting actual golang test coverage. It’s being built out in the test_writing branch, and so far has coverage with the crypto, data store, indexer, and utility libraries inside goiardi. Already the data store and indexer save and load functions have been reworked to make testing easier, and more tests will be coming soon.

Previously goiardi testing was done with chef-pedant, and that will continue to be the primary means of testing goiardi. However, there are areas where actual go tests will be useful, particularly with things like the data store and indexer that don’t get quite as exercised by chef-pedant. My current thinking is that the go tests should focus more on internals, while chef-pedant coverage is important for testing its public behavior.

18 Jan 2014, 15:10

Save Persistent Store Fix

Pushed up a change in the latest goiardi master last night that fixes an issue with saving the persistent index and data store files where, if the temp directory was not on the same partition, renaming the file after saving would cause an error. I was clued into this by a tweet in Japanese that seemed to say something about that. Ooops.

09 Jan 2014, 23:32

Reindexing search capability added

As of the most recent commit, goiardi can now rebuild its search index in case it accidentally got deleted or corrupted somehow.

To rebuild, just run knife index rebuild (or knife index rebuild -y if you don’t want to be bothered to press “y” to confirm that you really want to do it). You’ll get a message from knife that ‘knife index rebuild’ has been removed for Chef 11+ servers. This can be ignored; while goiardi identifies itself as a Chef 11+ server, this command will work because they’ve kept it in for older versions of Chef.

06 Jan 2014, 21:09

Version 0.3.0 of goiardi released

Originally posted on time.to.pullthepl.ug.

I am pleased to announce another goiardi (a Chef server written in go) release.

Version 0.3.0 of goiardi adds the ability to save the in-memory data store and the search index to disk, rather than losing everything everytime the server restarts. If the options are set, goiardi will save when it receives a SIGTERM or SIGINT signal, as well as save in the background periodically. The interval defaults to five minutes, but can be specified as an option as well. In the interest of safety and data integrity, goiardi will first save to a temporary file before renaming it to the save file name.

Of course, goiardi will also still happily run entirely in memory as before.

NB: Bear in mind that even if the data has been saved to disk goiardi still keeps the data store and search index in memory while it’s running. This could cause problems if you try to shove too much data into it. I expect most use cases would have no problems (famous last words), but the appropriate caution is warranted.

It still does no authentication or permission checking, and while it does its best to avoid data corruption and data loss it is not a real database and should not be relied on as such. It may be useful for situations where you do not want or need a full real Chef server, but could use persistent data which you don’t get with chef-zero.

Next tasks coming up: provide a way to reindex the data, improve existing docs as needed and write more, make actual go tests, fork chef-pedant and make a “goiardi” mode to simplify testing against chef-pedant.

Anyway, that’s what this release has. Have fun with it if you’re so inclined, and thank you if you do.

31 Dec 2013, 12:32

Introducing Goiardi

Originally posted on time.to.pullthepl.ug.

A while back, I decided that I would like to learn Go, which is a pretty neat language all around (in my humble opinion). I had to come up with some sort of project that would be ambitious enough to teach myself a lot of the language features, but not so huge that I’d give up in frustration. Then I thought of a name, and realized what I had to do.

After many evenings spent coding, I’m proud to announce the first offical release of Goiardi, an in-memory Chef server written in Go (the README has more information). (I started on it before I found out about Chef-Zero, but I decided to keep at it anyway. Chef-Zero was also a huge help in figuring out what the Chef server behavior ought to be.) At this time it has pretty much all the features Chef-Zero does, but sorting searches isn’t working yet. If anyone has real examples of how those sorts are specified, I’d love to hear them.

Goiardi does not support authenication or validation at all, so it isn’t very secure. Of course, it also doesn’t persist its contents, so you probably shouldn’t rely on it for your infrastructure provisioning needs.

Aside from the authentication and permission tests, goiardi performs pretty well against chef-pedant. There are a few minor tests that fail because of slight JSON formatting differences or error messages that don’t agree with chef-pedant, but otherwise it does quite well. Improving its performance against chef-pedant is, of course, an ongoing concern. It should do fine as a stand-in for chef-zero though, or some other situation where you need a chef server running but don’t care if it loses all its information when it quits.

Goiardi is, like many Chef things, open source and licensed under version 2.0 of the Apache License.

The Future

Where to go from here? Goiardi certainly has a lot of room for improvement. The first things I intend to do with it are allowing you to use a config file to store options, and allowing freezing the in-memory data structures to disk so they persist when goiardi is shut down. Those aren’t in this release because if I kept adding “just one more thing” before making a formal release, I’d never actually get it out.

Further in the future, handling real authenication and permissions is pretty important. A real Chef Server mode, where goiardi uses Solr and a real database, is on the roadmap down the road as well. SSL, better documentation, and making the Go code more idiomatic where needed are also on the TODO list.

Anyway, there is is. Feel free to check it out, bang on it, see where it breaks (and where it works). Suggestions, code, and comments are always welcome. Hopefully the comments are nice, but this is the Internet.

Update: At least part of the future is now. Goiardi has config file support now with 0.2.1, huzzah! This release also fixes a problem building goiardi with more recent versions of go-flags.

Goiardi has been tested to compile with the native go compiler and run on Mac OS X (10.7 and 10.8), Debian ‘wheezy’, and Arch Linux. It also builds and runs with gccgo (using the -compiler gccgo option to go), tested on Arch Linux. Efforts are ongoing to get it to build with gccgo without the go command on OmniOS, but so far they’ve been unsuccessful.

The Name

If you’re wondering about the name, Ettore Boiardi was the real Chef Boyardee. Wakka wakka.