Die Wunde! Die Wunde!

Another bugfix release that took way too long to get done, but various sorts of Real Life™ things kept popping up. The goiardi 0.11.7 - Amfortas release has some pretty serious memory usage improvements, along with some search fixes, more control over logging type information that gets stored, and more.

Packages are up on packagecloud.io and in the homebrew tap, and will be moving into Debian sid and testing shortly. New docker images on Docker Hub (along with some other goodies that have been languishing) should be coming soon as well.

From the CHANGELOG:

* Allow access to /debug/pprof with a whitelist of IP addresses
* Properly index arrays of hashes, arrays of arrays, etc. in object attributes.
* Pretty serious memory usage improvements with search (both the in-memory
  and postgres searches).
* Fix reconnecting to serf if the connection is somehow interrupted.
* Fix negated range queries (it turns out they *do* have a use after all), and 
  refactor how NOT queries are handled generally.
* Add options to purge old reports and node statuses.
* Add option to skip logging extended object information in the event log.
* A handful of other bugfixes.
* Bump up to using golang 1.9.3 for builds.
* Minor changes to the documentation.

On a frequently said note, I need to get a move on with Chef 12+ support - recent Chef clients no longer work with Chef 11 compatible servers. Eeeek!

Spookay!

This is another release in the ever increasing series of small bugfix releases in the 0.11.x series. The only changes in goiardi 0.11.5 - Spookay! are some fixes for certain somewhat obscure search query cases. Hopefully now I’ll get moving more on the long promised 1.0.0.

From the CHANGELOG, such as it is:

* Several search fixes:
  - With postgres search:
    * Fixed reindexing after it broke with the previous update that eliminated
    a lot of unneeded extra rows in the database.
    * Fixed basic queries with NOT statements.
    * Separately, fixed using NOT with subqueries. On a somewhat complicated
      note, but in a way that appears to match standard Solr behavior, when
      doing a query like "name:chef* AND NOT (admin:true OR admin:bleh)" it
      works as is, but when a negated subquery is followed by another basic
      query statement, it needs to have extra parentheses around the NOT +
      subquery, like "name:chef* AND (NOT (admin:true OR admin:bleh)) AND
      public_key:*". A convoluted and unlikely scenario, but it could happen.
  - With in-memory search:
    * NOT + subqueries was also broken with the in-mem search. The fixes for
      the pg-search partially fixed it for in-mem in that it no longer made the
      server panic, but it was returning incorrect results. Additional work
      ended up being needed for in-mem search.

Photo by DiBgd.

Remember, dimetrodons are not dinosaurs. They may be majestic and awesome, but as synapsids they’re actually more closely related to you than to any reptile, dinosaur or not.

The new goiardi 0.11.1 - Dimetrodon release is a relatively minor one, but it has some features I wanted to get out before stretch freezes for real.

Performance-wise, the most important change is a refactoring of the reindexing process with search. Now, it’s broken up into smaller chunks to make it more manageable, and only one reindexing job can run at a time. This avoids the problem where multiple reindexing jobs could be issued in quick succession, and it could happen that none of them would end up complete and the index would be in a bad state.

This release also lets you store secrets like public keys in an external service. Currently, the only supported service is vault. As of this writing client and user public keys, the shovey signing key, and user password hashes can be stored in vault instead of the database. The glaring omission right now is that the SSL certificate and keys for when goiardi itself is using TLS for HTTP connections are still stored as files on disk. That will be dealt with eventually, although having nginx sitting in front of goiardi and handling the TLS related duties is worth considering anyway.

From the CHANGELOG:

* Allow storing secrets (client & user public keys, shovey signing private
  keys, and user password hashes) in an external service. Currently only vault
  is supported.
* Rework reindexing to break it into smaller chunks and ensure that only one
  reindexing job can run at a time.
* Package goiardi for RHEL 7 and Debian jessie for s390x. Rather experimental,
  of course.

After years of extensive dwarven research, scientists have concluded that no, pig iron is not related to pigs in any way, and is therefore impossible to craft into ☼steel bacon☼.

It’s still not orgs and whatnot, but after a long simmering process goiardi 0.11.0 - Steel Bacon is now available. The fixes in this release were important enough to release before 1.0.0 is ready.

From the CHANGELOG:

* Ability to upload cookbooks to S3.
* Add script to upload local files to S3 to migrate.
* Change how items are indexed with the postgres indexer, to reduce the number of rows in the search_items table substantially (at the cost of possible differences in search results in a few weird corner cases).
* Search parser no longer chokes on Unicode. Unfortunately Postgres' ltree module does not accept all Unicode alphanumeric characters as valid still.
* Use vendoring.
* Rejigger the package building process a bit - changing how the different packages are built and how version numbers are determined.
* Fix a long-standing annoyance where the log file would get truncated when goiardi started or restarted.
* Allow passing environment variables to goiardi through the config file.
* Fix in-memory indexer to work with go 1.7.
* Add packages for CentOS 6 and 7. Also use a gox fork pulling in someone's PR with better ARM support until that gets merged upstream eventually.
* Change the postgres columns using the 'json' data type to use 'jsonb' instead. This is generally better, but does mean that goiardi now requires PostgreSQL 9.4 or later.

Packages are available at https://packagecloud.io/ct/goiardi, and individual binaries for a variety of architectures can be found on the release page.

The most important parts of this release are being able to upload files to AWS S3 (or compatible services), which you can read more about in the s3 docs, and the improvements with the Postgres search. These search changes reduce the number of rows in the search table by roughly half, at the expense of slightly more complex SQL queries behind the scenes. Upgrades to 0.11.0 from earlier versions of goiardi using the Postgres search will need to reindex the search tables by running knife index rebuild -y.

In any circumstance, earlier versions of goiardi using a version of Postgres older than 9.4 will need to upgrade postgres before updating goiardi and applying the sqitch patches.

This release also brings goiardi binaries for Linux on s390x (a.k.a z/Arch) (because go 1.7 will compile binaries for s390x now). Said binaries appear to work, so the next minor version of goiardi will include Debian and RHEL based packages for that architecture. This way, goiardi will obviously be Ready For The Enterprise™. Binaries for VMS will have to wait until such time as goiardi builds again with gccgo, sadly.

“Taihun” was the Gothic word for “ten”. For whatever reason all of the surviving Germanic languages have lost that ‘h’ sound in the middle of the word “ten”, but Gothic preserved it. You can also see some of the usual Germanic sound changes in that word, where a ’d’ sound became a ’t’, and at least some ‘k’ sounds became an ‘h’ sound. Compare Latin decem (bear in mind that the ‘c’ there had a ‘k’ sound originally). You see that sound change as well with “hundred” (compare Latin centum (same thing with the ‘c’)). It also shows a lack of satemization - compare the Russian де́сять (desyat’).

I am pleased to present goiardi version 0.10.0 - Taihun. This release brings the long discussed PostgreSQL based search, along with a search bug fix and some behind-the-scenes changes that won’t impact anyone directly.

From the CHANGELOG:

* Search architecture changed so different search backends can be used (thanks
  oker1 for your work on that).
* Postgres search is here at last! If you're using Postgres, instead of using
  the ersatz solr search, you can instead use Postgres to power your searches.
* Add a mutex for the original goiardi search - multiple simple queries
  executing simultaneously are not a problem, but multiple complex queries can 
  eat up all the RAM on the machine and cause goiardi to crash. This mitigates
  that situation.
* Be a little more forgiving with reporting protocol versions - allow
  specifying the protocol version as a query param instead of only as a
  header. This is to make showing reports with the webui a little easier.
* Bump the Chef Server version we claim to be from 11.1.6 to 11.1.7.

The Postgres search has come a long way since it went into preview. Performance with 10,000 fauxhai generated nodes is a lot more reasonable now, and it appears to be stable and behaving well. If it isn’t, of course, I’d like to hear about it. There is more detailed documentation about the new postgres search available, if you’re interested. The postgres search preview post also has useful information that’s still relevant, if you haven’t seen it yet.

For the future, once 1.0.0 is finished, I also want to make a standalone version of this search à la universe that could work with Chef Server. This would require some changes in erchef to populate the tables, though.

As mentioned in the release notes:

No matter what, whether you’re using the new postgres search or not, if you’re upgrading to this release you’ll need to rebuild your search index. If you’re using the old search, delete your index file first. Then, whether you’re using the old trie index or the new postgres based search, run knife index rebuild -y.

At last, the little issues that kept presenting themselves should all be dealt with. Development on 1.0.0, bringing orgs+RBAC to goiardi, can restart and hopefully be finished sooner rather than later.

It’s been stewing for way too long, moving in fits and starts over the last several months, as I’ve been busy with work stuff, real life stuff, and switching jobs stuff, but I’m pleased to announce that a new search backend is available for goiardi. Instead of just using the ersatz Solr trie-based search that goiardi’s had for a while, you can use a Postgres based search backend for Chef searches.

Since this is a pretty big thing, I’m giving it some time to stew in testing before merging it into master and making a release. Right now, it’s in the goiardi 0.10.0-dev branch.

Motivation

Goiardi’s search is fine for smaller workloads, but once you had a few hundred or so nodes in place it began to start chugging and getting very slow and the memory usage would begin getting too high. Solr, on the other hand, while a fine product, never seemed like it was quite the right tool for the job here. I thought that having search be done in Postgres was the way forward.

Testing it out

The postgres search is not yet in master (obviously), so you’ll need to install goiardi from source. Follow the instructions on the installation notes in the goardi documenation. After you run go get, you’ll need to cd into the goiardi directory (so something like cd $HOME/go/src/github.com/ctdk/goiardi), checkout the 0.10.0-dev branch with git checkout 0.10.0-dev, and then run go install github.com/ctdk/goiardi.

Once goiardi is installed, you’ll need to make the configuration file. Use the sample file in etc/ in the goiardi repository. The important settings are the postgres options (obviously) and the “Postgres and advanced search” options in that file. You’ll need to set the local-filestore-dir, and comment out or remove the data-file and index-file options.

To set up the database properly, install sqitch and checkout the pg-search branch of goiardi-schema (or use the not-finalized sqitch postgres bundle inside the goiardi source) as described in goiardi’s postgres documentation.

Once it’s up and running and you can connect to the goiardi instance, it’s time to start playing around. You could slowly create a bunch of nodes, roles, and what have you, but an even easier way to go about it is to use this node builder ruby script. It requires the chef-api (version 0.5.0) and fauxhai gems.

Take that script, fill in your hostname and path to your admin client’s key (making sure you can read it, of course), customize how many nodes you want to create on line 31, and let ‘er rip. It will run for a while, but once it’s done you can start throwing searches at it to see how it performs.

It’s also totally possible to just use it normally as you’d use a chef server, of course.

Caveats

• Obviously this requires Postgres. The in-mem/file and MySQL data stores have to use the old search.
• This should be able to handle all normal use cases for Chef search, and almost all weird cases. It’s totally possible to create a situation where you get weird results back, however. The big ones are that fuzzy and distance searches will not behave like you may have hoped, and that because ltree indices in Postgres cannot accept arbitrary characters, but only alphanumeric (plus ‘_‘) characters with ‘.’ as a path separator, goiardi has to convert attributes and search fields with forbidden characters to an acceptable alternative. This shouldn’t be a problem, but it’s true that if you were to have attributes named both “/dev/xvda1” and “dev_xvda1” you might not get the search results you expect. My advice at this moment is “don’t do that”.
• The goiardi postgres search has been tested with up to 10,000 nodes generated with fauxhai without problems, but it’s very new. It’s quite likely that there are situations where the traditional Chef search with Solr is the better choice. Right now you need to use erchef for the Solr search, but now that it’s possible to add arbitrary search backends to goiardi real solr support may come someday.
• Ideally, though, the pg-search should be able to handle any search query you toss at it (excepting the issues above with key names, and fuzzy and distance searches). If you find a bug in the Solr query parser where it chokes on a totally legitimate query, please report it as a bug.

Future

This search needn’t be limited to just goiardi. Per the proof of concept seen in the standalone goiardi universe server, this goiardi postgres search could also be carved out and run as a standalone service. Unfortunately that won’t really work until it gets integrated back into the 1.0.0 branch and development there restarts, but it’s an exciting concept. For multi-organization implementation, each organization will probably have their own search schema, rather than keeping them all together. It may or may not ever be a good fit for, say, Hosted Chef, but might work well in a self hosted installation.

Shoutouts

Major props are due for both @oker1, for providing the initial impetus to finally get this done and providing code to split the indexer and search apart from the backends, and @coderanger for giving me ideas and directing me towards using the ltree index instead of jsonb for searching.

Photo by Sailko

Arthropleura was a eight and a half foot long millipede that dwelled in the Carboniferous period 340 to 280 million years ago, as well as in your dreams tonight.

It’s been a few months, but goiardi 0.9.1 - Arthropleura is out now. As it happens ChefConf is going on right now. This release wasn’t actually timed to come out at the same time as ChefConf, and I wasn’t able to make it there this year, but that’s how it worked out.

This is yet another interim release addressing some issues that were brought to my attention over January and February that then got put on the back burner for a bit while I was on vacation, busy with real work, and just taking a bit of a goiardi break after I was getting kind of burned out from cranking on it too hard for too long. It slowed down goiardi development for a while, but I was getting pretty useless for a while there anyway.

From the CHANGELOG:

* Fix error where requests for zero byte cookbook files would crash.
* Authentication docs improvments (thanks oker1!)
* Rewritten and more robust cookbook depsolver.
* Fix for client creation with cheffish (thanks whiteley!)
* Fix for search where searching for something like "foo:bar AND NOT foo:bar"
  was returning incorrect results. (brought to my attention and test provided
  by brimstone, thanks!)
* Fixed a bug where clients could be created with the same name as a user (or
  vice versa) in in-memory mode.

This release has quite a few contributions from others. Thanks go out to oker1, whiteley, and brimstone on github for their pull requests and bringing issues to my attention.

Coming up, I’m planning on at least one more release before 1.0.0 comes out, which will bring more search improvements to finally deal with search performance when there are thousands of nodes. The current ersatz solr implementation works fine for smaller installations, but once there’s a lot of content to be indexed the RAM and speed become problems still. After that these new changes will need to be merged into 1.0.0 and then that branch will be wrapped up. When I left off that branch’s development it was coming along nicely, but still had areas that needed tightened up to fit standard Chef server behavior more closely.

The Spaceship-and-Sun, of course, is the sign of the Galactic Empire, and was once the symbol of order and prosperity from one end of the galaxy to the other. In other news, I re-read the original Foundation trilogy this past week.

This is another interim release while work chugs along on the 1.0.0-dev branch to get goiardi up to speed with Chef 12. That’s coming along, but I decided to address some issues that had been bothering me for a long time before finishing with that, so I’m happy to release goiardi 0.9.0 - Spaceship and Sun.

From the CHANGELOG:

* Validate IP address supplied on the command line or in the config file.
* Compress index docs to reduce memory usage with the search index.
* Ordering searches works now.
* Index and datastore files now only write to disk if there have been changes
  since the last time they were saved.
* In tandem with the previous change, freeze interval default has been changed
  from 300 seconds to 10 seconds.
* Bump Chef Server version we claim to be from 11.1.3 to 11.1.6.

Normally a release like this would not require a minor version bump, but it does change the format of the index file. Upgrading that is extremely easy, though: before starting the upgraded goiardi up, delete the index file. After goiardi has started, rebuild the index file with knife index rebuild. The new index format uses much less RAM - testing has shown the new index using between a quarter to a third as much RAM as before. Search results are also now ordered in a sensible fashion.

As mentioned in the release notes, if you are upgrading goiardi from source you will need to run go get -u github.com/ctdk/goiardi - the go-trie package has been updated. Goiardi now requires at least go 1.3 to build, and go 1.4 is recommended.