Chef 0.9.0 and Ohai 0.5.6 Released

Today I’m very happy to announce the release of Chef 0.9.0, Ohai 0.5.6, and mixlib-config 1.1.2. Chef 0.9.0 Brings some new features, a ton of under-the-hood cleanup, and lays the foundation for some big improvements ahead on the roadmap.

Improvements and New Features in Chef

The biggest update is to chef-servers’s cookbook handling: chef-server now can store multiple versions of cookbooks. In support of this addition, we’ve reworked the way cookbooks are uploaded to and stored on the server, so files are now uploaded individually, and organized on disk by the checksum of their contents. This means that when you upload an updated cookbook to the server, only the updated or newly added files will need to be transferred to and stored on the server. We’ve also introduced transactional behavior in the uploading process, so clients will always get a consistent view of their cookbooks, even if they’re running while you’re updating the cookbooks. Chef isn’t making much use of this functionality yet—the latest versions of cookbooks are still used almost everywhere—but we’re really excited about the upcoming features we’ll be able to provide because of it. Specifically, in an upcoming release, chef will give you the ability to define the different environments that comprise your infrastructure, such as QA, staging, and production, and pin specific versions of cookbooks to each environment.

We’ve also got a huge bump in our Windows support, thanks to our MVP, Doug MacEachern of VMware. Doug has added an environment provider that lets you automate the process of setting your environment variables, user and group support, and the ability to mount filesystems on Windows. Doug has also written a ton of plugins for ohai, and he even takes the time to write vbscripts to set up chef-client for his in-laws!

In this release, we’ve updated our handling of attribute precedence, fixing CHEF-838 and CHEF-45. You can now access the default and override attributes in your attribute files, which were previously only accessible in roles. This allows you to set default attributes in your attribute files with syntax like this:

default[:mysql][:bind_address] = '127.0.0.1'
default[:mysql][:datadir]      = '/var/lib/mysql'

Previously, that syntax would set the values only if they had not previously been set. Now, it will always set the attribute at the lowest precedence, allowing you to override these attributes via node data or roles. Continuing the example, users of this cookbook would be able to override these defaults in their roles, like this:

default_attributes :mysql => {:bind_address => '0.0.0.0'}

This value can be further overridden by setting a mysql[:bind_address] attribute on the node, or with an override attribute in a role. We now recommend using the default keyword to set attributes in any recipes you plan to share. This will give users of your cookbooks the most predictable behavior, and maximum flexibility in tailoring the attributes to their specific needs.

We’ve also moved attributes from ohai into a special “automatic_attrs” category. These automatic attributes have the highest precedence, and cannot be overridden by even an override attribute. This is really helpful when editing node attributes, as now the ohai data will be separated from the attributes you explicitly set on the node. For those of you upgrading from 0.8.x, you’ll need to run a migration on your existing nodes to purge ohai data from the normal node attributes. See the “upgrading” section below for more details.

Chef 0.9.0 is also the first version of Chef to fully support Ruby 1.9. On the server side, we’ve updated Chef to run on Merb 1.1.x, which was the last dependency we were waiting on for 1.9 support. On the client side, we’ve found and fixed a tricky issue with mixlib-config where Ruby 1.9 clients were incorrectly using default values for some configuration parameters. If you’re running chef-client on Ruby 1.9, be sure to update your mixlib-config to pick up this bug fix. We’ve already got a few eager adopters running on Chef on Ruby 1.9.2 preview 3 without issue, so if you’re comfortable running ruby 1.9 in production, you can bring Chef along for the ride.

In this release we also have expanded support for Solaris thanks to Toomas Pelberg, who contributed a provider for the service resource on Solaris.

Avishai Ish-Shalom changed our default backup behavior from storing the backups in the same directory, to storing the backups in a parallel tree. This solves an issue where updating a file in a conf.d directory would leave the backed-up file in the same directory and both files would be used by the application (you can enable the old behavior by setting the file_backup_path to nil in your client.rb configuration file). Avishai also contributed a patch to ohai to parse EC2 security groups into an array. Thanks, Avishai!

Our 0.8.16 MVP, Akzhan Abdulin, did not rest on his laurels. This time around, he found and fixed an issue where Chef was not setting the Content-Length header on POST and PUT requests, which was causing issues when running a chef server cluster behind nginx. Akzhan also made the webui sort nodes by name, so they’ll be easier to find, and patched ohai to support network devices with underscores in the name.

Joshua Sierles, in addition to being one of the earliest adopters of this release and providing invaluable testing for the attribute precedence updates, fixed the ability to access automatic attributes in attribute files.

Alexey Ivanov fixed a bug in Chef’s handling of installed zypper packages that have updates available.

Previous MVP Ian Meyer contributed some tasty fixes, including my personal favorite, CHEF-1337. Ian also updated our Solr schema so it won’t accidentally interpret string fields as integers or floats, added a sweet knife status command, cleaned up our test suite, and fixed an authorization issue in the WebUI.

Another early adopter this time around was Joe Williams, who found and fixed an issue showing nodes in the Web UI using the release candidate. Thanks for testing and for the patch, Joe.

Renaud Chaput, our 0.8.14 MVP, fixed a bug with knife where we had added the -f switch to set the output format, but this switch was already used to direct output to a file. The -F switch will now be used to define the output format.

Matthew Kent, who’s responsible for making Chef a snap to install on Red Hat and related distributions with his RPM packages, contributed manpages and other materials he’d created during packaging, and also patched the file provider to bypass checksumming when it’s not managing content.

We have a few more improvements from Opscode to announce in this release. The first is an updated rubygems package provider which uses rubygems’ ruby API. This fixes the lingering issues we’ve had with the changes in rubygems 1.3.7, and gives us an impressive speed boost in the process: in our tests, the new provider is 300–2000 times faster when the desired gem is already installed.

Finally, this release introduces a notification API: you can now configure chef-client to run notification handlers after each successful or unsuccessful chef-client run. An example will best explain the capabilities, so here’s a simple handler that simply logs the run status using Chef’s logger:

  LogItHandler < Chef::Handler
    # Define a report method to run your notification logic
    def report
      # write your notification code here.
      # you can access the start_time, end_time, node, exception (if any), 
      # backtrace (if any), success/failure status of the run, and a list
      # of all resources and all updated resources. Be careful, these can
      # be nil if chef crashed very early in the run.
      Chef::Log.info("report for #{node.name}")
      Chef::Log.info("run completed in #{elapsed_time} seconds")
      if success?
        Chef::Log.info("Updated #{updated_resources.size} of #{all_resources.size} resources")
      else
        Chef::Log.info("Sad panda: chef run failed with exception #{exception.inspect}")
        backtrace.each {|line| Chef::Log.info(line) }
      end
    end

In your client configuration file, you enable the handler like this:

    report_handlers << LogItHandler.new     # these fire at the end of a successful run
    exception_handlers << LogItHandler.new  # these fire at the end of a failed run

Deprecated in This Release

In this release, we’re deprecating some previously valid cookbook syntax. Before I discuss the specific changes, I’d like to make it clear that the old syntax is deprecated but not removed. The deprecated syntax will continue to work for the foreseeable future; Chef will simply log a deprecation warning whenever the deprecated syntax is used. At some point in the future, we will decide on a “drop dead date” for the deprecated syntax, and embed that date in the deprecation messages. So, what do you need to do about it? Just start using the new syntax whenever you write a new recipe or update an existing one. We’ll keep you updated as we move through the deprecation process. With that out of the way, let’s look at what’s been deprecated:

• Remote file is deprecated for accessing files in cookbooks: We have added a cookbook_file resource and provider for accessing files stored in the files directory of cookbooks, and deprecated the use of remote_file for this purpose. We think the new name expresses the intent—synchronize a file from a cookbook—more clearly than does “remote file.” Remote file still exists for the purpose of fetching files from arbitrary locations on the web, however.
• Accessing the @node instance variable in recipes is similarly deprecated. Instead, access the node via the node method.

Upgrading

Upgrading to 0.9 from 0.8 is a much smoother transition than from 0.7 to 0.8, but there are still some pitfalls to be aware of. First of all, 0.9 chef-clients can read from 0.8 servers, but cannot save to 0.8 servers. 0.8 chef-clients cannot read from 0.9 chef servers at all, so you will need to upgrade your clients and server in lock-step. As a consequence of the new cookbook upload process, chef-server has two new configurable locations where cookbooks are stored. These are configured with the sandbox_path and checksum_path parameters in your server.rb; they default to /var/chef/sandboxes and /var/chef/checksums, respectively. If you previously installed chef-server via the bootstrap recipe and used the default /srv/chef path for chef server data, you will probably want to set these new configuration parameters to match:

# snippet of server.rb for a bootstrapped chef-server using /srv
sandbox_path  "/srv/chef/sandboxes"
checksum_path "/srv/chef/checksums"

Another consequence of the new cookbook uploading and storage logic is that you will need to re-upload your cookbooks after you upgrade your chef-server.

With that in mind, the basic upgrade process for chef-server is described below. As always, remember to make backups and try the upgrade in a test environment first.

1. Stop chef-server, chef-server-webui, chef-solr-indexer, and chef-solr.
2. Upgrade the gems:
   sudo gem install chef
   sudo gem install chef-server
3. If you wish to use non-default values for sandbox_path and checksum_path, edit the server configuration file (server.rb).
4. Start chef-solr, chef-solr-indexer, chef-server, and chef-server-webui
5. Re-upload your cookbooks: Run knife cookbook upload -a -o /path/to/cookbooks from a machine with chef 0.9 installed.

We’re very proud of the improvements in this release and how far we’ve come, but we’re even more proud of the community that’s formed around Chef. We’re constantly impressed by everyone taking the time to help each other get started with chef, troubleshoot problems, share tips, tricks, and insights on IRC and our mailing lists. If you have any questions or need any help, just stop by.

Release Notes – Chef – Version 0.9.0

Bug

• [CHEF-585] – no service provider for Solaris
• [CHEF-1072] – chef-server-webui incompatible with merb 1.1.0
• [CHEF-1085] – cookbook loader fails somewhat silently when given an invalid cookbook
• [CHEF-1096] – backups of config files gets included in various programs
• [CHEF-1104] – intermittent closed stream error on packages and templates
• [CHEF-1141] – Nodes webui screen needs to be sorted by name.
• [CHEF-1148] – broken direct access to ohai attributes in attribute files
• [CHEF-1161] – undefined method `cookbook_loader=’ for nil:NilClass
• [CHEF-1168] – RubyGems 1.3.7 will introduce an issue where Chef’s gem_package won’t be able to install arch-specific packages
• [CHEF-1189] – Authenticated subversion checkouts fail due to prompt
• [CHEF-1211] – Versions of packages should be specified in at most one place per sub-project
• [CHEF-1219] – Data bag item should throw Chef::Exceptions::ValidationFailed when validation failed
• [CHEF-1221] – Exception if package needs updating (zypper)
• [CHEF-1230] – Knife : -f from —format overwrites -f from other options
• [CHEF-1232] – knife —help doesn’t display full help installed as debian package or w/ debian’s rubygems
• [CHEF-1234] – remote_file does not work with binary files on Windows platforms
• [CHEF-1236] – DeepMerge fails to merge production data
• [CHEF-1246] – Override attributes from roles get written to the node
• [CHEF-1253] – Solr configuration uses dynamicFields Declaration, creates havoc during indexation
• [CHEF-1260] – file resource checksums everything
• [CHEF-1269] – The webui needs to be updated for cookbook and run list changes
• [CHEF-1270] – chef-solr-indexer dies converting an argument error to a string (possibly ruby1.9 related)
• [CHEF-1271] – The client needs to trust the server’s provided manifest when fetching cookbooks
• [CHEF-1273] – Chef::Mixin::Language uses defunct @node should use node (method)
• [CHEF-1275] – knife doesn’t work without highline, and highline is not in the gemspec
• [CHEF-1280] – Cookbook upload dies on some files
• [CHEF-1281] – Knife cannot upload a cookbook without metadata
• [CHEF-1282] – Specs throw warning "parenthesize argument(s) for future version"
• [CHEF-1284] – Remote File provider has dead code
• [CHEF-1286] – JSON Attribs and precedence
• [CHEF-1289] – API does not check for admin rights for user management
• [CHEF-1291] – Deploy resource seems to re-apply the whole recipe stack, not just the portion we specify
• [CHEF-1293] – Knife cookbook delete should default to latest if a version is not specified
• [CHEF-1294] – Cookbook uploads that are only metadta changes fail
• [CHEF-1295] – cookbook file for preseed files in package resource needs to have its run context set
• [CHEF-1296] – Regression: We download all cookbook files, even those we may not need.
• [CHEF-1299] – knife configure should ask different question for client name based on whether -i is specified or not
• [CHEF-1301] – gem_package prints gem installation messages on stdout with log
• [CHEF-1304] – Setting attributes fails when a hash of another precedence has an intermediate value
• [CHEF-1305] – ShellOut segfaults older ruby patchlevels
• [CHEF-1306] – knife cookbook site vendor fails to extract due to incorrect cwd
• [CHEF-1308] – File Cache purging may incorrectly purge or not purge cookbook files (nee remote files) and templates
• [CHEF-1309] – rubygems providers tests don’t run on older versions of rubygems
• [CHEF-1323] – when chef tries to match a process against the ps output, it should print the regex with #inspect and not #to_s
• [CHEF-1324] – Bring back the syntax check cache
• [CHEF-1325] – knife cookbook download should not fail when a version is not specified
• [CHEF-1326] – 500 error when attempting to show a cookbook with a bad/non-existent version
• [CHEF-1335] – Mixlib Config is defining methods on a metaclass of a metaclass, causing config_attr_writer to fail
• [CHEF-1336] – Chef::REST should paper over a bug in net/http
• [CHEF-1341] – rewrite the handler API
• [CHEF-1342] – Updates to mixlib-config interact poorly with method stubs, resulting in spec failures
• [CHEF-1346] – Add elapsed time to the list of methods that Chef::Handler delegates to @run_status
• [CHEF-1347] – knife recipe list shows ‘.rb’ at the end of the recipe names
• [CHEF-1348] – undefined local variable or method full_recipe_list in views/nodes/show.html.haml
• [CHEF-1353] – Cookbook uploading fails
• [CHEF-1358] – The source line in resource objects is always recipe_definition_dsl_core.rb
• [CHEF-1361] – knife search -a option does not handle nil attributes gracefully

Improvement

• [CHEF-45] – Some attributes (e.g. Ohai’s) are effectively immutable and should be read-only, lockable or namespaced.
• [CHEF-349] – Refactor rubygems provider to use Gem classes, internal rubygems API instead of calling out to the CLI
• [CHEF-838] – attributes set with the "default" keyword should not be persisted to the node, and should have lower precedence
• [CHEF-979] – Include status module for knife
• [CHEF-1130] – sync distro work from rpm packages
• [CHEF-1179] – Ruby 1.9 Support on Chef Server, WebUI, Solr and Solr Indexer
• [CHEF-1243] – Remote file should be deprecated for fetching cookbook files—this should be a cookbook file resource/provider
• [CHEF-1258] – Knife ec2 support
• [CHEF-1259] – Knife should support per-directory configuration files
• [CHEF-1264] – Chef::Solr::Query initialization should take couchdb object instead of just the database
• [CHEF-1268] – Cookbooks should sync on a file-by-file basis, support versioning, and be updated atomically
• [CHEF-1337] – Chef is too 1337 4 u

New Feature

• [CHEF-682] – Add an exception notification hook
• [CHEF-914] – knife data bag subcommands should include data bag from file
• [CHEF-1041] – Cookbook (up)loader should be version aware
• [CHEF-1220] – knife ssh tmux
• [CHEF-1228] – env provider
• [CHEF-1262] – Add user and group providers for Windows platforms
• [CHEF-1267] – Add mount provider for Windows platforms
• [CHEF-1288] – @node can no longer be accessed directly in recipes, so we need to provide a check for this on cookbook upload

Task

• [CHEF-1210] – updated man pages, init scripts and supporting distro specifics in source to make packaging consistent
• [CHEF-1263] – The docs/ directory in chef should be removed
• [CHEF-1272] – Merge the completed work on chef 1269 into master so we can release alpha 4
• [CHEF-1277] – Create a chef-server meta-gem
• [CHEF-1331] – Update knife’s manpage source

Release Notes – Ohai – Version 0.5.6

Bug

• [OHAI-132] – /lib/ohai/plugins/linux/network.rb line 40 regex is missing \_
• [OHAI-194] – ec2 security groups should be output as array
• [OHAI-196] – IO.select hangs forever reading device files on older linux kernels

Improvement

• [OHAI-192] – windows::hostname fqdn should fallback to gethostbyaddr
• [OHAI-193] – windows::cpu plugin

New Feature

• [OHAI-195] – languages::c plugin