Chef

Table Of Contents

About Node Objects

For the chef-client, two important aspects of nodes are groups of attributes and run-lists. An attribute is a specific piece of data about the node, such as a network interface, a file system, the number of clients a service running on a node is capable of accepting, and so on. A run-list is an ordered list of recipes and/or roles that are run in an exact order. The node object consists of the run-list and node attributes, which is a JSON file that is stored on the Chef server. The chef-client gets a copy of the node object from the Chef server during each chef-client run and places an updated copy on the Chef server at the end of each chef-client run.

An attribute is a specific detail about a node. Attributes are used by the chef-client to understand:

  • The current state of the node
  • What the state of the node was at the end of the previous chef-client run
  • What the state of the node should be at the end of the current chef-client run

Attributes are defined by:

  • The state of the node itself
  • Cookbooks (in attribute files and/or recipes)
  • Roles
  • Environments

During every chef-client run, the chef-client builds the attribute list using:

  • Data about the node collected by Ohai
  • The node object that was saved to the Chef server at the end of the previous chef-client run
  • The rebuilt node object from the current chef-client run, after it is updated for changes to cookbooks (attribute files and/or recipes), roles, and/or environments, and updated for any changes to the state of the node itself

After the node object is rebuilt, all of attributes are compared, and then the node is updated based on attribute precedence. At the end of every chef-client run, the node object that defines the current state of the node is uploaded to the Chef server so that it can be indexed for search.

Attributes

An attribute is a specific detail about a node, such as an IP address, a host name, a list of loaded kernel modules, the version(s) of available programming languages that are available, and so on. An attribute may be unique to a specific node or it can be identical across every node in the organization. Attributes are most commonly set from a cookbook, by using Knife, or are retrieved by Ohai from each node prior to every chef-client run. All attributes are indexed for search on the Chef server. Good candidates for attributes include:

  • any cross-platform abstraction for an application, such as the path to a configuration files
  • default values for tunable settings, such as the amount of memory assigned to a process or the number of workers to spawn
  • anything that may need to be persisted in node data between chef-client runs

In general, attribute precedence is set to enable cookbooks and roles to define attribute defaults, for normal attributes to define the values that should be specific for a node, and for override attributes to force a certain value, even when a node already has that value specified.

One approach is to set attributes at the same precedence level by setting attributes in a cookbook’s attribute files, and then also setting the same default attributes (but with different values) using a role. The attributes set in the role will be deep merged on top of the attributes from the attribute file, and the attributes set by the role will take precedence over the attributes specified in the cookbook’s attribute files.

Another (much less common) approach is to set a value only if an attribute has no value. This can be done by using the _unless variants of the attribute priority methods: default_unless, set_unless, and override_unless. (normal_unless is an alias of set_unless; use either alias to set an attribute with a normal attribute precedence.) These variants should be used carefully because when they are used, the attributes applied to nodes may become out of sync with the values in the cookbooks as these cookbooks are updated. This approach can create situations where two otherwise identical nodes end up having slightly different configurations. This approach can also be a challenge to debug, so it is recommended to use the _unless variants carefully (and only when they are really necessary).

Note

Attributes can be configured in cookbooks (attribute files and recipes), roles, and environments. In addition, Ohai collects attribute data about each node at the start of the chef-client run. See http://docs.getchef.com/chef_overview_attributes.html for more information about how all of these attributes fit together.

Attribute Types

Attribute types can be any of the following:

Attribute Type Description
default A default attribute is automatically reset at the start of every chef-client run and has the lowest attribute precedence. A cookbook should be authored to use default attributes as often as possible.
force_default A force_default attribute is used to ensure that an attribute defined in a cookbook (by an attribute file or by a recipe) takes precedence over a default attribute set by a role or an environment.
normal A normal attribute is a setting that persists in the node object. A normal attribute has a higher attribute precedence than a default attribute.
override An override attribute is automatically reset at the start of every chef-client run and has a higher attribute precedence than default, force_default, and normal attributes. An override attribute is most often specified in a recipe, but can be specified in an attribute file, for a role, and/or for an environment. A cookbook should be authored so that it uses override attributes only when required.
force_override A force_override attribute is used to ensure that an attribute defined in a cookbook (by an attribute file or by a recipe) takes precedence over an override attribute set by a role or an environment.
automatic An automatic attribute contains data that is identified by Ohai at the beginning of every chef-client run. An automatic attribute cannot be modified and always has the highest attribute precedence.

Attribute Persistence

At the beginning of a chef-client run, all default, override, and automatic attributes are reset. The chef-client rebuilds them using data collected by Ohai at the beginning of the chef-client run and by attributes that are defined in cookbooks, roles, and environments. Normal attributes are never reset. All attributes are then merged and applied to the node according to attribute precedence. At the conclusion of the chef-client run, all default, override, and automatic attributes disappear, leaving only a collection of normal attributes that will persist until the next chef-client run.

Attribute Precedence

Attributes are always applied by the chef-client in the following order:

  1. A default attribute located in a cookbook attribute file
  2. A default attribute located in a recipe
  3. A default attribute located in an environment
  4. A default attribute located in role
  5. A force_default attribute located in a cookbook attribute file
  6. A force_default attribute located in a recipe
  7. A normal attribute located in a cookbook attribute file
  8. A normal attribute located in a recipe
  9. An override attribute located in a cookbook attribute file
  10. An override attribute located in a recipe
  11. An override attribute located in a role
  12. An override attribute located in an environment
  13. A force_override attribute located in a cookbook attribute file
  14. A force_override attribute located in a recipe
  15. An automatic attribute identified by Ohai at the start of the chef-client run

where the last attribute in the list is the one that is applied to the node.

Note

The attribute precedence order for roles and environments is reversed for default and override attributes. The precedence order for default attributes is environment, then role. The precedence order for override attributes is role, then environment. Applying environment override attributes after role override attributes allows the same role to be used across multiple environments, yet ensuring that values can be set that are specific to each environment (when required). For example, the role for an application server may exist in all environments, yet one environment may use a database server that is different from other environments.

Attribute precedence, viewed from the same perspective as the overview diagram, where the numbers in the diagram match the order of attribute precedence:

_images/overview_chef_attributes_precedence.png

Attribute precedence, when viewed as a table:

_images/overview_chef_attributes_table.png

Run-lists

A run-list defines all of the configuration settings that are necessary for a node that is under management by Chef to be put into the desired state. A run-list is:

  • An ordered list of roles and/or recipes that are run in an exact order; if a recipe appears more than once in the run-list, the chef-client will never run that recipe twice
  • Always specific to the node on which it runs, though it is possible for many nodes to have run-lists that are similar or even identical
  • Stored as part of the node object on the Chef server
  • Maintained using Knife and uploaded to the Chef server or via the Chef Manage user interface

A recipe is the most fundamental configuration element within the organization. A recipe:

  • Is authored using Ruby, which is a programming language designed to read and behave in a predictable manner
  • Is mostly a collection of resources, defined using patterns (resource names, attribute-value pairs, and actions); helper code is added around this using Ruby, when needed
  • Must define everything that is required to configure part of a system
  • Must be stored in a cookbook
  • May be included in a recipe
  • May use the results of a search query and read the contents of a data bag (including an encrypted data bag)
  • May have a dependency on one (or more) recipes
  • May tag a node to facilitate the creation of arbitrary groupings
  • Must be added to a run-list before it can be used by the chef-client
  • Is always executed in the same order as listed in a run-list

A role is a way to define certain patterns and processes that exist across nodes in an organization as belonging to a single job function. Each role consists of zero (or more) attributes and a run list. Each node can have zero (or more) roles assigned to it. When a role is run against a node, the configuration details of that node are compared against the attributes of the role, and then the contents of that role’s run list are applied to the node’s configuration details. When a chef-client runs, it merges its own attributes and run lists with those contained within each assigned role.

Deep Merge

Attributes are typically stored in cookbooks and recipes, roles, and environments. These attributes are rolled-up to the node level during a chef-client run. For example, a recipe can store attributes using a multi-level hash or array; a group of attributes for web servers might be:

override_attributes(
  :apache => {
    :listen_ports => [ 80 ],
    :prefork => {
      :startservers => 20,
      :minspareservers => 20,
      :maxspareservers => 40
    }
  }
)

But what if all of the web servers are not the same? What if some of the web servers required a single attribute to have a different value? You could store these settings in two locations, once just like the preceding example and once just like the following:

override_attributes(
  :apache => {
    :listen_ports => [ 80 ],
    :prefork => {
      :startservers => 30,
      :minspareservers => 20,
      :maxspareservers => 40
    }
  }
)

But that is not very efficient, especially because most of them are identical. The deep merge capabilities of the chef-client allows attributes to be layered across recipes and cookbooks, roles, and environments. This allows an attribute to be reused across nodes, making use of default attributes set at the cookbook level, but also providing a way for certain attributes (with a higher attribute precedence) to be applied only when they are supposed to be. For example, a role named baseline.rb:

name "baseline"
description "The most basic role for all configurations"
run_list "recipe[baseline]"

override_attributes(
  :apache => {
    :listen_ports => [ 80 ],
    :prefork => {
      :startservers => 20,
      :minspareservers => 20,
      :maxspareservers => 40
    }
  }
)

and then a role named web.rb:

name "web"
description "Web server config"
run_list "role[baseline]"

override_attributes(
  :apache => {
    :prefork => {
      :startservers => 30
    }
  }
)

Both of these files are similar. They share the same structure. When an attribute is of the same type of data, such as a hash or an array, that data is merged when the attribute precedence levels are the same and is replaced when the attribute precedence levels are different.

For example, the web.rb references the baseline.rb role. The web.rb file only provides a value for one attribute: :startservers. When the chef-client compares these attributes, the deep merge feature will ensure that :startservers (and its value of 30) will be applied to any node for which the web.rb attribute structure should be applied.

This approach will allow a recipe like this:

include_recipe "apache2"
Chef::Log.info(node['apache']['prefork'].to_hash)

and a run_list like this:

run_list/web.json
{
  "run_list": [ "role[web]" ]
}

to produce results like this:

[Tue, 16 Aug 2011 14:44:26 -0700] INFO:
         {
           "startservers"=>30,
           "minspareservers"=>20,
           "maxspareservers"=>40,
           "serverlimit"=>400,
           "maxclients"=>400,
           "maxrequestsperchild"=>10000
         }

Even though the web.rb file does not contain attributes and values for minspareservers, maxspareservers, serverlimit, maxclients, and maxrequestsperchild, the deep merge capabilities pulled them in.

The following sections show how the logic works for using deep merge to perform substitutions and additions of attributes.

Substitution

The following examples show how the logic works for substituting an existing string using a hash:

role_or_environment 1 { :x => "1", :y => "2" }
+
role_or_environment 2 { :y => "3" }
=
{ :x => "1", :y => "3" }

For substituting an existing boolean using a hash:

role_or_environment 1 { :x => true, :y => false }
+
role_or_environment 2 { :y => true }
=
{ :x => true, :y => true }

For substituting an array with a hash:

role_or_environment 1 [ "1", "2", "3" ]
+
role_or_environment 2 { :x => "1" , :y => "2" }
=
{ :x => "1", :y => "2" }

When items cannot be merged through substitution, the original data is overwritten.

Addition

The following examples show how the logic works for adding a string using a hash:

role_or_environment 1 { :x => "1", :y => "2" }
+
role_or_environment 2 { :z => "3" }
=
{ :x => "1", :y => "2", :z => "3" }

For adding a string using an array:

role_or_environment 1 [ "1", "2" ]
+
role_or_environment 2 [ "3" ]
=
[ "1", "2", "3" ]

For adding a string using a multi-level hash:

role_or_environment 1 { :x => { :y => "2" } }
+
role_or_environment 2 { :x => { :z => "3" } }
=
{ :x => { :y => "2", :z => "3" } }

For adding a string using a multi-level array:

role_or_environment 1 [ [ 1, 2 ] ]
+
role_or_environment 2 [ [ 3 ] ]
=
[ [ 1, 2 ], [ 3 ] ]