Packages

New in version 1.0.0.

This page documents the Packages plugin. Packages is an alternative to Pkgmgr for specifying package entries for clients. Where Pkgmgr explicitly specifies package entry information, Packages delegates control of package version information to the underlying package manager, installing the latest version available through those channels.

Limiting sources to groups

Packages/sources.xml processes <Group> and <Client> tags just like Bundles. In addition to any groups or clients specified that way, clients must be a member of the appropriate architecture group as specified in a Source stanza.

Memberships in architecture groups is needed so that Packages can map software sources to clients. There is no other way to handle this than to impose membership in the appropriate architecture group.

When multiple sources are specified, clients are associated with each source to which they apply (based on group memberships, as described above). Packages and dependencies are resolved from all applicable sources.

Setup

Three basic steps are required for Packages to work properly.

1. Create Packages/sources.xml. This file should look approximately like the example below, and describes both which software repositories should be used, and which clients are eligible to use each one.
2. Ensure that clients are members of the proper groups. Each client should be a member of all of the groups listed in the sources.xml (like ubuntu-intrepid or centos-5.2 in the following examples), and one of the architecture groups listed in the source configuration (i386, amd64 or x86_64 in the following examples). ‘’‘Failure to do this will result in the source either not applying to the client, or only architecture independent packages being made available to the client.’‘’
3. Add Package entries to bundles.
4. Sit back and relax, as dependencies are resolved, and automatically added to client configurations.

sources.xml

sources.xml is where all package sources are configured for the Packages plugin. It processes <Group> and <Client> tags just like Bundles. The primary element in sources.xml is the Source tag:

element Source

Attributes:

Name	Description	Values	Required	Default
type	The type of the repository. This corresponds to the Packages plugin driver that will handle the source.	yum | apt | pac | pkgng	Yes	None
debsrc	Include deb-src lines in the generated APT configuration. This only applies to sources with type = apt.	true | false	No	false
essential	Include essential packages from this repo by default (i.e., without needing to specify them in a bundle). This only applies to sources with type = apt.	true | false	No	true
name	Specifiy an explicit name for the source and do not generate it automatically.	string	No	None
pulp_id	The Pulp repository ID for this repo. This only applies to sources with type = yum. Due to the amount of data that can be queried directly from Pulp, there’s rarely a need to supply other attributes.	string	No	None
rawurl	The raw URL to the (single) repository defined by this source. Component and version are ignored if this is given.	string	No	None
recommended	Include packages recommended as dependencies by APT. This only applies to sources with type = apt. You must regenerate the Packages cache after changing this attribute.	true | false	No	false
url	The base URL to use when generating URLs for this source. If url is used, you must also provide the Arch tag, at least one Component tag, and the version attribute. You must not specify rawurl. For each combination of component and Arch tag, a URL is created in the format: <url>/<version>/<component>/<arch>	string	No	None
version	The OS version this source applies to. This is used to generate URLs if the url attribute is given, and ignored otherwise.	string	No	None

Attribute groups:

• py:genshiAttrs

Child elements:

• element Component

  Components are used to build multiple repository URLs from a single Source tag. This is only meaningful if the url attribute is specified; see that attribute above for more detail.

  Component takes only text content, which may be the following values: string

• element Arch

  The architecture(s) of the repository. A client must be a member of one of the listed architecture groups in order for this source to apply to the client. Additionally, if the url attribute is specified, the Arch tag is used to generate URLs. See the url attribute for more detail.

  Arch takes only text content, which may be the following values: string

• element GPGKey

  The GPG key(s) for the repository. This only applies to sources with type = yum. If GPG keys are specified, then GPG checking will be automatically enabled for the repository, both on the Bcfg2 server (if yum libraries are in use) and on the Bcfg2 client (if you use Packages to generate your Yum config).

  GPGKey takes only text content, which may be the following values: string

• element Options

  Arbitrary options to be used in the repository configuration.

  Attributes:

  Name	Description	Values	Required	Default
  clientonly	The options given in this tag will only be used on the Bcfg2 clients, not on the server.	true | false	No	None
  serveronly	The options given in this tag will only be used on the Bcfg2 server, not on the clients.	true | false	No	None
  *	All other (arbitrary) attributes will be added to the repository configuration.	Any	No	None

  Attribute groups:

  • py:genshiAttrs

• element Blacklist

  Blacklist the given package(s) from the Packages plugin. This prevents them from being included in automatically-resolved dependencies.

  Blacklist takes only text content, which may be the following values: string

• element Whitelist

  If Whitelist is specified, only packages listed will be included by the Packages plugin.

  Whitelist takes only text content, which may be the following values: string

Element groups:

py:genshiElements

Handling GPG Keys

New in version 1.2.0.

If you have yum libraries installed, Packages can automatically handle GPG signing keys for Yum and Pulp repositories. (You do not need to use the native yum resolver; if yum libraries are available, GPG signing keys can be handled automatically.) Simply specify the URL to the GPG key(s) for a repository with GPGKey elements:

<Source type="yum"
        rawurl="http://mirror.example.com/centos6-x86_64/RPMS.os">
  <Arch>x86_64</Arch>
  <GPGKey>http://mirror.example.com/keys/RPM-GPG-KEY-CentOS-6</GPGKey>
</Source>

More than one <GPGKey> tag can be specified per Source.

With the keys specified thusly, Packages will include the keys in the generated yum config file, and will ensure that the keys are imported on the client.

There is no need to specify <GPGKey> tags for Pulp sources; that data is pulled directly from the Pulp REST API.

Arbitrary Repo Options

New in version 1.2.3.

You can specify arbitrary options to be added to the repository config on the server side, if you are using the native yum libraries, and on the client side if you are using the ability of Packages to automatically generate your Yum config. To do this, add an Options tag to a Source; all of its attributes will be added verbatim to the repository in the generated config. For instance:

<Source type="yum" rawurl="http://mirror.example.com/centos-6-os">
  <Arch>x86_64</Arch>
  <Options proxy="http://proxy.example.com"/>
</Source>

If you are using native yum libraries and need to set options only on the Bcfg2 server, you can set the serveronly attribute to “true”; or, if you need to set options only on the client, you can set the clientonly attribute to “true”. For instance, if your Bcfg2 server needed to use a proxy to access a repo, and you wanted to expire metadata caches very quickly on the client, you could do:

<Source type="yum" rawurl="http://mirror.example.com/centos-6-os">
  <Arch>x86_64</Arch>
  <Options serveronly="true" proxy="http://proxy.example.com"/>
  <Options clientonly="true" metadata_expire="0"/>
</Source>

Prerequisite Resolution

Packages provides a prerequisite resolution mechanism which has no analogue in Pkgmgr. During configuration generation, all structures are processed. After this phase, but before entry binding, a list of packages and the client metadata instance is passed into Packages’ resolver. This process determines a superset of packages that will fully satisfy dependencies of all package entries included in structures, and reports any prerequisites that cannot be satisfied.

Disabling dependency resolution

New in version 1.1.0.

Dependency resolution can be disabled by adding the following setting to bcfg2.conf in the packages section:

[packages]
resolver=0

All metadata processing can be disabled as well:

[packages]
metadata=0

This setting implies disabling the resolver.

Blacklisting faulty dependencies

If you encounter an issue with faulty dependency resolution due to Packages, please file a bug report so that we can fix the problem in future releases. In the meantime, you can work around this issue by blacklisting the offending Package in your Sources. The Blacklist element should immediately follow the Component section of your source and should look like the following:

<Blacklist>unwanted-packagename</Blacklist>

If you use the built-in Yum config generator, blacklisted packages will be added to the exclude list for the source.

Example usage

Create a sources.xml file in the Packages directory that looks something like this:

<Sources>
  <Group name="ubuntu-intrepid">
    <Source type="apt"
            url="http://us.archive.ubuntu.com/ubuntu"
            version="intrepid"
            debsrc="true">
      <Component>main</Component>
      <Component>universe</Component>
      <Arch>i386</Arch>
      <Arch>amd64</Arch>
    </Source>
  </Group>
</Sources>

Note

New in version 1.1.0.

The default behavior of the Packages plugin is to not make any assumptions about which packages you want to have added automatically [1]. For that reason, neither Recommended nor Suggested packages are added as dependencies by default. You will notice that the default behavior for apt is to add Recommended packages as dependencies. You can configure the Packages plugin to add recommended packages by adding the recommended attribute, e.g.:

<Source type="apt" recommended="true" ...>

[1]	Bcfg2 will by default add Essential packages to the client specification. You can disable this behavior by setting the essential attribute to false: <Source type="apt" essential="false" ...>

Yum sources can be similarly specified:

<Sources>
  <Group name="centos-5.2">
    <Source type="yum"
            url="http://mirror.centos.org/centos/"
            version="5.2">
      <Component>os</Component>
      <Component>updates</Component>
      <Component>extras</Component>
      <Arch>i386</Arch>
      <Arch>x86_64</Arch>
      <GPGKey>http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5</GPGKey>
    </Source>
  </Group>
</Sources>

For sources with a url attribute, the version attribute is also necessary.

Pulp sources are very simple to specify due to the amount of data that can be queried from Pulp itself:

<Sources>
  <Group name="centos-6-x86_64">
    <Source type="yum" pulp_id="centos-6-x86_64-os"/>
    <Source type="yum" pulp_id="centos-6-x86_64-updates"/>
    <Source type="yum" pulp_id="centos-6-x86_64-extras"/>
  </Group>
</Sources>

Note

There is also a rawurl attribute for specifying sources that don’t follow the conventional layout.

<Sources>
  <Group name="centos5.4">
    <Source type="yum"
            rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.os">
      <Arch>x86_64</Arch>
    </Source>
    <Source type="yum"
            rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.updates">
      <Arch>x86_64</Arch>
    </Source>
    <Source type="yum"
            rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.extras">
      <Arch>x86_64</Arch>
    </Source>
  </Group>
</Sources>

<Sources>
  <Group name="ubuntu-lucid">
    <Source type="apt"
            rawurl="http://hudson-ci.org/debian/binary">
      <Arch>amd64</Arch>
    </Source>
    <Source type="apt"
            rawurl=http://hudson-ci.org/debian/binary">
      <Arch>i386</Arch>
    </Source>
  </Group>
</Sources>

Configuration Updates

Packages will reload its configuration upon an explicit command via bcfg2-admin:

[0:3711] bcfg2-admin xcmd Packages.Refresh
True

During this command (which will take some time depending on the quantity and size of the sources listed in the configuration file), the server will report information like:

Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-i386/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-amd64/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-i386/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-amd64/Packages.gz
...
Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/filelists.xml.gz
Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/primary.xml.gz

One line per file download needed. Packages/sources.xml will be reloaded at this time, so any source specification changes (new or modified sources in this file) will be reflected by the server at this point.

This process is much, much faster if you use the native yum library support.

Soft reload

New in version 1.2.0.

A soft reload can be performed to reread the configuration file and download only missing sources.:

[0:3711] bcfg2-admin xcmd Packages.Reload
True

This is done automatically any time sources.xml is updated.

Availability

Support for the following clients is currently available. Support for other package managers (Portage, Zypper, IPS, etc) remain to be added.

apt

All dpkg based clients (for example Debian, Ubuntu or Nexenta) could be handled with the apt module:

<Source type="apt"
        url="http://us.archive.ubuntu.com/ubuntu"
        version="intrepid">
  <Component>main</Component>
  <Component>universe</Component>
  <Arch>i386</Arch>
  <Arch>amd64</Arch>
</Source>

pac

For Arch Linux or Parabola GNU/Linux-libre you could use the pac module for packages. You do not need to supply a version attribute as the mirrors are rolling release and does not supply different versions.

<Source type="pac"
        url="http://mirrors.kernel.org/archlinux/">
  <Component>core</Component>
  <Component>extra</Component>
  <Component>community</Component>
  <Arch>i686</Arch>
  <Arch>x86_64</Arch>
</Source>

pkgng

The support for the Next Generation package management tool for FreeBSD is called pkgng. It downloads the packagesite file from the mirror and parses the dependencies out of it. It currently does not use the DNS SRV record lookup mechanism to get the correct mirror and does not verify the signature inside the packagesite file.

<Source type="pkgng"
        url="http://pkg.freebsd.org/"
        version="10">
  <Component>latest</Component>
  <Arch>x86:64</Arch>
  <Arch>x86:32</Arch>
</Source>

yum

Rpm based clients (for example RedHat, CentOS or Fedora) could be handled with the yum module:

<Source type="yum"
        url="http://mirror.centos.org/centos/"
        version="5.2">
  <Component>os</Component>
  <Component>updates</Component>
  <Component>extras</Component>
  <Arch>i386</Arch>
  <Arch>x86_64</Arch>
</Source>

Package Checking and Verification

In order to do disable per-package verification, you will need to use BoundEntries, e.g.:

<BoundPackage name="mem-agent" priority="1" version="auto"
              type="yum" verify="false"/>

Generating Client APT/Yum Configurations

The Packages plugin has native support for generating Yum and Apt configs. Simply add entries like these to the appropriate bundles:

<Path name="/etc/yum.repos.d/bcfg2.repo"/>
<Path name="/etc/apt/sources.list.d/bcfg2-packages-generated-sources.list"/>

If you want to change the path to either of those files, you can set yum_config or apt_config in bcfg2.conf to the path to the config files you want to generate:

[packages]
yum_config=/etc/yum.repos.d/all.repo
apt_config=/etc/apt/sources.d/all

If you need to distribute a config to different places on different hosts, you can use the altsrc attribute, e.g.:

<Bundle>
  <Group name="sles">
    <Path name="/etc/yum/yum.repos.d/bcfg2.repo"
          altsrc="/etc/yum.repos.d/bcfg2.repo"/>
  </Group>
  <Group name="sles" negate="true">
    <Path name="/etc/yum.repos.d/bcfg2.repo"/>
  </Group>
</Bundle>

See Configuration for more details on these options.

Note

Support for generating Yum configs was added in 1.2.0, and Apt configs was added in 1.3.0. Before that, you could use Genshi Templates or Cheetah Templates to generate your configs.

Package Groups

Some packaging systems provide package groups. To include a package group, use the group attribute of the Package tag.

pac

New in version 1.4.0.

Pacman groups are supported:

<Package group="base"/>

yum

Yum package groups are supported by both the native Yum libraries and Bcfg2’s internal dependency resolver. You can use either the short group ID or the long group name:

<Package group="SNMP Support"/>
<Package group="system-management-snmp"/>

By default, only those packages considered the “default” packages in a group will be installed. You can change this behavior using the type attribute:

<Package group="development" type="optional"/>
<Package group="Administration Tools" type="mandatory"/>

Valid values of “type” are:

• mandatory: Only install mandatory packages in the group.
• default: Install default packages from the group (the default).
• optional or all: Install all packages in the group, including mandatory, default, and optional packages.

See PackageStructure for details.

You can view the packages in a group by category with the yum groupinfo command. More information about the different levels can be found at http://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups#Installation

Using Native Yum Libraries

New in version 1.2.0.

By default, Bcfg2 uses an internal implementation of Yum’s dependency resolution and other routines so that the Bcfg2 server can be run on a host that does not support Yum itself. If you run the Bcfg2 server on a machine that does have Yum libraries, however, you can enable use of those native libraries in Bcfg2 by setting use_yum_libraries to 1 in the [packages:yum] section of bcfg2.conf.

Benefits to this include:

• Much lower memory usage by the bcfg2-server process.
• Much faster Packages.Refresh behavior.
• More accurate dependency resolution.
• Better use of multiple processors/cores.

Drawbacks include:

• Resolution of package dependencies is slower and more resource-intensive. At times it can be much slower, particularly after running Packages.Refresh.
• More disk I/O. This can be alleviated by putting /var/lib/bcfg2/Packages/cache on tmpfs, but that offsets the lower memory usage. In some cases, you may have to raise the open file limit for the user who runs your Bcfg2 server process, particularly if you have a lot of repositories.

Configuring the Yum Helper

Due to poor memory management by the Yum API, the long-lived bcfg2-server process uses an external short-lived helper, bcfg2-yum-helper, to do the actual Yum API calls for native yum library support. By default, Bcfg2 looks for this helper in $PATH, or, failing that, at /usr/sbin/bcfg2-yum-helper. If you have installed the helper elsewhere, you will need to configure that location with the helper option in the [packages:yum] section, e.g.:

[packages:yum]
use_yum_libraries = 1
helper = /usr/local/sbin/bcfg2-yum-helper

Setting Yum Options

In bcfg2.conf, any options you set in the [packages:yum] section other than use_yum_libraries and helper will be passed along verbatim to the configuration of the Yum objects used in the Bcfg2 server. The following options are set by default, and should not generally be overridden:

• cachedir is set to a hashed value unique to each distinct Yum configuration. Don’t set this unless you know what you’re doing.
• keepcache is set to 0; there is no benefit to changing this.
• sslverify is set to 0; change this if you know what you’re doing.
• reposdir is set to /dev/null to prevent the server’s Yum configuration from being read; do not change this.

Abstract Package Tags

If you are using the native Yum libraries, the abstract Package tag supports several attributes in addition to the standard name:

complexType PackageStructure

Abstract description of a package or package group to be installed.

Attributes:

Name	Description	Values	Required	Default
group	Install the named package group. Package groups are only supported for Pac and Yum Source repositories. Either group or name must be specified.	string	No	None
name	Install the named package. Either name or group must be specified.	string	No	None
recommended	Whether also the recommended packages should be installed. This is currently only used with the APT driver.	true | false	No	None
type	The package set to select from a given package group. Only meaningful if group is specified.	optional | default | mandatory | all	No	default
verify	Whether or not to verify the package.	true | false	No	true

Attribute groups:

• py:genshiAttrs

Pulp Support

New in version 1.2.0.

Bcfg2 contains explicit support for repositories managed by Pulp (http://pulpproject.org/).

Note

Only the Pulp 1.x API is supported at this time. When the Pulp 2.x API is finalized support will be added for it.

Due to the amount of data about a repository that can be retrieved directly from Pulp, the only thing necessary to configure a Pulp repo is the repo ID, in pulp_id:

<Sources>
  <Group name="centos-6-x86_64">
    <Source type="yum" pulp_id="centos-6-x86_64-os"/>
    <Source type="yum" pulp_id="centos-6-x86_64-updates"/>
    <Source type="yum" pulp_id="centos-6-x86_64-extras"/>
  </Group>
</Sources>

Pulp sources require some additional configuration. First, the Bcfg2 server must have a valid /etc/pulp/consumer/consumer.conf that is readable by the user your Bcfg2 server runs as; the Pulp server, URLs, and so on, are determined from this.

Secondly, in bcfg2.conf you must set the following options in the [packages:pulp] section:

• username and password: The username and password of a Pulp user that will be used to register new clients and bind them to repositories. Membership in the default consumer-users role is sufficient.

Bcfg2 clients using Pulp sources will be registered to the Pulp server as consumers, and will be bound to the appropriate repositories.

Debugging unexpected behavior

New in version 1.2.1.

Using bcfg2-info

The dependency resolver used in Packages can be run in debug mode:

$ bcfg2-info packageresolve foo.example.com bcfg2-server zlib
...
2 initial packages
    bcfg2-server
    zlib
54 new packages added
    sqlite
    less
    libxml2
    expat
    ...
1 unknown packages
    libglib-2.0.so.0()(64bit)

This will show why the resolver is acting as it is. Replace foo.example.com and bcfg2-server with a client name and list of packages, respectively.

Note that resolving a partial package list (as above) may result in more unknown entries than you’d have otherwise; some of the package drivers (Yum in particular) consider the full package list when resolving multiple providers, and will not be able to properly resolve some dependencies without a full package list.

You can also view the sources applicable to a client:

$ bcfg2-info packagesources foo.example.com
...
Name: centos-6-x86_64-updates
  Type: yum
  URL: http://mirror.example.com/centos-6-x86_64-updates
  GPG Key(s): http://mirror.example.com/centos-6-x86_64-updates/RPM-GPG-KEY-CentOS-6

Name: centos-6-x86_64-os
  Type: yum
  URL: http://mirror.example.com/centos-6-x86_64-os
  GPG Key(s): http://mirror.example.com/centos-6-x86_64-os/RPM-GPG-KEY-CentOS-6

Using bcfg2-server

Once the server is started, enable debugging via bcfg2-admin:

$ bcfg2-admin xcmd Packages.toggle_debug

TODO list

• Zypper support
• Portage support

Configuration

bcfg2.conf contains miscellaneous configuration options for the Packages plugin. Any booleans in the config file accept the values “1”, “yes”, “true”, and “on” for True, and “0”, “no”, “false”, and “off” for False. For historical reasons, resolver and metadata also accept “enabled” and “disabled”.

It understands the following directives:

[packages] section

Name	Description	Values	Default
backends	List of backends that should be loaded for the dependency resolution.	List	Yum,Apt,Pac,Pkgng
resolver	Enable dependency resolution	Boolean	True
metadata	Enable metadata processing. Disabling metadata implies disabling resolver as well.	Boolean	True
yum_config	The path at which to generate Yum configs.	String	/etc/yum.repos.d/bcfg2.repo
apt_config	The path at which to generate APT configs.	String	/etc/apt/sources.list.d/bcfg2-packages-generated-sources.list
gpg_keypath	The path on the client RPM GPG keys will be copied to before they are imported on the client.	String	/etc/pki/rpm-gpg
version	Set the version attribute used when binding Packages	any|auto	auto
cache	Path where Packages will store its cache	String	<repo>/Packages/cache

[packages:yum] section

Name	Description	Values	Default
use_yum_libraries	Whether or not to use the native yum library support	Boolean	False
helper	Path to bcfg2-yum-helper	String	See below

To find bcfg2-yum-helper if none is specified, Bcfg2 looks first in $PATH and then in /usr/sbin/bcfg2-yum-helper for the helper.

All other options in the [packages:yum] section will be passed along verbatim to the Yum configuration if you are using the native Yum library support.

[packages:pulp] section

Name	Description	Values	Default
username	The username of a Pulp user that will be used to register new clients and bind them to repositories.	String	None
password	The password of the Pulp user	String	None

The user should be a member of the default consumer-users role.