Packages¶

New in version 1.0.0.

This page documents the Packages plugin. Packages is an alternative to Pkgmgr for specifying package entries for clients. Where Pkgmgr explicitly specifies package entry information, Packages delegates control of package version information to the underlying package manager, installing the latest version available through those channels.

“Magic Groups”¶

Packages is the only plugin that uses “magic groups”. Most plugins operate based on client group memberships, without any concern for the particular names chosen for groups by the user. The Packages plugin is the sole exception to this rule. Packages needs to “know” two different sorts of facts about clients. The first is the basic OS/distro of the client, enabling classes of sources. The second is the architecture of the client, enabling sources for a given architecture. In addition to these magic groups, each source may also specify non-magic groups to limit the source’s applicability to group member clients.

Source	OS Group	Architecture
Apt	debian	i386
Apt	ubuntu	amd64
Apt	nexenta
Apt	apt
Yum	redhat	i386
Yum	centos	x86_64
Yum	fedora
Yum	yum

Note

New in version 1.2.0.

Magic OS groups can be disabled in Bcfg2 1.2 and greater by setting magic_groups to 0 in Packages/packages.conf. This may give you greater flexibility in determining which source types to use for which OSes. Magic architecture groups cannot be disabled.

Limiting sources to groups¶

Packages/sources.xml processes <Group> and <Client> tags just like Bundles. In addition to any groups or clients specified that way, clients must be a member of the appropriate architecture group as specified in a Source stanza. In total, in order for a source to be associated with a client, the client must be in one of the magic groups (debian, ubuntu, or nexenta), any explicit groups or clients specified in sources.xml, and any specified architecture groups.

Memberships in architecture groups is needed so that Packages can map software sources to clients. There is no other way to handle this than to impose membership in the appropriate architecture group.

When multiple sources are specified, clients are associated with each source to which they apply (based on group memberships, as described above). Packages and dependencies are resolved from all applicable sources.

Note

To recap, a client needs to be a member of the OS Group, Architecture group, and any other groups defined in your Packages/sources.xml file in order for the client to be associated to the proper sources.

Setup¶

Three basic steps are required for Packages to work properly.

Create Packages/sources.xml. This file should look approximately like the example below, and describes both which software repositories should be used, and which clients are eligible to use each one.
Ensure that clients are members of the proper groups. Each client should be a member of one of the magic groups listed above, all of the groups listed in the sources.xml (like ubuntu-intrepid or centos-5.2 in the following examples), and one of the architecture groups listed in the source configuration (i386, amd64 or x86_64 in the following examples). ‘’‘Failure to do this will result in the source either not applying to the client, or only architecture independent packages being made available to the client.’‘’
Add Package entries to bundles.
Sit back and relax, as dependencies are resolved, and automatically added to client configurations.

Prerequisite Resolution¶

Packages provides a prerequisite resolution mechanism which has no analogue in Pkgmgr. During configuration generation, all structures are processed. After this phase, but before entry binding, a list of packages and the client metadata instance is passed into Packages’ resolver. This process determines a superset of packages that will fully satisfy dependencies of all package entries included in structures, and reports any prerequisites that cannot be satisfied. This facility should largely remove the need to use the Base plugin.

Disabling dependency resolution¶

New in version 1.1.0.

Dependency resolution can be disabled by adding this to Packages/packages.conf in the global section:

[global]
resolver=0

All metadata processing can be disabled as well:

[global]
metadata=0

This setting implies disabling the resolver.

Blacklisting faulty dependencies¶

If you encounter an issue with faulty dependency resolution due to Packages, please file a bug report so that we can fix the problem in future releases. In the meantime, you can work around this issue by blacklisting the offending Package in your Sources. The blacklist element should immediately follow the Component section of your source and should look like the following:

<Blacklist>unwanted-packagename</Blacklist>

If you use the built-in Yum config generator, blacklisted packages will be added to the exclude list for the source.

Handling GPG Keys¶

New in version 1.2.0.

If you have yum libraries installed, Packages can automatically handle GPG signing keys for Yum and Pulp repositories. (You do not need to use the native yum resolver; if yum libraries are available, GPG signing keys can be handled automatically.) Simply specify the URL to the GPG key(s) for a repository in sources.xml:

<Source type="yum"
        rawurl="http://mirror.example.com/centos6-x86_64/RPMS.os">
  <Arch>x86_64</Arch>
  <GPGKey>http://mirror.example.com/keys/RPM-GPG-KEY-CentOS-6</GPGKey>
</Source>

More than one <GPGKey> tag can be specified per Source.

With the keys specified thusly, Packages will include the keys in the generated yum config file, and will ensure that the keys are imported on the client.

There is no need to specify <GPGKey> tags for Pulp sources; that data is pulled directly from the Pulp REST API.

Arbitrary Repo Options¶

New in version 1.2.3.

You can specify arbitrary options to be added to the repository config on the server side, if you are using the native yum libraries, and on the client side if you are using the ability of Packages to automatically generate your Yum config. To do this, add an <Options> tag to a Source; all of its attributes will be added verbatim to the repository in the generated config. For instance:

<Source type="yum" rawurl="http://mirror.example.com/centos-6-os">
  <Arch>x86_64</Arch>
  <Options proxy="http://proxy.example.com"/>
</Source>

If you are using native yum libraries and need to set options only on the Bcfg2 server, you can set the serveronly attribute to “true”; or, if you need to set options only on the client, you can set the clientonly attribute to “true”. For instance, if your Bcfg2 server needed to use a proxy to access a repo, and you wanted to expire metadata caches very quickly on the client, you could do:

<Source type="yum" rawurl="http://mirror.example.com/centos-6-os">
  <Arch>x86_64</Arch>
  <Options serveronly="true" proxy="http://proxy.example.com"/>
  <Options clientonly="true" metadata_expire="0"/>
</Source>

Example usage¶

Create a sources.xml file in the Packages directory that looks something like this:

<Sources>
  <Group name="ubuntu-intrepid">
    <Source type="apt"
            url="http://us.archive.ubuntu.com/ubuntu"
            version="intrepid">
      <Component>main</Component>
      <Component>universe</Component>
      <Arch>i386</Arch>
      <Arch>amd64</Arch>
    </Source>
  </Group>
</Sources>

Note

New in version 1.1.0.

The default behavior of the Packages plugin is to not make any assumptions about which packages you want to have added automatically. For that reason, neither Recommended nor Suggested packages are added as dependencies by default. You will notice that the default behavior for apt is to add Recommended packages as dependencies. You can configure the Packages plugin to add recommended packages by adding the recommended attribute, e.g.:

<Source type="apt" recommended="true" ...>

Warning

You must regenerate the Packages cache when adding or removing the recommended attribute.

Yum sources can be similarly specified:

<Sources>
  <Group name="centos-5.2">
    <Source type="yum"
            url="http://mirror.centos.org/centos/"
            version="5.2">
      <Component>os</Component>
      <Component>updates</Component>
      <Component>extras</Component>
      <Arch>i386</Arch>
      <Arch>x86_64</Arch>
      <GPGKey>http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5</GPGKey>
    </Source>
  </Group>
</Sources>

For sources with a URL attribute, the Version attribute is also necessary.

Pulp sources are very simple to specify due to the amount of data that can be queried from Pulp itself:

<Sources>
  <Group name="centos-6-x86_64">
    <Source type="yum" pulp_id="centos-6-x86_64-os"/>
    <Source type="yum" pulp_id="centos-6-x86_64-updates"/>
    <Source type="yum" pulp_id="centos-6-x86_64-extras"/>
  </Group>
</Sources>

Note

There is also a rawurl attribute for specifying sources that don’t follow the conventional layout.

<Sources>
  <Group name="centos5.4">
    <Source type="yum"
            rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.os">
      <Arch>x86_64</Arch>
    </Source>
    <Source type="yum"
            rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.updates">
      <Arch>x86_64</Arch>
    </Source>
    <Source type="yum"
            rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.extras">
      <Arch>x86_64</Arch>
    </Source>
  </Group>
</Sources>

<Sources>
  <Group name="ubuntu-lucid">
    <Source type="apt"
            rawurl="http://hudson-ci.org/debian/binary">
      <Arch>amd64</Arch>
    </Source>
    <Source type="apt"
            rawurl=http://hudson-ci.org/debian/binary">
      <Arch>i386</Arch>
    </Source>
  </Group>
</Sources>

Configuration Updates¶

Packages will reload its configuration upon an explicit command via bcfg2-admin:

[0:3711] bcfg2-admin xcmd Packages.Refresh
True

During this command (which will take some time depending on the quantity and size of the sources listed in the configuration file), the server will report information like:

Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-i386/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-amd64/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-i386/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-amd64/Packages.gz
...
Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/filelists.xml.gz
Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/primary.xml.gz

Once line per file download needed. Packages/sources.xml will be reloaded at this time, so any source specification changes (new or modified sources in this file) will be reflected by the server at this point.

This process is much, much faster if you use the native yum library support.

Soft reload¶

New in version 1.2.0.

A soft reload can be performed to reread the configuration file and download only missing sources.:

[0:3711] bcfg2-admin xcmd Packages.Reload
True

This is done automatically any time Packages/sources.xml is updated.

Availability¶

Support for clients using yum and apt is currently available. Support for other package managers (Portage, Zypper, IPS, etc) remain to be added.

Validation¶

A schema for Packages/sources.xml is included; sources.xml can be validated using bcfg2-lint.

Note

The schema requires that elements be specified in the above order.

Limitations¶

Packages does not do traditional caching as other plugins do. Modifying sources in the Packages sources.xml file requires a server restart for the time being. You do not have to restart the server after changing packages.conf or after adding new sources to sources.xml.

Package Checking and Verification¶

In order to do disable per-package verification Pkgmgr style, you will need to use BoundEntries, e.g.:

<BoundPackage name="mem-agent" priority="1" version="auto"
              type="yum" verify="false"/>

Generating Client APT/Yum Configurations¶

New in version 1.2.0.

The Packages plugin has native support for generating Yum configs. You must set yum_config in Packages/packages.conf to the path to the yum config file you want to generate:

[global]
yum_config=/etc/yum.repos.d/all.repo

Then add the corresponding Path entry to your Yum bundle.

New in version 1.1.0.

APT repository information can be generated automatically from software sources using TGenshi or TCheetah. A list of source urls are exposed in the client’s metadata as metadata.Packages.sources. E.g.:

# bcfg2 maintained apt

{% for s in metadata.Packages.sources %}\
deb ${s.url}${s.version} ${s.groups[0]} {% for comp in s.components %}$comp {% end %}

{% end %}\

Using Native Yum Libraries¶

New in version 1.2.0.

By default, Bcfg2 uses an internal implementation of Yum’s dependency resolution and other routines so that the Bcfg2 server can be run on a host that does not support Yum itself. If you run the Bcfg2 server on a machine that does have Yum libraries, however, you can enable use of those native libraries in Bcfg2 by setting use_yum_libraries to 1 in the [yum] section of Packages/packages.conf.

Benefits to this include:

Much lower memory usage by the bcfg2-server process.
Much faster Packages.Refresh behavior.
More accurate dependency resolution.
Support for package groups.

Drawbacks include:

More disk I/O. In some cases, you may have to raise the open file limit for the user who runs your Bcfg2 server process, particularly if you have a lot of repositories.
Resolution of package dependencies is slower in some cases, particularly after running Packages.Refresh.
If you have a very large number of clients using a very small number of repositories, using native yum libraries may actually increase memory usage.

Configuring the Yum Helper¶

Due to poor memory management by the Yum API, the long-lived bcfg2-server process uses an external short-lived helper, bcfg2-yum-helper, to do the actual Yum API calls for native yum library support. By default, Bcfg2 looks for this helper in $PATH, or, failing that, at /usr/sbin/bcfg2-yum-helper. If you have installed the helper elsewhere, you will need to configure that location with the helper option in the [yum] section, e.g.:

[yum]
use_yum_libraries = 1
helper = /usr/local/sbin/bcfg2-yum-helper

Setting Yum Options¶

In Packages/packages.conf, any options you set in the [yum] section other than use_yum_libraries and helper will be passed along verbatim to the configuration of the Yum objects used in the Bcfg2 server. The following options are set by default, and should not generally be overridden:

cachedir is set to a hashed value unique to each distinct Yum configuration. Don’t set this unless you know what you’re doing.
keepcache is set to 0; there is no benefit to changing this.
sslverify is set to 0; change this if you know what you’re doing.
reposdir is set to /dev/null to prevent the server’s Yum configuration from being read; do not change this.

Package Groups¶

Yum package groups are supported by the native Yum libraries. To include a package group, use the group attribute of the Package tag. You can use either the short group ID or the long group name:

<Package group="SNMP Support"/>
<Package group="system-management-snmp"/>

By default, only those packages considered the “default” packages in a group will be installed. You can change this behavior using the “type” attribute:

<Package group="development" type="optional"/>
<Package group="Administration Tools" type="mandatory"/>

Valid values of “type” are:

mandatory: Only install mandatory packages in the group.
default: Install default packages from the group (the default).
optional or all: Install all packages in the group, including mandatory, default, and optional packages.

You can view the packages in a group by category with the yum groupinfo command. More information about the different levels can be found at http://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups#Installation

Pulp Support¶

New in version 1.2.0.

Bcfg2 contains explicit support for repositories managed by Pulp (http://pulpproject.org/). Due to the amount of data about a repository that can be retrieved directly from Pulp, the only thing necessary to configure a Pulp repo is the repo ID:

<Sources>
  <Group name="centos-6-x86_64">
    <Source type="yum" pulp_id="centos-6-x86_64-os"/>
    <Source type="yum" pulp_id="centos-6-x86_64-updates"/>
    <Source type="yum" pulp_id="centos-6-x86_64-extras"/>
  </Group>
</Sources>

Pulp sources require some additional configuration. First, the Bcfg2 server must have a valid /etc/pulp/consumer/consumer.conf that is readable by the user your Bcfg2 server runs as; the Pulp server, URLs, and so on, are determined from this.

Secondly, in Packages/packages.conf you must set the following options in the [pulp] section:

username and password: The username and password of a Pulp user that will be used to register new clients and bind them to repositories. Membership in the default consumer-users role is sufficient.

Bcfg2 clients using Pulp sources will be registered to the Pulp server as consumers, and will be bound to the appropriate repositories.

Debugging unexpected behavior¶

New in version 1.2.1.

Using bcfg2-info¶

The dependency resolver used in Packages can be run in debug mode:

$ bcfg2-info packageresolve foo.example.com bcfg2-server zlib
...
2 initial packages
    bcfg2-server
    zlib
54 new packages added
    sqlite
    less
    libxml2
    expat
    ...
1 unknown packages
    libglib-2.0.so.0()(64bit)

This will show why the resolver is acting as it is. Replace foo.example.com and bcfg2-server with a client name and list of packages, respectively.

Note that resolving a partial package list (as above) may result in more unknown entries than you’d have otherwise; some of the package drivers (Yum in particular) consider the full package list when resolving multiple providers, and will not be able to properly resolve some dependencies without a full package list.

You can also view the sources applicable to a client:

$ bcfg2-info packagesources foo.example.com
...
Name: centos-6-x86_64-updates
  Type: yum
  URL: http://mirror.example.com/centos-6-x86_64-updates
  GPG Key(s): http://mirror.example.com/centos-6-x86_64-updates/RPM-GPG-KEY-CentOS-6

Name: centos-6-x86_64-os
  Type: yum
  URL: http://mirror.example.com/centos-6-x86_64-os
  GPG Key(s): http://mirror.example.com/centos-6-x86_64-os/RPM-GPG-KEY-CentOS-6

Using bcfg2-server¶

Once the server is started, enable debugging via bcfg2-admin:

$ bcfg2-admin xcmd Packages.toggle_debug

TODO list¶

Zypper support
Portage support
Explicit version pinning (a la Pkgmgr)

Developing for Packages¶

In order to support a given client package tool driver, that driver must support use of the auto value for the version attribute in Package entries. In this case, the tool driver views the current state of available packages, and uses the underlying package manager’s choice of correct package version in lieu of an explicit, centrally-specified, version. This support enables Packages to provide a list of Package entries with version=’auto’. Currently, the APT and YUMng drivers support this feature. Note that package management systems without any network support cannot operate in this fashion, so RPMng and SYSV will never be able to use Packages. Emerge, Zypper, IPS, and Blastwave all have the needed features to be supported by Packages, but support has not yet been written.

Packages fills two major functions in configuration generation. The first is to provide entry level binding support for Package entries included in client configurations. This function is quite easy to implement; Packages determines (based on client group membership) if the package is available for the client system, and which type it has. Because version=’auto’ is used, no version determination needs to be done.

The second major function is more complex. Packages ensures that client configurations include all package-level prerequisites for package entries explicitly included in the configuration. In order to support this, Packages needs to directly process network data for package management systems (the network sources for apt or yum, for examples), process these files, and build data structures describing prerequisites and the providers of those functions/paths. To simplify implementations of this, there is a generic base class (Bcfg2.Server.Plugins.Packages.Source) that provides a framework for fetching network data via HTTP, processing those sources (with subclass defined methods for processing the specific format provided by the tool), a generic dependency resolution method, and a caching mechanism that greatly speeds up server/bcfg2-info startup.

Each source type must define:

a get_urls attribute (and associated urls property) that describes the URLS where to get data from.
a read_files method that reads and processes the downloaded files

Sources may define a get_provides method, if provides are complex. For example, provides in rpm can be either rpm names or file paths, so multiple data sources need to be multiplexed.

The APT source in src/lib/Server/Plugins/Packages.py provides a relatively simple implementation of a source.

packages.conf¶

packages.conf contains miscellaneous configuration options for the Packages plugin. Any booleans in the config file accept the values “1”, “yes”, “true”, and “on” for True, and “0”, “no”, “false”, and “off” for False

It understands the following directives:

[global] section¶

resolver: Enable dependency resolution. Default is 1 (true). For historical reasons, this also accepts “enabled” and “disabled”.
metadata: Enable metadata processing. Default is 1 (true). If metadata is disabled, it’s implied that resolver is also disabled. For historical reasons, this also accepts “enabled” and “disabled”.
yum_config: The path at which to generate Yum configs. No default.
apt_config: The path at which to generate APT configs. No default.
gpg_keypath: The path on the client RPM GPG keys will be copied to before they are imported on the client. Default is “/etc/pki/rpm-gpg”.
version: Set the version attribute used when binding Packages. Default is auto.

[yum] section¶

use_yum_libraries: Whether or not to use the native yum library support. Default is 0 (false).

All other options in the [yum] section will be passed along verbatim to the Yum configuration if you are using the native Yum library support.

[pulp] section¶

username and password: The username and password of a Pulp user that will be used to register new clients and bind them to repositories. Membership in the default consumer-users role is sufficient.

Packages¶

“Magic Groups”¶

Limiting sources to groups¶

Setup¶

Prerequisite Resolution¶

Disabling dependency resolution¶

Blacklisting faulty dependencies¶

Handling GPG Keys¶

Arbitrary Repo Options¶

Example usage¶

Configuration Updates¶

Soft reload¶

Availability¶

Validation¶

Limitations¶

Package Checking and Verification¶

Generating Client APT/Yum Configurations¶

Using Native Yum Libraries¶

Configuring the Yum Helper¶

Setting Yum Options¶

Package Groups¶

Pulp Support¶

Debugging unexpected behavior¶

Using bcfg2-info¶

Using bcfg2-server¶

TODO list¶

Developing for Packages¶

packages.conf¶

[global] section¶

[yum] section¶

[pulp] section¶

Table Of Contents

Previous topic

Next topic

This Page

Navigation

Packages¶

“Magic Groups”¶

Limiting sources to groups¶

Setup¶

Prerequisite Resolution¶

Disabling dependency resolution¶

Blacklisting faulty dependencies¶

Handling GPG Keys¶

Arbitrary Repo Options¶

Example usage¶

Configuration Updates¶

Soft reload¶

Availability¶

Validation¶

Limitations¶

Package Checking and Verification¶

Generating Client APT/Yum Configurations¶

Using Native Yum Libraries¶

Configuring the Yum Helper¶

Setting Yum Options¶

Package Groups¶

Pulp Support¶

Debugging unexpected behavior¶

Using bcfg2-info¶

Using bcfg2-server¶

TODO list¶

Developing for Packages¶

packages.conf¶

[global] section¶

[yum] section¶

[pulp] section¶

Table Of Contents

Previous topic

Next topic

This Page

Quick search

Navigation