.. -*- mode: rst -*- .. _client-tools-yumng: ================================ Bcfg2 RPMng/YUMng Client Drivers ================================ Introduction ============ The goal of this driver is to resolve the issues that exist with the RPM and Yum client tool drivers. For the most part, the issues are due to RPM being able to have multiple packages of the same name installed. This is an issue on all Red Hat and SUSE based distributions. Examples of this are: * SLES10 and openSUSE 10.2 both install six GPG keys. From an RPM perspective this means that there are six packages with the name gpg-pubkey. * YUM always installs, as opposed to upgrades, kernel packages. This is hard coded in YUM (actually it can be overridden in yum.conf), so systems using YUM will eventually have multiple kernel packages installed. * Red Hat family x86_64 based systems frequently have both an x86_64 and an i386 version of the same package installed. The new Pkgmgr format files with Instances are therefore the only way to accurately describe an RPM based system. It is recommended that all RPM based systems be changed to use the new format configuration files and the RPMng driver. Alternatively, you can use the newer :ref:`Packages ` plugin. Development Status ================== Initial development of the drivers was done on Centos 4.4 x86_64, with testing on openSUSE 10.2 x86_64. Centos has been tested with a new style Pkgmgr file and openSUSE with an old style file (see the Configuration section below for what this means). Testing has now moved to Centos 5 x86_64 and old style files are no longer being tested. RPMng/YUMng are the default RPM drivers. Features ======== * Limited support for 0.9.4 and earlier Pkgmgr configuration files. See Configuration below for details. * Full RPM package identification using epoch, version, release and arch. * Support for multiple instances of packages with the Instance tag. * Better control of the RPM verification using the pkg_checks, pkg_verify and verify_flags attributes. * Support for install only packages such as the kernel packages. * Support for per instance ignoring of individual files for the RPM verification with the Ignore tag. * Multiple package Instances with full version information listed in interactive mode. * Support for installation and removal of gpg-pubkey packages. * Support for controlling what action is taken on package verification failure with the install_action, version_fail_action and verify_fail_action attributes. RPMng Driver Overview ===================== The RPMng driver uses a mixture of rpm commands and rpm-python as detailed in the sections below. rpmtools module --------------- The rpmtools module conatins most of the rpm-python code and is imported by RPMng.py and YUMng.py. RPMng.RefreshPackages() ----------------------- The RPMng.RefreshPackages method generates the installed dict using rpm-python code from the rpmtools module. Full name, epoch, version, release and arch information is stored. RPMng.VerifyPackages() ---------------------- The RPMng.VerifyPackages method generates a number of structures that record the state of the of the system compared to the Bcfg2 literal configuration retrieved from the server. These structures are mainly used by the RPMng.Install method. AS part of the verification process an rpm package level verification is carried out using rpm-python code from the rpmtools module. Full details of the failures are returned in a complicated dict/list structure for later use. RPMng.Install() --------------- The RPMng.Install method attempts to fix what the RPMng.VerifyPackages method found wrong. It does this by installing, reinstalling, deleting and upgrading RPMs. RPMng.Install does not use rpm-python. It does use the following rppm commands as appropriate:: rpm -install rpm --import rpm -upgrade A method (RPMng.to reinstall_check()) to decide whether to do a reinstall of a package instance or not has been added, but is very simple at this stage. Currently it will prevent a reinstall if the only reason for a verification failure was due to an RPM configuration (%config) file. A package reinstall will not replace these, so there is no point reinstalling. RPMng.Remove() -------------- The RPMng.Remove method is written using rpm-python code in the rpmtools module. Full nevra information is used in the selection of the package removal. Installation ============ isprelink --------- This is a Python C extension module that checks to see if a file has been prelinked or not. It should be built and installed on systems that have the prelink package installed (only Red Hat family systems as far as I can tell). rpmtools will function without the isprelink module, but performance is not good. Source can be found here ftp://ftp.mcs.anl.gov/pub/bcfg/isprelink-0.1.2.tar.gz To compile and install prelink, execute:: python setup.py install in the rpmtools directory. The elfutils-libelf-devel package is required for the compilation. There are Centos x86_64 RPMs here ftp://ftp.mcs.anl.gov/pub/bcfg/redhat/ Configuration and Usage ======================= Loading of RPMng ---------------- The RPMng driver can be loaded by command line options, client configuration file options or as the default driver for RPM packages. From the command line:: bcfg2 -n -v -d -D Action,POSIX,Chkconfig,RPMng This produces quite a bit of output so you may want to redirect the output to a file for review. In the ``bcfg2.conf`` file:: [client] #drivers = Action,Chkconfig,POSIX,YUMng drivers = Action,Chkconfig,POSIX,RPMng .. note:: Note that loading this driver will unload the RPM driver, so the Yum driver will not work. Configuration File Options -------------------------- A number of paramters can be set in the client configuration for both the RPMng and YUMng drivers. Each driver has its own section. A full client configuration file with all the options specified is below:: [communication] protocol = xmlrpc/ssl password = xxxxxx user = yyyyyyy [components] bcfg2 = https://bcfg2:6789 [client] #drivers = Action,Chkconfig,POSIX,YUMng drivers = Action,Chkconfig,POSIX,RPMng [RPMng] pkg_checks = true pkg_verify = true erase_flags = allmatches installonlypackages = kernel, kernel-bigmem, kernel-enterprise, kernel-smp, kernel-modules, kernel-debug, kernel-unsupported, kernel-source, kernel-devel, kernel-default, kernel-largesmp-devel, kernel-largesmp, kernel-xen, gpg-pubkey install_action = install version_fail_action = upgrade verify_fail_action = reinstall [YUMng] pkg_checks = True pkg_verify = true erase_flags = allmatches autodep = true installonlypackages = kernel, kernel-bigmem, kernel-enterprise, kernel-smp, kernel-modules, kernel-debug, kernel-unsupported, kernel-source, kernel-devel, kernel-default, kernel-largesmp-devel, kernel-largesmp, kernel-xen, gpg-pubkey install_action = install version_fail_action = upgrade verify_fail_action = reinstall installOnlyPkgs ^^^^^^^^^^^^^^^ Install only packages are packages that should only ever be installed or deleted, not upgraded. The only packages for which this is an absolute on, are the gpg-pubkey packages. It is however 'best' practice to only ever install/delete kernel packages. The wisdom being that the package for the currently running kernel should always be installed. Doing an upgrade would delete the running kernel package. The RPMng driver follows the YUM practice of having a list of install only packages. A default list is hard coded in RPMng.py. This maybe over ridden in the client configuration file. Note that except for gpg-pubkey packages (which are always added to the list by the driver) the list in the client configuration file completely replaces the default list. An empty list means that there are no install only packages (except for gpg-pubkey), which is the behaviour of the old RPM driver. Example - an empty list:: [RPMng] installonlypackages = Example - The default list:: [RPMng] installonlypackages = kernel, kernel-bigmem, kernel-enterprise, kernel-smp, kernel-modules, kernel-debug, kernel-unsupported, kernel-source, kernel-devel, kernel-default, kernel-largesmp-devel, kernel-largesmp, kernel-xen, gpg-pubkey erase_flags ^^^^^^^^^^^ erase_flags are rpm options used by 'rpm -erase' in the client Remove() method. The RPMng erase is written using rpm-python and does not use the rpm command. The erase flags are specified in the client configuration file as a comma separated list and apply to all RPM erase operations. The default is:: [RPMng] erase_flags = allmatches The following rpm erase options are supported, see the rpm man page for details.:: noscripts notriggers repackage allmatches nodeps .. note:: Note that specifying erase_flags in the configuration file completely replaces the default. pkg_checks ^^^^^^^^^^ The RPMng/YUMng drivers do the following three checks/status: #. Installed #. Version #. rpm verify Setting pkg_checks = true (the default) in the client configuration file means that all three checks will be done for all packages. Setting pkg_checks = false in the client configuration file means that only the Installed check will be done for all packages. The true/false value can be any combination of upper and lower case. .. note:: #. pkg_checks must evaluate true for both the client (this option) and the package (see the Package Tag pkg_checks attribute below) for the action to take place. #. If pkg_checks = false then the Pkgmgr entries do not need the version information. See the examples towards the bottom of the page. pkg_verify ^^^^^^^^^^ The RPMng/YUMng drivers do the following three checks/status: #. Installed #. Version #. rpm verify Setting pkg_verify = true (the default) in the client configuration file means that all three checks will be done for all packages as long as pkg_checks = true. Setting pkg_verify = false in the client configuration file means that the rpm verify wil not be done for all packages on the client. The true/false value can be any combination of upper and lower case. .. note:: #. pkg_verify must evaluate true for both the client (this option) and the package instance (see the Instance Tag pkg_verify attribute below) for the action to take place. install_action ^^^^^^^^^^^^^^ The RPMng/YUMng drivers do the following three checks/status: #. Installed #. Version #. rpm verify install_action controls whether or not a package instance will be installed if the installed check fails (i.e. if the package instance isn't installed). If install_action = install then the package instance is installed. If install_action = none then the package instance is not installed. .. note:: #. install_action must evaluate true for both the client (this option) and the package instance (see the Instance Tag install_action attribute below) for the action to take place. version_fail_action ^^^^^^^^^^^^^^^^^^^ The RPMng/YUMng drivers do the following three checks/status: #. Installed #. Version #. rpm verify version_fail_action controls whether or not a package instance will be updated if the version check fails (i.e. if the installed package instance isn't the same version as specified in the configuration). If version_fail_action = upgrade then the package instance is upgraded (or downgraded). If version_fail_action = none then the package instance is not upgraded (or downgraded). .. note:: #. verion_fail_action must evaluate true for both the client (this option) and the package instance (see the Instance Tag version_fail_action attribute below) for the action to take place. verify_fail_action ^^^^^^^^^^^^^^^^^^ The RPMng/YUMng drivers do the following three checks/status: #. Installed #. Version #. rpm verify verify_fail_action controls whether or not a package instance will be reinstlled if the version check fails (i.e. if the installed package instance isn't the same version as specified in the configuration). If verify_fail_action = reinstall then the package instance is reinstalled. If verify_fail_action = none then the package instance is not reinstalled. .. note:: #. verify_fail_action must evaluate true for both the client (this option) and the package instance (see the Instance Tag verify_fail_action attribute below) for the action to take place. #. yum cannot reinstall packages, so this option is really only relevant to RPMng. #. RPMng will not attempt to reinstall a package instance if the only failure is an RPM configuration file. #. RPMng will not attempt to reinstall a package instance if the only failure is an RPM dependency failure. Interactive Mode ---------------- Running the client in interactive mode (-I) prompts for the actions to be taken as before. Prompts are per package and may apply to multiple instances of that package. Each per package prompt will contain a list of actions per instance. Actions are encoded as D - Delete I - Install R - Reinstall U - Upgrade/Downgrade An example is below. The example is from a system that is still using the old Pkgmgr format, so the epoch and arch appear as '*'.:: Install/Upgrade/delete Package aaa_base instance(s) - R(*:10.2-38.*) (y/N) Install/Upgrade/delete Package evms instance(s) - R(*:2.5.5-67.*) (y/N) Install/Upgrade/delete Package gpg-pubkey instance(s) - D(*:9c800aca-40d8063e.*) D(*:0dfb3188-41ed929b.*) D(*:7e2e3b05-44748aba.*) D(*:a1912208-446a0899.*) D(*:9c777da4-4515b5fd.*) D(*:307e3d54-44201d5d.*) (y/N) Install/Upgrade/delete Package module-init-tools instance(s) - R(*:3.2.2-62.*) (y/N) Install/Upgrade/delete Package multipath-tools instance(s) - R(*:0.4.7-29.*) (y/N) Install/Upgrade/delete Package pam instance(s) - R(*:0.99.6.3-29.1.*) (y/N) Install/Upgrade/delete Package perl-AppConfig instance(s) - U(None:1.52-4.noarch -> *:1.63-17.*) (y/N) Install/Upgrade/delete Package postfix instance(s) - R(*:2.3.2-28.*) (y/N) Install/Upgrade/delete Package sysconfig instance(s) - R(*:0.60.4-3.*) (y/N) Install/Upgrade/delete Package udev instance(s) - R(*:103-12.*) (y/N) GPG Keys -------- GPG is used by RPM to 'sign' packages. All vendor packages are signed with the vendors GPG key. Additional signatures maybe added to the rpm file at the users discretion. It is normal to have multiple GPG keys installed. For example, SLES10 out of the box has six GPG keys installed. To the RPM database all GPG 'packages' have the name 'gpg-pubkey', which may be nothing like the name of the file specified in the rpm -import command. For example on Centos 4 the file name is RPM-GPG-KEY-centos4. For SLES10 this means that there are six packages with the name 'gpg-pubkey' installed. RPM does not check GPG keys at package installation, YUM does. RPMng uses the rpm command for installation and does not therefore check GPG signatures at package install time. RPMng uses rpm-python for verification and does by default do signature checks as part of the client Inventory process. To do the signature check the appropriate GPG keys must be installed. rpm-python is not very friendly if the required key(s) is not installed (it crashes the client). The RPMng driver detects, on a per package instance basis, if the appropriate key is installed. If it is not, a warning message is printed and the signature check is disabled for that package instance, for that client run only. GPG keys can be installed and removed by the RPMng driver. To install a GPG key configure it in Pkgmgr/Rules as a package and add gpg-pubkey to the clients abstract configuration. The gpg-pubkey package/instance is treated as an install only package. gpg-pubkey packages are installed by the RPMng driver with the rpm -import command. gpg-pubkey packages will be removed by ``bcfg2 -r packages`` if they are not in the clients configuration. .. code-block:: xml Example gpg-pubkey Pkgmgr configuration file. Pkgmgr Configuration -------------------- Also see the general :ref:`Pkgmgr ` and :ref:`server-plugins-structures-altsrc` pages. Package Tag (Old style) ^^^^^^^^^^^^^^^^^^^^^^^ Old style (meaning no Instance tag) Pkgmgr files have limited support. Specifically the multiarch and verify attributes are ignored. If multiarch type support is needed a new style format file must be used. If some control over the verification is needed, replace the verify attribute with the pkg_checks or verify_flags attributes. The pkg_checks and verify_flags attributes are detailed under the Instance tag heading. Package Tag (New Style) and Attributes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The new style package tag supports the name and pkg_checks attributes and requires the use of Instance tag entries. New style configuration files must be generated from the RPM headers. Either from RPM files or from the RPM DB. The included pkgmgr_gen.py can be used as a starting point for generating configuration files from directories of RPM package files. pkgmgr_gen.py --help for the options. The included pkgmgr_update.py can be used to update the package instance versions in configuration files from directories of package files. pkgmgr_update.py --help for the options. +------------+---------------------------------------+------------------------+ | Attribute | Description | Values | +============+=======================================+========================+ | name | Package name. | String | +------------+---------------------------------------+------------------------+ | pkg_checks | Do the version and rpm verify checks. | true(default) or false | +------------+---------------------------------------+------------------------+ Instance Tag and Attributes ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The instance tag supports the following attributes: +---------------------+----------------------------+--------------------------+ | Attribute | Description | Values | +=====================+============================+==========================+ | simplefile | Package file name. | String (see Notes below) | +---------------------+----------------------------+--------------------------+ | epoch | Package epoch. | String (numeric only) | | | | (optional) | +---------------------+----------------------------+--------------------------+ | version | Package version. | String | +---------------------+----------------------------+--------------------------+ | release | Package release. | String | +---------------------+----------------------------+--------------------------+ | arch | Package architecture. | Architecture String e.g. | | | | (i386|i586|i686|x86_64) | +---------------------+----------------------------+--------------------------+ | verify_flags | Comma separated list of | nodeps, nodigest, | | | rpm --verify options. See | nofiles, noscripts, | | | the rpm man page for their | nosignature, nolinkto, | | | details. | nomd5, nosize, nouser, | | | | nogroup, nomtime, | | | | nomode, nordev | +---------------------+----------------------------+--------------------------+ | pkg_verify | Do the rpm verify | true(default) or false | +---------------------+----------------------------+--------------------------+ | install_action | Install package instance | install(default) or none | | | if it is not installed. | | +---------------------+----------------------------+--------------------------+ | version_fail_action | Upgrade package if the | upgrade(default) or none | | | incorrect version is | | | | installed. | | +---------------------+----------------------------+--------------------------+ | verify_fail_action | Reinstall the package | reinstall(default) or | | | instance if the rpm verify | none | | | failed | | +---------------------+----------------------------+--------------------------+ .. note:: The simplefile attribute doesn't need to be just the filename, meaning the basename. It is joined with the uri attribute from the PackageList Tag to form the URL that the client will use to download the package. So the uri could just be the host portion of the url and simple file could be the directory path. e.g. .. code-block:: xml The values for epoch, version, release and arch attributes must come from the RPM header, not the RPM file name. Epoch is a strange thing. In short:: epoch not set == epoch=None < epoch='0' < epoch='1' and it is an int, but elementtree attributes have to be str or unicode, so the driver is constantly converting. Ignore Tag ^^^^^^^^^^ The Ignore tag is used to "mask out" individual files from the RPM verification. This is done by comparing the verification failure results with the Ignore tag name. If there is a match, that entry is not used by the client to determine if a package has failed verification. Ignore tag entries can be specified at both the Package level, in which case they apply to all Instances, and/or at the Instance level, in which case they only apply to that instance. Ignore tag entries are used by the RPMng driver. They can be specified in both old and new style Pkgmgr files. The Ignore Tag supports the following attributes: +-----------+-------------+--------+ | Attribute | Description | Values | +===========+=============+========+ | name | File name. | String | +-----------+-------------+--------+ Example .. code-block:: xml POSIX 'ignore' Path entries ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The YUMng analog to the Ignore Tag used by RPMng is the use of Path entries of type 'ignore'. The following shows an example for the centos-release package which doesn't verify if you remove the default repos and replace them with a custom repo. .. code-block:: xml Automated Generation of Pkgmgr Configuration Files -------------------------------------------------- The two utilities detailed below are provided in the tools directory of the source tarball. Also see the general :ref:`Pkgmgr ` and :ref:`server-plugins-structures-altsrc` pages. pkgmgr_gen.py ^^^^^^^^^^^^^ pkgmgr_gen will generate a Pkgmgr file from a list of directories containing RPMs or from a list of YUM repositories.:: [root@bcfg2 Pkgmgr]# pkgmgr_gen.py --help usage: pkgmgr_gen.py [options] options: -h, --help show this help message and exit -aARCHS, --archs=ARCHS Comma separated list of subarchitectures to include. The highest subarichitecture required in an architecture group should specified. Lower subarchitecture packages will be loaded if that is all that is available. e.g. The higher of i386, i486 and i586 packages will be loaded if -a i586 is specified. (Default: all). -dRPMDIRS, --rpmdirs=RPMDIRS Comma separated list of directories to scan for RPMS. Wilcards are permitted. -eENDDATE, --enddate=ENDDATE End date for RPM file selection. -fFORMAT, --format=FORMAT Format of the Output. Choices are yum or rpm. (Default: yum) -gGROUPS, --groups=GROUPS List of comma separated groups to nest Package entities in. -iINDENT, --indent=INDENT Number of leading spaces to indent nested entries in the output. (Default:4) -oOUTFILE, --outfile=OUTFILE Output file name. -P, --pkgmgrhdr Include PackageList header in output. -pPRIORITY, --priority=PRIORITY Value to set priority attribute in the PackageList Tag. (Default: 0) -rRELEASE, --release=RELEASE Which releases to include in the output. Choices are all or latest. (Default: latest). -sSTARTDATE, --startdate=STARTDATE Start date for RPM file selection. -uURI, --uri=URI URI for PackageList header required for RPM format ouput. -v, --verbose Enable verbose output. -yYUMREPOS, --yumrepos=YUMREPOS Comma separated list of YUM repository URLs to load. NOTE: Each URL must end in a '/' character. .. note:: The startdate and enddate options are not yet implemented. pkgmgr_update.py ---------------- pkgmgr_update will update the release (meaning the epoch, version and release) information in an existing Pkgrmgr file from a list of directories containing RPMs or from a list of YUM repositories. All Tags and other attributes in the existing file will remain unchanged.:: [root@bcfg2 Pkgmgr]# pkgmgr_update.py --help usage: pkgmgr_update.py [options] options: -h, --help show this help message and exit -cCONFIGFILE, --configfile=CONFIGFILE Existing Pkgmgr configuration file name. -dRPMDIRS, --rpmdirs=RPMDIRS Comma separated list of directories to scan for RPMS. Wilcards are permitted. -oOUTFILE, --outfile=OUTFILE Output file name or new Pkgrmgr file. -v, --verbose Enable verbose output. -yYUMREPOS, --yumrepos=YUMREPOS Comma separated list of YUM repository URLs to load. NOTE: Each URL must end in a '/' character. Pkgmgr Configuration Examples ----------------------------- verify_flags ^^^^^^^^^^^^ This entry was used for the Centos test client used during RPMng development. .. code-block:: xml Multiple Instances ^^^^^^^^^^^^^^^^^^ .. code-block:: xml Kernel ^^^^^^ .. note:: Multiple instances with the same architecture must be in the installOnlyPkgs list. .. code-block:: xml Per Instance Ignore ^^^^^^^^^^^^^^^^^^^ .. note:: In this case a per instance ignore is actually a bad idea as the verify failure is because of multiarch issues where the last package installed wins. So this would be better as a Package level ignore. Ignore tag entries only work with the RPMng driver. They do not appear to be supported in YUMng as of 1.0pre5. .. code-block:: xml pkg_checks ^^^^^^^^^^ If pkg_checks = false the version information is not required. If pkg_checks = true the full information is needed as normal. For YUMng a minimal entry is .. code-block:: xml In fact for YUMng, with pkg_checks = false, any combination of the nevra attributes that will build a valid yum package name (see the Misc heading on the yum man page) is valid. .. code-block:: xml For RPMng a minimal entry is .. code-block:: xml verify_fail_action ^^^^^^^^^^^^^^^^^^ The way I have Bcfg2 configured for my development systems. This way it reports bad, but doesn't do anything about it. .. code-block:: xml