The Packages plugin offers multiple backends to support different types of software repositories. New backends can be written to handle new types of software repositories.
Each new Packages backend must be contained in its own module in Bcfg2.Server.Plugins.Packages. Each module must implement two classes: A Bcfg2.Server.Plugins.Packages.Collection.Collection subclass called <module>Collection, and a Bcfg2.Server.Plugins.Packages.Source.Source subclass called <module>Source. E.g., the Bcfg2.Server.Plugins.Packages.Yum backend has Bcfg2.Server.Plugins.Packages.Yum.YumCollection and Bcfg2.Server.Plugins.Packages.Yum.YumSource objects. These interfaces are explained in detail below.
Collection objects represent the set of Bcfg2.Server.Plugins.Packages.Source.Source objects that apply to a given client, and can be used to query all software repositories for a client in aggregate. In some cases this can give faster or more accurate results.
In most cases, Collection methods have been designed to defer the call to the Sources in the Collection and aggregate the results as appropriate. The simplest Collection implemention is thus often a simple subclass that adds no additional functionality.
As noted above, the Collection object is written expressly so that you can subclass it and override no methods or attributes, and it will work by deferring all calls to the Source objects it contains. There are thus three approaches to writing a Collection subclass:
In either case, you may want to override Collection.get_groups(), Collection.get_group(), Collection.get_essential(), Collection.get_config(), Collection.filter_unknown(), and Collection.build_extra_structures().
Collection objects have to translate Bcfg2 entries, lxml.etree._Element objects, into objects suitable for use by the backend for resolving dependencies. This is handled by two functions:
Because of this translation layer, the return type of any functions below that return packages (e.g., Collection.get_group()) is actually indeterminate; they must return an object suitable for passing to Collection.packages_to_entry(). Similarly, functions that take a package as an argument (e.g., Collection.is_package()) take the appropriate package object. In the documentation below, the actual parameter return type (usually .``string``) used in this base implementation is noted, as well as this fact.
Bases: list, Bcfg2.Server.Plugin.base.Debuggable
Collection objects represent the set of Bcfg2.Server.Plugins.Packages.Source objects that apply to a given client, and can be used to query all software repositories for a client in aggregate. In some cases this can give faster or more accurate results.
| Parameters: |
|
|---|
Whether or not this Packages backend supports package groups
Get a list of group names used by this Collection type in resolution of “Magic Groups”.
The base implementation simply aggregates the results of Bcfg2.Server.Plugins.Packages.Source.Source.basegroups.
Add additional entries to the <Independent/> section of the final configuration. This can be used to handle, e.g., GPG keys and other entries besides packages that need to be handled for a complete client configuration.
| Parameters: | independent (lxml.etree._Element) – The XML tag to add extra entries to. This is modified in place. |
|---|
A list of the full path to all cachefiles used by this collection.
The base implementation simply aggregates Bcfg2.Server.Plugins.Packages.Source.Source.cachefile attributes.
A unique identifier for the set of sources contained in this Collection object. This is unique to a set of sources, not necessarily to the client, which lets clients with identical sources share cache data.
Build a complete list of all packages and their dependencies.
| Parameters: | packagelist (set of strings, but see Conversion Between Package Objects and XML Entries) – Set of initial packages computed from the specification. |
|---|---|
| Returns: | tuple of sets - The first element contains a set of strings (but see Conversion Between Package Objects and XML Entries) describing the complete package list, and the second element is a set of symbols whose dependencies could not be resolved. |
After complete(), filter out packages that appear in the list of unknown packages but should not be presented to the user. E.g., packages that you expect to be unknown.
The base implementation filters out packages that are expected to be unknown by any source in this collection.
| Parameters: | unknown (set of strings, but see Conversion Between Package Objects and XML Entries) – A set of unknown packages. The set should be modified in place. |
|---|
Get additional Bcfg2.Server.Plugin.interfaces.Connector data to be supplied to Bcfg2.Server.Plugins.Packages.Packages.get_additional_data() (and thence to client metadata objects).
The base implementation simply aggregates Bcfg2.Server.Plugins.Packages.Source.Source.url_map attributes.
| Returns: | list of additional Connector data |
|---|
Get the configuration for the package tool used by this source type. This should be a config appropriate for use on either the server (to resolve dependencies) or the client.
Subclasses must override this method in order to be able to generate configs. By default it logs an error and returns the empty string.
| Returns: | string |
|---|
Get a list of the dependencies of the given package.
The base implementation simply aggregates the results of Bcfg2.Server.Plugins.Packages.Source.Source.get_deps().
| Parameters: | package (string) – The name of the symbol, but see Conversion Between Package Objects and XML Entries |
|---|---|
| Returns: | list of strings, but see Conversion Between Package Objects and XML Entries |
Get a list of packages that are essential to the repository.
The base implementation simply aggregates Bcfg2.Server.Plugins.Packages.Source.Source.essentialpkgs attributes
| Returns: | list of strings, but see Conversion Between Package Objects and XML Entries |
|---|
Get the list of packages of the given type in a package group.
The base implementation simply aggregates the results of Bcfg2.Server.Plugins.Packages.Source.Source.get_group().
| Parameters: | |
|---|---|
| Returns: | list of strings - package names, but see Conversion Between Package Objects and XML Entries |
Given a list of package group names, return a dict of <group name>: <list of packages>. This method is provided since some backends may be able to query multiple groups at once faster than serially.
The base implementation simply aggregates the results of Bcfg2.Server.Plugins.Packages.Source.Source.get_group().
| Parameters: | grouplist (list of strings - group names) – The list of groups to query |
|---|---|
| Returns: | dict of <group name>: <list of packages> |
In this implementation the packages will be strings, but see Conversion Between Package Objects and XML Entries.
Compute the difference between the complete package list (as returned by complete()) and the initial package list computed from the specification. This is necessary because the format may be different between the two lists due to packages_to_entry() and packages_from_entry(). See Conversion Between Package Objects and XML Entries for more details.
| Parameters: |
|
|---|---|
| Returns: | set of strings, but see Conversion Between Package Objects and XML Entries - the set of packages that are in complete but not in initial |
Get a list of all symbols provided by the given package.
The base implementation simply aggregates the results of Bcfg2.Server.Plugins.Packages.Source.Source.get_provides().
| Parameters: | package (string) – The name of the package, but see Conversion Between Package Objects and XML Entries |
|---|---|
| Returns: | list of strings, but see Conversion Between Package Objects and XML Entries |
Get all groups that might be relevant to determining which sources apply to this collection’s client.
The base implementation simply aggregates the results of Bcfg2.Server.Plugins.Packages.Source.Source.get_relevant_groups()
| Returns: | list of strings - group names |
|---|
Get a list of all virtual packages provided by all sources.
The base implementation simply aggregates the results of Bcfg2.Server.Plugins.Packages.Source.Source.get_vpkgs().
| Returns: | list of strings, but see Conversion Between Package Objects and XML Entries |
|---|
Return True if a package is a package, False otherwise.
The base implementation returns True if any Source object’s Bcfg2.Server.Plugins.Packages.Source.Source.is_package() returns True.
| Parameters: | package (string) – The name of the package, but see Conversion Between Package Objects and XML Entries |
|---|---|
| Returns: | bool |
Return True if a name is a virtual package (i.e., is a symbol provided by a real package), False otherwise.
The base implementation returns True if any Source object’s Bcfg2.Server.Plugins.Packages.Source.Source.is_virtual_package() returns True.
| Parameters: | package (string) – The name of the symbol, but see Conversion Between Package Objects and XML Entries |
|---|---|
| Returns: | bool |
Returns True if the client’s “Magic Groups” match the magic groups for any of the sources contained in this Collection.
The base implementation returns True if any source Bcfg2.Server.Plugins.Packages.Source.Source.magic_groups_match() returns True.
| Returns: | bool |
|---|
Given a Package or BoundPackage entry, get a list of the package(s) described by it in a format appropriate for passing to complete(). By default, that’s just the name; only the Bcfg2.Server.Plugins.Packages.Yum backend supports versions or other extended data. See Conversion Between Package Objects and XML Entries for more details.
| Parameters: | entry (lxml.etree._Element) – The XML entry describing the package or packages. |
|---|---|
| Returns: | list of strings, but see Conversion Between Package Objects and XML Entries |
Given a list of package objects as returned by packages_from_entry() or complete(), return an XML tree describing the BoundPackage entries that should be included in the client configuration. See Conversion Between Package Objects and XML Entries for more details.
| Parameters: |
|
|---|
Do any collection-level data setup tasks. This is called when sources are loaded or reloaded by Bcfg2.Server.Plugins.Packages.Packages.
The base implementation is a no-op; the child Bcfg2.Server.Plugins.Packages.Source.Source objects will handle all data setup.
| Parameters: | force_update (bool) – Ignore all local cache and setup data from its original upstream sources (i.e., the package repositories) |
|---|
Given a source type, determine the class of Collection object that should be used to contain these sources. Note that source_type is not a Bcfg2.Server.Plugins.Packages.Source.Source subclass; it’s the name of a source type as given in sources.xml.
| Parameters: | source_type (string) – The type of source, e.g., “yum” or “apt” |
|---|---|
| Returns: | type - the Collection subclass that should be used to instantiate an object to contain sources of the given type. |
Source objects represent a single <Source> tag in sources.xml. Note that a single Source tag can itself describe multiple repositories (if it uses the “url” attribute instead of “rawurl”), and so can the Source object. This can be the source (har har) of some confusion. See Bcfg2.Server.Plugins.Packages.Collection.Collection.sourcelist() for the proper way to get all repos from a Source object.
Source objects are aggregated into Bcfg2.Server.Plugins.Packages.Collection.Collection objects, which are actually called by Bcfg2.Server.Plugins.Packages.Packages. This way a more advanced subclass can query repositories in aggregate rather than individually, which may give faster or more accurate results.
The base Source object must be subclassed to handle each repository type. How you subclass Source will depend on how you subclassed Bcfg2.Server.Plugins.Packages.Collection.Collection; see Bcfg2.Server.Plugins.Packages.Collection for more details on different methods for doing that.
If you are using the stock (or a near-stock) Bcfg2.Server.Plugins.Packages.Collection.Collection object, then you will need to implement the following methods and attributes in your Source subclass:
Additionally, you may want to consider overriding the following methods and attributes:
For an example of this kind of Source object, see Bcfg2.Server.Plugins.Packages.Apt.
If you are overriding the Collection object in more depth, then you have more leeway in what you might want to override or implement in your Source subclass. For an example of this kind of Source object, see Bcfg2.Server.Plugins.Packages.Yum.
A regular expression used to determine the base name of a repo from its URL. This is used when generating repo configs and by Source.get_repo_name(). It handles Pulp and mrepo repositories specially, and otherwise grabs the last component of the URL (as delimited by slashes).
Bases: Bcfg2.Server.Plugin.base.Debuggable
Source objects represent a single <Source> tag in sources.xml. Note that a single Source tag can itself describe multiple repositories (if it uses the “url” attribute instead of “rawurl”), and so can the Source object.
Note that a number of the attributes of this object may be more or less specific to one backend (e.g., essentialpkgs, recommended, gpgkeys, but they are included in the superclass to make the parsing of sources from XML more consistent, and to make it trivial for other backends to support those features.
| Parameters: | |
|---|---|
| Raises : |
Return true if this source applies to the given client, i.e., the client is in all necessary groups and “Magic Groups”.
| Parameters: | metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – The client metadata to check to see if this source applies |
|---|---|
| Returns: | bool |
A list of the arches supported by this source
The list of “Magic Groups” that make sources of this type available to clients.
The base filesystem path under which cache data for this source should be stored
A list of the the names of packages that are blacklisted from this source
The file (or directory) used for this source’s cache data
A unique key for this source that will be used to generate cachefile and other cache paths
A dict of repository options that will be included in the configuration generated for the client (if that is supported by the backend)
A list of the text of all ‘Component’ attributes of this source from XML
A list of predicates that are used to determine if this source applies to a given Bcfg2.Server.Plugins.Metadata.ClientMetadata object.
Whether or not to include deb-src lines in the generated APT configuration
A dict of <package name> -> <list of dependencies>. This will not necessarily be populated, particularly by backends that reimplement large portions of Bcfg2.Server.Plugins.Packages.Collection.Collection
Given a URL to a repository metadata file, return the full path to a file suitable for storing that file locally. This is acheived by replacing all forward slashes in the URL with @.
| Parameters: | url (string) – The URL to escape |
|---|---|
| Returns: | string |
Whether or not to include essential packages from this source
A set of package names that are deemed “essential” by this source
After Bcfg2.Server.Plugins.Packages.Collection.Collection.complete(), filter out packages that appear in the list of unknown packages but should not be presented to the user. unknown_filter is called to assess whether or not a package is expected to be unknown.
| Parameters: | unknown (set of strings) – A set of unknown packages. The set should be modified in place. |
|---|
Get a list of architectures that the given client has and for which this source provides packages for. The return value will always include global.
| Parameters: | metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – The client metadata to get matching architectures for |
|---|---|
| Returns: | list of strings |
Get a list of the dependencies of the given package.
| Parameters: | package (string) – The name of the symbol |
|---|---|
| Returns: | list of strings |
Get the list of packages of the given type in a package group.
| Parameters: | |
|---|---|
| Returns: | list of strings - package names |
Get a list of all symbols provided by the given package.
| Parameters: | package (string) – The name of the package |
|---|---|
| Returns: | list of strings |
Get all groups that might be relevant to determining which sources apply to this collection’s client.
| Returns: | list of strings - group names |
|---|
Try to find a sensible name for a repository. Since sources.xml doesn’t provide for repository names, we have to try to guess at the names when generating config files or doing other operations that require repository names. This function tries several approaches:
Once that is done, all characters disallowed in yum source names are replaced by dashes. See below for the exact regex. The yum rules are used here because they are so restrictive.
get_repo_name is not guaranteed to return a unique name. If you require a unique name, then you will need to generate all repo names and make them unique through the approach of your choice, e.g., appending numbers to non-unique repository names. See Bcfg2.Server.Plugins.Packages.Yum.Source.get_repo_name() for an example.
| Parameters: | url_map (dict) – A single url_map dict, i.e., any single element of url_map. |
|---|---|
| Returns: | string - the name of the repository. |
Get a list of all virtual packages provided by all sources.
| Returns: | list of strings |
|---|
A list of URLs to GPG keys that apply to this source
Formerly, Packages only supported applying package sources to groups; that is, they could not be assigned by more complicated logic like per-client repositories and group or client negation. This attribute attempts to provide for some limited backwards compat with older code that relies on this.
Return True if a package is a package, False otherwise.
| Parameters: | package (string) – The name of the package |
|---|---|
| Returns: | bool |
Return True if a name is a virtual package (i.e., is a symbol provided by a real package), False otherwise.
| Parameters: | package (string) – The name of the symbol, but see Conversion Between Package Objects and XML Entries |
|---|---|
| Returns: | bool |
Load saved state from cachefile. If caching and state is handled by the package library, then this function does not need to be implemented.
| Raises : | OSError - If the saved data cannot be read |
|---|---|
| Raises : | cPickle.UnpicklingError - If the saved data is corrupt |
Returns True if the client’s “Magic Groups” match the magic groups this source. Also returns True if magic groups are off in the configuration and the client’s architecture matches (i.e., architecture groups are always checked).
| Returns: | bool |
|---|
A set of all package names in this source. This will not necessarily be populated, particularly by backends that reimplement large portions of Bcfg2.Server.Plugins.Packages.Collection.Collection
Given dicts of depends and provides generated by read_files(), this generates deps and provides and calls save_state() to save the cached data to disk.
Both arguments are dicts of dicts of lists. Keys are the arches of packages contained in this source; values are dicts whose keys are package names and values are lists of either dependencies for each package the symbols provided by each package.
| Parameters: |
|
|---|
A dict of <package name> -> <list of provided symbols>. This will not necessarily be populated, particularly by backends that reimplement large portions of Bcfg2.Server.Plugins.Packages.Collection.Collection
The Package type handled by this Source class. The type attribute of Package entries will be set to the value ptype when they are handled by Bcfg2.Server.Plugins.Packages.
The “rawurl” attribute from xsource, if applicable. A trailing slash is automatically appended to this if there wasn’t one already present.
Read and parse locally downloaded metadata files and populates Bcfg2.Server.Plugins.Packages.Source.Source.pkgnames. Should call Bcfg2.Server.Plugins.Packages.Source.Source.process_files() as its final step.
Whether or not to include recommended packages from this source
Save state to cachefile. If caching and state is handled by the package library, then this function does not need to be implemented.
A dict of repository options that will be included in the configuration generated on the server side (if such is applicable; most backends do not generate any sort of repository configuration on the Bcfg2 server)
A Bcfg2 options dict
Perform all data fetching and setup tasks. For most backends, this involves downloading all metadata from the repository, parsing it, and caching the parsed data locally. The order of operations is:
Obviously with a backend that leverages repo access libraries to avoid downloading all metadata, many of the functions called by setup_data can be no-ops (or nearly so).
| Parameters: | force_update (bool) – Ignore all locally cached and downloaded data and fetch the metadata anew from the upstream repository. |
|---|
A predicate that is used by filter_unknown() to filter packages from the results of Bcfg2.Server.Plugins.Packages.Collection.Collection.complete() that should not be shown to the end user (i.e., that are not truly unknown, but are rather packaging system artifacts). By default, excludes any package whose name starts with “choice”
| Parameters: | package (string) – The name of a package that was unknown to the backend |
|---|---|
| Returns: | bool |
Download metadata from the upstream repository and cache it locally.
| Raises : | ValueError - If any URL in urls is malformed |
|---|---|
| Raises : | OSError - If there is an error writing the local cache |
| Raises : | HTTPError - If there is an error fetching the remote data |
The “url” attribute from xsource, if applicable. A trailing slash is automatically appended to this if there wasn’t one already present.
A list of dicts, each of which describes the URL to one repository contained in this source. Each dict contains the following keys:
A list of URLs to the base metadata file for each repository described by this source.
A list of the the names of packages that are whitelisted in this source
The XML tag that describes this source
Packages resolves Package entries on the Bcfg2 server in order to present a complete list of Package entries to the client in order to determine the completeness of the client configuration.
The default path for generated apt configs
Bases: _abcoll.MutableMapping
This maps a set of keys to a set of value-getting functions; the values are populated on-the-fly by the functions as the values are needed (and not before). This is used by Bcfg2.Server.Plugins.Packages.Packages.get_additional_data(); see the docstring for that function for details on why.
Unlike a dict, you should not specify values for for the righthand side of this mapping, but functions that get values. E.g.:
d = OnDemandDict(foo=load_foo,
bar=lambda: "bar");
Bases: Bcfg2.Server.Plugin.base.Plugin, Bcfg2.Server.Plugin.interfaces.Caching, Bcfg2.Server.Plugin.interfaces.StructureValidator, Bcfg2.Server.Plugin.interfaces.Generator, Bcfg2.Server.Plugin.interfaces.Connector, Bcfg2.Server.Plugin.interfaces.ClientRunHooks
Packages resolves Package entries on the Bcfg2 server in order to present a complete list of Package entries to the client in order to determine the completeness of the client configuration. It does so by delegating control of package version information to a number of backends, which may parse repository metadata directly or defer to package manager libraries for truly dynamic resolution.
| Parameters: |
|
|---|---|
| Raises : | OSError if adding a file monitor failed; Bcfg2.Server.Plugin.exceptions.PluginInitError on other errors |
List of names of methods to be exposed as XML-RPC functions
Bind configuration entries. HandleEntry handles entries two different ways:
| Parameters: |
|
|---|---|
| Returns: | lxml.etree._Element - The fully bound entry |
Determine if the given entry can be handled. Packages handles two kinds of entries:
| Parameters: |
|
|---|---|
| Returns: | bool - Whether or not this plugin can handle the entry |
| Raises : |
Packages.Refresh() => True|False
Reload configuration specification and download sources
Packages.Refresh() => True|False
Reload configuration specification and sources
Perform dependency resolution and build the complete list of packages that need to be included in the specification by validate_structures(), based on the initial list of packages.
| Parameters: |
|
|---|
Packages does a potentially tremendous amount of on-disk caching. cachepath holds the base directory to where data should be cached.
clients is a cache mapping of hostname -> Bcfg2.Server.Plugins.Packages.Collection.Collection.cachekey Unlike collections, this _is_ used to return a Bcfg2.Server.Plugins.Packages.Collection.Collection object when one is requested, so each entry is very short-lived – it’s purged at the end of each client run.
We cache Bcfg2.Server.Plugins.Packages.Collection.Collection objects in collections so that calling Refresh() or Reload() can tell the collection objects to clean up their cache, but we don’t actually use the cache to return a Collection object when one is requested, because that prevents new machines from working, since a Collection object gets created by get_additional_data(), which is called for all clients at server startup and various other times. (It would also prevent machines that change groups from working properly; e.g., if you reinstall a machine with a new OS, then returning a cached Collection object would give the wrong sources to that client.) These are keyed by the collection Bcfg2.Server.Plugins.Packages.Collection.Collection.cachekey, a unique key identifying the collection by its config, which could be shared among multiple clients.
Packages is an alternative to Bcfg2.Server.Plugins.Pkgmgr and conflicts with it.
Create yum/apt config for the specified client.
| Parameters: |
|
|---|
Report whether or not metadata processing is enabled.
This property cannot be set.
Report the state of the resolver. This can be disabled in the configuration. Note that disabling metadata (see disableMetaData) implies disabling the resolver.
This property cannot be set.
Hook to clear the cache for this client in clients, which must persist only the duration of a client run.
| Parameters: | metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – The client metadata |
|---|
Hook to clear the cache for this client in clients once statistics are processed to ensure that a stray cached Bcfg2.Server.Plugins.Packages.Collection.Collection object is not built during statistics and preserved until a subsequent client run.
| Parameters: | metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – The client metadata |
|---|
Return additional data for the given client. This will be an Bcfg2.Server.Plugins.Packages.OnDemandDict containing two keys:
This uses an OnDemandDict instead of just a normal dict because loading a source collection can be a fairly time-consuming process, particularly for the first time. As a result, when all metadata objects are built at once (such as after the server is restarted, or far more frequently if Metadata caching is disabled), this function would be a major bottleneck if we tried to build all collections at the same time. Instead, they’re merely built on-demand.
| Parameters: | metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – The client metadata |
|---|---|
| Returns: | dict of lists of url_map data |
Get a Bcfg2.Server.Plugins.Packages.Collection.Collection object for this client.
| Parameters: | metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – The client metadata to get a Collection for |
|---|---|
| Returns: | An instance of the appropriate subclass of Bcfg2.Server.Plugins.Packages.Collection.Collection that contains all relevant sources that apply to the given client |
Get yum/apt config, as a string, for the specified client.
| Parameters: | metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – The client to create the config for. |
|---|
groupcache caches group lookups. It maps Collections (via Bcfg2.Server.Plugins.Packages.Collection.Collection.cachekey) to sets of package groups, and thence to the packages indicated by those groups.
Where Packages should store downloaded GPG key files
pkgcache caches complete package sets. It maps Collections (via Bcfg2.Server.Plugins.Packages.Collection.Collection.cachekey) to sets of initial packages, and thence to the final (complete) package selections resolved from the initial packages
The Bcfg2.Server.Plugins.Packages.PackagesSources.PackagesSources object used to generate Bcfg2.Server.Plugins.Packages.Source.Source objects for this plugin.
Do the real work of Packages. This does two things:
| Parameters: |
|
|---|---|
| Returns: | None |
The default path for generated yum configs
PackagesSources handles the Packages sources.xml file
Bases: Bcfg2.Server.Plugin.helpers.StructFile, Bcfg2.Server.Plugin.base.Debuggable
PackagesSources handles parsing of the Bcfg2.Server.Plugins.Packages sources.xml file, and the creation of the appropriate Bcfg2.Server.Plugins.Packages.Source.Source object for each Source tag.
| Parameters: |
|
|---|---|
| Raises : | Bcfg2.Server.Plugin.exceptions.PluginInitError - If sources.xml cannot be read |
HandleEvent is called whenever the FAM registers an event.
When loaded becomes True, Bcfg2.Server.Plugins.Packages.Packages.Reload() is called to reload all plugin data from the configured sources.
| Parameters: | event (Bcfg2.Server.FileMonitor.Event) – The event object |
|---|---|
| Returns: | None |
Index() is called by HandleEvent() every time the data changes, and parses the data into usable data as required.
Index is responsible for calling source_from_xml() for each Source tag in each file.
The full path to the directory where Bcfg2.Server.Plugins.Packages.Source.Source data will be cached
Whether or not all XML files (sources.xml and everything XIncluded in it) have been parsed. This flag is used to determine if the Packages plugin should be told to load its data.
The set of all XML files that have been successfully parsed. This is used by loaded to determine if the sources have been fully parsed and the Bcfg2.Server.Plugins.Packages.Packages plugin should be told to reload its data.
The Bcfg2.Server.Plugins.Packages.Packages that instantiated this PackagesSources object
The Bcfg2 options dict
Create a Bcfg2.Server.Plugins.Packages.Source.Source subclass object from XML representation of a source in sources.xml. source_from_xml determines the appropriate subclass of Source to instantiate according to the type attribute of the Source tag.
| Parameters: | xsource (lxml.etree._Element) – The XML tag representing the source |
|---|---|
| Returns: | Bcfg2.Server.Plugins.Packages.Source.Source subclass, or None on error |
Yum backend for Bcfg2.Server.Plugins.Packages. This module is the most complex backend because it has to handle Yum sources without yum Python libraries, with yum Python libraries, and Pulp sources. (See Using Native Yum Libraries for details on using the yum Python libraries and Pulp Support for details on Pulp sources.)
If using the yum Python libraries, YumCollection makes shell calls to an external command, bcfg2-yum-helper, which performs the actual yum API calls. This is done because the yum libs have horrific memory leaks, and apparently the right way to get around that in long-running processes it to have a short-lived helper. This is how it’s done by yum itself in yum-updatesd, which is a long-running daemon that checks for and applies updates.
Bcfg2.Server.Plugins.Packages.Collection.Collection objects have the option to translate from some backend-specific representation of packages to XML entries; see Conversion Between Package Objects and XML Entries for more information on this. If you are using the Python yum libraries, Bcfg2.Server.Plugins.Packages.Yum.YumCollection opts to do this, using the yum tuple representation of packages, which is:
(<name>, <arch>, <epoch>, <version>, <release>)
For shorthand this is occasionally abbrevated “naevr”. Any datum that is not defined is None. So a normal package entry that can be any version would be passed to bcfg2-yum-helper as:
("somepackage", None, None, None, None)
A package returned from the helper might look more like this:
("somepackage", "x86_64", None, "1.2.3", "1.el6")
We translate between this representation and the XML representation of packages with YumCollection.packages_from_entry() and YumCollection.packages_to_entry().
Bases: Bcfg2.Server.Plugin.helpers.SpecificData
Handle pulp consumer certificate data for PulpCertificateSet
| Parameters: |
|
|---|
Given an abstract entry, add data to it and return it. PulpCertificateSet handles binding entry metadata.
| Parameters: | entry (lxml.etree._Element) – The abstract entry to bind data to |
|---|---|
| Returns: | lxml.etree._Element - the bound entry |
Bases: Bcfg2.Server.Plugin.helpers.EntrySet
Handle Pulp consumer certificates.
| Parameters: | path (string) – The path to the directory where Pulp consumer certificates will be stored |
|---|
Handle FAM events on certificate files.
| Parameters: | event (Bcfg2.Server.FileMonitor.Event) – The event to handle |
|---|
The path to certificates on consumer machines
Service the FAM events queued up by the key generation so the data structure entries will be available for binding.
NOTE: We wait for up to ten seconds. There is some potential for race condition, because if the file monitor doesn’t get notified about the new key files in time, those entries won’t be available for binding. In practice, this seems “good enough.”
| Parameters: | filename (string) – The filename to check for events on |
|---|
Write a new certificate to the filesystem.
| Parameters: |
|
|---|
Bases: Bcfg2.Server.Plugins.Packages.Collection.Collection
Handle collections of Yum sources. If we’re using the yum Python libraries, then this becomes a very full-featured Bcfg2.Server.Plugins.Packages.Collection.Collection object; if not, then it defers to the YumSource object.
Add GPG keys instances to a Package entry. This is called from build_extra_structures() to add GPG keys to the specification.
| Parameters: |
|
|---|
Get a Pulp consumer object for the client.
| Parameters: | consumerapi (pulp.client.api.consumer.ConsumerAPI) – A Pulp ConsumerAPI object. If none is passed, one will be instantiated. |
|---|---|
| Returns: | dict - the consumer. Returns None on failure (including if there is no existing Pulp consumer for this client. |
Add additional entries to the <Independent/> section of the final configuration. This adds several kinds of entries:
| Parameters: | independent (lxml.etree._Element) – The XML tag to add extra entries to. This is modified in place. |
|---|
Define a unique cache file for this collection to use for cached yum metadata
Make a call to bcfg2-yum-helper. The yum libs have horrific memory leaks, so apparently the right way to get around that in long-running processes it to have a short-lived helper. No, seriously – check out the yum-updatesd code. It’s pure madness.
| Parameters: |
|
|---|---|
| Returns: | Varies depending on the return value of the bcfg2-yum-helper command. |
The path to the server-side config file used when resolving packages with the Python yum libraries
A Bcfg2.Utils.Executor object to use to run external commands
Build a complete list of all packages and their dependencies.
When using the Python yum libraries, this defers to the bcfg2-yum-helper; when using the builtin yum parser, this defers to Bcfg2.Server.Plugins.Packages.Collection.Collection.complete().
| Parameters: | packagelist (set of strings, but see Conversion Between Package Objects and XML Entries) – Set of initial packages computed from the specification. |
|---|---|
| Returns: | tuple of sets - The first element contains a set of strings (but see Conversion Between Package Objects and XML Entries) describing the complete package list, and the second element is a set of symbols whose dependencies could not be resolved. |
Report whether or not metadata processing is enabled. This duplicates code in Packages/__init__.py, and can probably be removed in Bcfg2 1.4 when we have a module-level setup object.
If ‘arch’ for each source is the same, return that arch, otherwise None.
This helps bcfg2-yum-helper when the client arch is incompatible with the bcfg2 server’s arch.
In case multiple arches are found, punt back to the default behavior.
Get the yum configuration for this collection.
| Parameters: | raw (bool) – Return a ConfigParser.SafeConfigParser object representing the configuration instead of a string. This is useful if you need to modify the config before writing it (as write_config() does in order to produce a server-specific configuration). |
|---|---|
| Returns: | string or ConfigParser.SafeConfigParser |
If using the yum libraries, given a list of package group names, return a dict of <group name>: <list of packages>. This is much faster than implementing Bcfg2.Server.Plugins.Packages.Collection.Collection.get_group(), since we have to make a call to the bcfg2 Yum helper, and each time we do that we make another call to yum, which means we set up yum metadata from the cache (hopefully) each time. So resolving ten groups once is much faster than resolving one group ten times.
If you are using the builtin yum parser, this raises a warning and returns an empty dict.
| Parameters: | grouplist (list of strings - group names) – The list of groups to query |
|---|---|
| Returns: | dict of <group name>: <list of packages> |
In this implementation the packages may be strings or tuples. See Package Objects for more information.
Compute the difference between the complete package list (as returned by complete()) and the initial package list computed from the specification, allowing for package tuples. See Package Objects and Conversion Between Package Objects and XML Entries for more information on this process.
| Parameters: |
|
|---|---|
| Returns: | set of tuples |
The full path to bcfg2-yum-helper. First, we check in the config file to see if it has been explicitly specified; next we see if it’s in $PATH (which we do by making a call to it; I wish there was a way to do this without forking, but apparently not); finally we check in /usr/sbin, the default location.
Options that are included in the [packages:yum] section of the config but that should not be included in the temporary yum.conf we write out
When using the Python yum libraries, convert a Package entry to a list of package tuples. See Package Objects and Conversion Between Package Objects and XML Entries for more information on this process.
| Parameters: | entry (lxml.etree._Element) – The Package entry to convert |
|---|---|
| Returns: | list of tuples |
When using the Python yum libraries, convert a list of package tuples to a Package entry. See Package Objects and Conversion Between Package Objects and XML Entries for more information on this process.
If pkglist contains only one package, then its data is converted to a single BoundPackage entry that is added as a subelement of entry. If pkglist contains more than one package, then a parent BoundPackage entry is created and child Instance entries are added to it.
| Parameters: |
|
|---|---|
| Returns: | None |
PulpCertificateSet object used to handle Pulp certs
Do any collection-level data setup tasks. This is called when sources are loaded or reloaded by Bcfg2.Server.Plugins.Packages.Packages.
If the builtin yum parsers are in use, this defers to Bcfg2.Server.Plugins.Packages.Collection.Collection.setup_data(). If using the yum Python libraries, this cleans up cached yum metadata, regenerates the server-side yum config (in order to catch any new sources that have been added to this server), then regenerates the yum cache.
| Parameters: | force_update (bool) – Ignore all local cache and setup data from its original upstream sources (i.e., the package repositories) |
|---|
Write the server-side config file to cfgfile based on the data from get_config()
Bases: Bcfg2.Server.Plugins.Packages.Source.Source
Handle yum sources
| Parameters: | |
|---|---|
| Raises : |
“Magic Groups” for YumSource are “yum”, “redhat”, “centos”, and “fedora”
After Bcfg2.Server.Plugins.Packages.Collection.Collection.complete(), filter out packages that appear in the list of unknown packages but should not be presented to the user. unknown_filter is called to assess whether or not a package is expected to be unknown.
| Parameters: | unknown (set of strings) – A set of unknown packages. The set should be modified in place. |
|---|
Get the list of packages of the given type in a package group.
| Parameters: | |
|---|---|
| Returns: | list of strings - package names |
Try to find a sensible name for a repository. First use a repository’s Pulp ID, if it has one; if not, then defer to Bcfg2.Server.Plugins.Packages.Source.Source.get_repo_name
| Parameters: | url_map (dict) – A single url_map dict, i.e., any single element of url_map. |
|---|---|
| Returns: | string - the name of the repository. |
Get a list of all virtual packages provided by all sources.
| Returns: | list of strings |
|---|
Return True if a package is a package, False otherwise.
| Parameters: | package (string) – The name of the package |
|---|---|
| Returns: | bool |
If using the builtin yum parser, load saved state from cachefile. If using the Python yum libraries, yum handles caching and state and this method is a no-op.
YumSource sets the type on Package entries to “yum”
When using the builtin yum parser, read and parse locally downloaded metadata files. This diverges from the stock Bcfg2.Server.Plugins.Packages.Source.Source.read_files() quite a bit.
If using the builtin yum parser, save state to cachefile. If using the Python yum libraries, yum handles caching and state and this method is a no-op.
setup_data is only used by the builtin yum parser. Perform all data fetching and setup tasks. For most backends, this involves downloading all metadata from the repository, parsing it, and caching the parsed data locally. The order of operations is:
Obviously with a backend that leverages repo access libraries to avoid downloading all metadata, many of the functions called by setup_data can be no-ops (or nearly so).
| Parameters: | force_update (bool) – Ignore all locally cached and downloaded data and fetch the metadata anew from the upstream repository. |
|---|
By default, Bcfg2.Server.Plugins.Packages.Source.Source filters out unknown packages that start with “choice”, but that doesn’t mean anything to Yum or RPM. Instead, we filter out unknown packages that start with “rpmlib”, although this is likely legacy behavior; that would seem to indicate that a package required some RPM feature that isn’t provided, which is a bad thing. This should probably go away at some point.
| Parameters: | package (string) – The name of a package that was unknown to the backend |
|---|---|
| Returns: | bool |
APT backend for Bcfg2.Server.Plugins.Packages
Bases: Bcfg2.Server.Plugins.Packages.Collection.Collection
Handle collections of APT sources. This is a no-op object that simply inherits from Bcfg2.Server.Plugins.Packages.Collection.Collection, overrides nothing, and defers all operations to PacSource
| Parameters: |
|
|---|
Bases: Bcfg2.Server.Plugins.Packages.Source.Source
Handle APT sources
| Parameters: | |
|---|---|
| Raises : |
“Magic Groups” for AptSource are “apt”, “debian”, “ubuntu”, and “nexenta”
AptSource sets the type on Package entries to “deb”
Read and parse locally downloaded metadata files and populates Bcfg2.Server.Plugins.Packages.Source.Source.pkgnames. Should call Bcfg2.Server.Plugins.Packages.Source.Source.process_files() as its final step.
Pacman backend for Bcfg2.Server.Plugins.Packages
Bases: Bcfg2.Server.Plugins.Packages.Collection.Collection
Handle collections of Pacman sources. This is a no-op object that simply inherits from Bcfg2.Server.Plugins.Packages.Collection.Collection, overrides nothing, and defers all operations to PacSource
| Parameters: |
|
|---|
Bases: Bcfg2.Server.Plugins.Packages.Source.Source
Handle Pacman sources
| Parameters: | |
|---|---|
| Raises : |
“Magic Groups” for PacSource are “arch” and “parabola”
PacSource sets the type on Package entries to “pacman”
Read and parse locally downloaded metadata files and populates Bcfg2.Server.Plugins.Packages.Source.Source.pkgnames. Should call Bcfg2.Server.Plugins.Packages.Source.Source.process_files() as its final step.