sampledoc

bcfg2-lint Plugin Development

bcfg2-lint, like most parts of Bcfg2, has a pluggable backend that lets you easily write your own plugins to verify various parts of your Bcfg2 specification.

Plugins are included in a module of the same name as the plugin class in Bcfg2.Server.Lint, e.g., Bcfg2.Server.Lint.Validate.

Note

It is no longer possible to include lint plugins directly in a Bcfg2 server plugin, e.g., Bcfg2.Server.Plugins.Metadata.MetadataLint.

Plugin Types

There are two types of bcfg2-lint plugins:

Serverless plugins

Serverless plugins are run before bcfg2-lint starts up a local Bcfg2 server, so the amount of introspection they can do is fairly limited. They can directly examine the Bcfg2 specification, of course, but they can’t examine the entries handled by a given plugin or anything that requires a running server.

If a serverless plugin raises a lint error, however, the server will not be started and no Server plugins will be run. This makes them useful to check for the sorts of errors that might prevent the Bcfg2 server from starting properly.

Serverless plugins must subclass Bcfg2.Server.Lint.ServerlessPlugin.

Bcfg2.Server.Lint.Validate is an example of a serverless plugin.

Server plugins

Server plugins are run after a local Bcfg2 server has been started, and have full access to all of the parsed data and so on. Because of this, they tend to be easier to use than Serverless plugins, and thus are more common.

Server plugins are only run if all Serverless plugins run successfully (i.e., raise no errors).

Server plugins must subclass Bcfg2.Server.Lint.ServerPlugin.

Bcfg2.Server.Lint.Genshi is an example of a server plugin.

Error Handling

The job of a bcfg2-lint plugin is to find errors. Each error that a plugin may produce must have a name, a short string that briefly describes the error and will be used to configure error levels in bcfg2.conf. It must also have a default reporting level. Possible reporting levels are “error”, “warning”, or “silent”. All of the errors that may be produced by a plugin must be returned as a dict by Bcfg2.Server.Lint.Plugin.Errors(). For instance, consider Bcfg2.Server.Lint.InfoXML.InfoXML.Errors():

@classmethod
def Errors(cls):
    return {"no-infoxml": "warning",
            "deprecated-info-file": "warning",
            "paranoid-false": "warning",
            "required-infoxml-attrs-missing": "error"}

This means that the Bcfg2.Server.Lint.InfoXML.InfoXML lint plugin can produce five lint errors, although four of them are just warnings by default.

The errors returned by each plugin’s Errors() method will be passed to Bcfg2.Server.Lint.ErrorHandler.RegisterErrors(), which will use that information and the information in the config file to determine how to display (or not display) each error to the end user.

Errors are produced in a plugin with Bcfg2.Server.Lint.Plugin.LintError(), which takes two arguments: the name of the error, which must correspond to a key in the dict returned by Bcfg2.Server.Lint.Plugin.Errors(), and a freeform string that will be displayed to the end user. Note that the error name and its display are thus only tied together when the error is produced; that is, a single error (by name) can have two completely different outputs.

Basics

Base classes for Lint plugins and error handling

class Bcfg2.Server.Lint.CLI[source]

Bases: object

The bcfg2-lint CLI

get_errorhandler()[source]

get a Bcfg2.Server.Lint.ErrorHandler object

run()[source]

Run bcfg2-lint

run_server_plugins()[source]

run plugins that require a running server to run

run_serverless_plugins()[source]

Run serverless plugins

class Bcfg2.Server.Lint.ErrorHandler(errors=None)[source]

Bases: object

A class to handle errors for bcfg2-lint plugins

Parameters:errors (dict) – An initial dict of errors to register
RegisterErrors(errors)[source]

Register a dict of errors that a plugin may raise. The keys of the dict are short strings that describe each error; the values are the default error handling for that error (“error”, “warning”, or “silent”).

Parameters:errors (dict) – The error dict
debug(msg)[source]

Log a silent/debug condition.

Parameters:msg (string) – The freeform message to display to the end user.
dispatch(err, msg)[source]

Dispatch an error to the correct handler.

Parameters:
error(msg)[source]

Log an error condition.

Parameters:msg (string) – The freeform message to display to the end user.
errors = None

The number of errors passed to this error handler

errortypes = None

A dict of registered errors

warn(msg)[source]

Log a warning condition.

Parameters:msg (string) – The freeform message to display to the end user.
warnings = None

The number of warnings passed to this error handler

class Bcfg2.Server.Lint.LintPluginAction(*args, **kwargs)[source]

Bases: Bcfg2.Options.Actions.ComponentAction

Option parser action to load lint plugins

class Bcfg2.Server.Lint.LintPluginOption(*args, **kwargs)

Bases: Bcfg2.Options.Options.Option

Option class for the lint_plugins

See argparse.ArgumentParser.add_argument() for a full list of accepted parameters.

In addition to supporting all arguments and keyword arguments from argparse.ArgumentParser.add_argument(), several additional keyword arguments are allowed.

Parameters:
  • cf (tuple) – A tuple giving the section and option name that this argument can be referenced as in the config file. The option name may contain the wildcard ‘*’, in which case the value will be a dict of all options matching the glob. (To use a wildcard in the section, use a Bcfg2.Options.WildcardSectionGroup.)
  • env (string) – An environment variable that the value of this option can be taken from.
  • man (string) – A detailed description of the option that will be used to populate automatically-generated manpages.
early_parsing_hook(namespace)

We want a usefull default for the enabled lint plugins. Therfore we use all importable plugins, that either pertain with enabled server plugins or that has no matching plugin.

class Bcfg2.Server.Lint.Plugin(errorhandler=None, files=None)[source]

Bases: object

Base class for all bcfg2-lint plugins

Parameters:
classmethod Errors()[source]

Returns a dict of errors the plugin supplies, in a format suitable for passing to Bcfg2.Server.Lint.ErrorHandler.RegisterErrors().

Must be overloaded by child classes.

Returns:dict
HandlesFile(fname)[source]

Returns True if the given file should be handled by the plugin according to Bcfg2.Server.Lint.Plugin.files, False otherwise.

LintError(err, msg)[source]

Raise an error from the lint process.

Parameters:
RenderXML(element, keep_text=False)[source]

Render an XML element for error output. This prefixes the line number and removes children for nicer display.

Parameters:
  • element (lxml.etree._Element) – The element to render
  • keep_text (boolean) – Do not discard text content from the element for display
Run()[source]

Run the plugin. Must be overloaded by child classes.

errorhandler = None

The error handler

files = None

The list of files that bcfg2-lint should be run against

list_matching_files(path)

list all files matching the path in self.files or the bcfg2 repo.

class Bcfg2.Server.Lint.ServerPlugin(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.Plugin

Base class for bcfg2-lint plugins that check things that require the running Bcfg2 server.

Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
core = None

The server core

metadata = None

The metadata plugin

class Bcfg2.Server.Lint.ServerlessPlugin(errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.Plugin

Base class for bcfg2-lint plugins that are run before the server starts up (i.e., plugins that check things that may prevent the server from starting up).

Parameters:
Bcfg2.Server.Lint.get_termsize()[source]

get a tuple of (width, height) giving the size of the terminal

Existing bcfg2-lint Plugins

AWSTags

bcfg2-lint plugin to check all given AWSTags patterns for validity.

class Bcfg2.Server.Lint.AWSTags.AWSTags(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

bcfg2-lint plugin to check all given AWSTags patterns for validity.

Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
check(entry, attr)[source]

Check a single attribute (name or value) of a single entry for validity.

Bundler

bcfg2-lint plugin for Bundler

class Bcfg2.Server.Lint.Bundler.Bundler(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

Perform various Bundler checks.

Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
bundle_names(bundle)[source]

Verify that deprecated bundle .genshi bundles and explicit bundle names aren’t used

missing_bundles()[source]

Find bundles listed in Metadata but not implemented in Bundler.

Comments

Check files for various required comments.

class Bcfg2.Server.Lint.Comments.Comments(*args, **kwargs)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

The Comments lint plugin checks files for header comments that give information about the files. For instance, you can require SVN keywords in a comment, or require the name of the maintainer of a Genshi template, and so on.

check_bundles()[source]

Check bundle files for required comments.

check_cfg()[source]

Check Cfg files and info.xml files for required comments.

check_lines(filename, lines, rtype)[source]

Generic header check for a set of lines.

Parameters:
  • filename (string) – The filename
  • lines (list of strings) – The data to check
  • rtype (string) – The type of file. Available types are documented in bcfg2-lint.conf(5).
check_metadata()[source]

Check Metadata files for required comments.

check_plaintext(filename, data, rtype)[source]

Generic check to check a plain text file for required comments.

Parameters:
  • filename (string) – The filename
  • data (string) – The file data
  • rtype (string) – The type of file. Available types are documented in bcfg2-lint.conf(5).
check_probes()[source]

Check Probes for required comments

check_properties()[source]

Check Properties files for required comments.

check_xml(filename, xdata, rtype)[source]

Generic check to check an XML file for required comments.

Parameters:
  • filename (string) – The filename
  • xdata (lxml.etree._Element) – The file data
  • rtype (string) – The type of file. Available types are documented in bcfg2-lint.conf(5).
has_all_xincludes(mfile)[source]

Return True if Bcfg2.Server.Lint.Plugin.files includes all XIncludes listed in the specified metadata type, false otherwise. In other words, this returns True if bcfg2-lint is dealing with complete metadata.

Parameters:mfile (string) – The metadata file (“clients.xml” or “groups.xml”) to check for XIncludes
Returns:bool
required_comments(rtype)[source]

Given a file type, fetch the list of required comments from the bcfg2-lint config. Valid file types are documented in bcfg2-lint.conf(5).

Parameters:rtype (string) – The file type
Returns:list - the required items
required_items(rtype, itype)[source]

Given a file type and item type (comment or keyword), fetch the list of required items from the bcfg2-lint config. Valid file types are documented in bcfg2-lint.conf(5).

Parameters:
  • rtype (string) – The file type
  • itype (string) – The item type (comment or keyword)
Returns:

list - the required items

required_keywords(rtype)[source]

Given a file type, fetch the list of required VCS keywords from the bcfg2-lint config. Valid file types are documented in bcfg2-lint.conf(5).

Parameters:rtype (string) – The file type
Returns:list - the required items

Genshi

Check Genshi templates for syntax errors.

class Bcfg2.Server.Lint.Genshi.Genshi(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

Check Genshi templates for syntax errors.

Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
check_bundler()[source]

Check templates in Bundler for syntax errors.

check_cfg()[source]

Check genshi templates in Cfg for syntax errors.

check_template(loader, fname, cls=None)[source]

Generic check for all genshi templates (XML and text)

GroupNames

Ensure that all named groups are valid group names.

class Bcfg2.Server.Lint.GroupNames.GroupNames(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

Ensure that all named groups are valid group names.

Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
check_bundles()[source]

Check groups used in the Bundler plugin for validity.

check_cfg()[source]

Check groups used in group-specific files in the Cfg plugin for validity.

check_entries(entries, fname)[source]

Check a generic list of XML entries for <Group> tags with invalid name attributes.

Parameters:
  • entries (list of lxml.etree._Element) – A list of XML <Group> tags whose name attributes will be validated.
  • fname (string) – The filename the entry list came from
check_grouppatterns()[source]

Check groups used in the GroupPatterns plugin for validity

check_metadata()[source]

Check groups used or declared in the Metadata plugin for validity.

check_rules()[source]

Check groups used in the Rules plugin for validity.

pattern = '\\S+$'

A string regex that matches only valid group names. Currently, a group name is considered valid if it contains only non-whitespace characters.

valid = <_sre.SRE_Pattern object at 0x80a4ea0>

A compiled regex for Bcfg2.Server.Lint.GroupNames.GroupNames.pattern

GroupPatterns

bcfg2-lint plugin for GroupPatterns

class Bcfg2.Server.Lint.GroupPatterns.GroupPatterns(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

bcfg2-lint plugin to check all given GroupPatterns patterns for validity. This is simply done by trying to create a Bcfg2.Server.Plugins.GroupPatterns.PatternMap object for each pattern, and catching exceptions and presenting them as bcfg2-lint errors.

Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
check(entry, groups, ptype='NamePattern')[source]

Check a single pattern for validity

InfoXML

Ensure that all config files have a valid info.xml file.

class Bcfg2.Server.Lint.InfoXML.InfoXML(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

Ensure that all config files have a valid info.xml file. This plugin can check for:

  • Missing info.xml files;
  • Use of deprecated info/:info files;
  • Paranoid mode disabled in an info.xml file;
  • Required attributes missing from info.xml
Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
check_infoxml(fname, xdata)[source]

Verify that info.xml contains everything it should.

MergeFiles

find Probes or Cfg files with multiple similar files that might be merged into one

class Bcfg2.Server.Lint.MergeFiles.MergeFiles(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

find Probes or Cfg files with multiple similar files that might be merged into one

Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
check_cfg()[source]

check Cfg for similar files

check_probes()[source]

check Probes for similar files

get_similar(entries)[source]

Get a list of similar files from the entry dict. Return value is a list of lists, each of which gives the filenames of similar files

Bcfg2.Server.Lint.MergeFiles.threshold(val)[source]

Option type processor to accept either a percentage (e.g., “threshold=75”) or a ratio (e.g., “threshold=.75”)

Metadata

bcfg2-lint plugin for Metadata

class Bcfg2.Server.Lint.Metadata.Metadata(core, errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

bcfg2-lint plugin for Metadata. This checks for several things:

  • <Client> tags nested inside other <Client> tags;
  • Deprecated options (like location="floating");
  • Profiles that don’t exist, or that aren’t profile groups;
  • Groups or clients that are defined multiple times;
  • Multiple default groups or a default group that isn’t a profile group.
Parameters:
  • core (Bcfg2.Server.Core.BaseCore) – The Bcfg2 server core
  • errorhandler (Bcfg2.Server.Lint.ErrorHandler) – A Bcfg2.Server.Lint.ErrorHandler that will be used to handle lint errors. If one is not provided, a new one will be instantiated.
  • files (list of strings) – A list of files to run bcfg2-lint against. (See the bcfg2-lint --stdin option.)
bogus_profiles()[source]

Check for clients that have profiles that are either not flagged as profile groups in groups.xml, or don’t exist.

default_is_profile()[source]

Ensure that the default group is a profile group.

deprecated_options()[source]

Check for the location='floating' option, which has been deprecated in favor of floating='true'.

duplicate_clients()[source]

Check for clients that are defined more than once.

duplicate_default_groups()[source]

Check for multiple default groups.

duplicate_entries(allentries, etype)[source]

Generic duplicate entry finder.

Parameters:
  • allentries (list of lxml.etree._Element) – A list of all entries to check for duplicates.
  • etype (string) – The entry type. This will be used to determine the error name (duplicate-<etype>) and for display to the end user.
duplicate_groups()[source]

Check for groups that are defined more than once. There are two ways this can happen:

  1. The group is listed twice with contradictory options.
  2. The group is listed with no options first, and then with options later.

In this context, ‘first’ refers to the order in which groups are parsed; see the loop condition below and _handle_groups_xml_event above for details.

nested_clients()[source]

Check for a <Client/> tag inside a <Client/> tag, which is either redundant or will never match.

Pkgmgr

bcfg2-lint plugin for Pkgmgr

class Bcfg2.Server.Lint.Pkgmgr.Pkgmgr(errorhandler=None, files=None)[source]

Bases: Bcfg2.Server.Lint.ServerlessPlugin

Find duplicate Pkgmgr entries with the same priority.

Parameters:

RequiredAttrs

Verify attributes for configuration entries that cannot be verified with an XML schema alone.

class Bcfg2.Server.Lint.RequiredAttrs.RequiredAttrs(*args, **kwargs)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

Verify attributes for configuration entries that cannot be verified with an XML schema alone.

check_bundles()[source]

Check bundles for BoundPath and BoundPackage entries with missing attrs.

check_default_acl(path)[source]

Check that a default ACL contains either no entries or minimum required entries

check_entry(entry, filename)[source]

Generic entry check.

Parameters:
  • entry (lxml.etree._Element) – The XML entry to check for missing attributes.
  • filename (string) – The filename the entry came from
check_packages()[source]

Check Packages sources for Source entries with missing attributes.

check_rules()[source]

check Rules for Path entries with missing attrs

Bcfg2.Server.Lint.RequiredAttrs.is_device_mode(val)[source]

Return True if val is a string describing a positive integer

Bcfg2.Server.Lint.RequiredAttrs.is_filename(val)[source]

Return True if val is a string describing a valid full path

Bcfg2.Server.Lint.RequiredAttrs.is_octal_mode(val)[source]

Return True if val is a string describing a valid octal permissions mode

Bcfg2.Server.Lint.RequiredAttrs.is_selinux_type(val)[source]

Return True if val is a string describing a valid (although not necessarily existent) SELinux type

Bcfg2.Server.Lint.RequiredAttrs.is_selinux_user(val)[source]

Return True if val is a string describing a valid (although not necessarily existent) SELinux user

Bcfg2.Server.Lint.RequiredAttrs.is_username(val)[source]

Return True if val is a string giving either a positive integer uid, or a valid Unix username

TemplateHelper

bcfg2-lint plugin for TemplateHelper

class Bcfg2.Server.Lint.TemplateHelper.TemplateHelper(*args, **kwargs)[source]

Bases: Bcfg2.Server.Lint.ServerPlugin

bcfg2-lint plugin to ensure that all TemplateHelper modules are valid. This can check for:

  • A TemplateHelper module that cannot be imported due to syntax or other compile-time errors;
  • A TemplateHelper module that does not have an __export__ attribute, or whose __export__ is not a list;
  • Bogus symbols listed in __export__, including symbols that don’t exist, that are reserved, or that start with underscores.
check_helper(helper)[source]

Check a single helper module.

Parameters:helper (string) – The filename of the helper module

Validate

Validate XML files.

Ensure that all XML files in the Bcfg2 repository validate according to their respective schemas.

class Bcfg2.Server.Lint.Validate.Validate(*args, **kwargs)[source]

Bases: Bcfg2.Server.Lint.ServerlessPlugin

Ensure that all XML files in the Bcfg2 repository validate according to their respective schemas.

check_properties()[source]

Check Properties files against their schemas.

filesets = None

A dict of <file glob>: <schema file> that maps files in the Bcfg2 specification to their schemas. The globs are extended fnmatch globs that also support **, which matches any number of any characters, including forward slashes. The schema files are relative to the schema directory, which can be controlled by the bcfg2-lint --schema option.

get_filelists()[source]

Get lists of different kinds of files to validate. This doesn’t return anything, but it sets Bcfg2.Server.Lint.Validate.Validate.filelists to a dict whose keys are path globs given in Bcfg2.Server.Lint.Validate.Validate.filesets and whose values are lists of the full paths to all files in the Bcfg2 repository (or given with bcfg2-lint --stdin) that match the glob.

parse(filename)[source]

Parse an XML file, raising the appropriate LintErrors if it can’t be parsed or read. Return the lxml.etree._ElementTree parsed from the file.

Parameters:filename (string) – The full path to the file to parse
Returns:lxml.etree._ElementTree - the parsed data
validate(filename, schemafile, schema=None)[source]

Validate a file against the given schema.

Parameters:
  • filename (string) – The full path to the file to validate
  • schemafile (string) – The full path to the schema file to validate against
  • schema (lxml.etree.Schema) – The loaded schema to validate against. This can be used to avoid parsing a single schema file for every file that needs to be validate against it.
Returns:

bool - True if the file validates, false otherwise