sampledoc

Server Core Development

New in version 1.3.0.

Bcfg2 1.3 added a pluggable server core system so that the server core itself can be easily swapped out to use different technologies. It currently ships with several backends: a builtin core written from scratch using the various server tools in the Python standard library; a variant on the builtin core that uses Python 2.6’s multiprocessing library to process requests in parallel; and an experimental CherryPy based core. This page documents the server core interface so that other cores can be written to take advantage of other technologies, e.g., Tornado or Twisted.

A core implementation needs to:

A core that wants to use the network (i.e., a core that isn’t used entirely for introspection, as in bcfg2-info, or other local tasks) should inherit from Bcfg2.Server.Core.NetworkCore, and must also override Bcfg2.Server.Core.NetworkCore._daemonize() to handle daemonization, writing the PID file, and dropping privileges.

Nearly all XML-RPC handling is delegated entirely to the core implementation. It needs to:

  • Call Bcfg2.Server.Core.NetworkCore.authenticate() to authenticate clients.
  • Handle xmlrpclib.Fault exceptions raised by the exposed XML-RPC methods as appropriate.
  • Dispatch XML-RPC method invocations to the appropriate method, including Plugin RMI. The client address pair (a tuple of remote IP address and remote hostname) must be prepended to the argument list passed to built-in methods (i.e., not to plugin RMI).

Additionally, running and configuring the server is delegated to the core. It needs to honor the configuration options that influence how and where the server runs, including the server location (host and port), listening interfaces, and SSL certificate and key.

Base Core

Bcfg2.Server.Core provides the base core object that server core implementations inherit from.

class Bcfg2.Server.Core.Core[source]

Bases: object

The server core is the container for all Bcfg2 server logic and modules. All core implementations must inherit from Core.

_run()[source]

Start up the server; this method should return immediately. This must be overridden by a core implementation.

_block()[source]

Enter the infinite loop. This method should not return until the server is killed. This must be overridden by a core implementation.

_file_monitor_thread()[source]

The thread that runs the Bcfg2.Server.FileMonitor.FileMonitor. This also queries Bcfg2.Server.Plugin.interfaces.Version plugins for the current revision of the Bcfg2 repo.

_perflog_thread()[source]

The thread that periodically logs performance statistics to syslog.

AssertProfile(*args, **kwargs)[source]

Set profile for a client.

Parameters:address (tuple) – Client (address, port) pair
Returns:bool - True on success
Raises :xmlrpclib.Fault
Bind(entry, metadata)[source]

Bind a single entry using the appropriate generator.

Parameters:
BindStructure(obj, *args, **kwargs)[source]

Bind all elements in a single structure (i.e., bundle).

Parameters:
BindStructures(obj, *args, **kwargs)[source]

Given a list of structures (i.e. bundles), bind all the entries in them and add the structures to the config.

Parameters:
  • structures (list of lxml.etree._Element objects) – The list of structures for this client
  • metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – Client metadata to bind structures for
  • config (lxml.etree._Element) – The configuration document to add fully-bound structures to. Modified in-place.
BuildConfiguration(client)[source]

Build the complete configuration for a client.

Parameters:client (string) – The hostname of the client to build the configuration for
Returns:lxml.etree._Element - A complete Bcfg2 configuration document
DeclareVersion(obj, *args, **kwargs)[source]

Declare the client version.

Parameters:
  • address (tuple) – Client (address, port) pair
  • version (string) – The client’s declared version
Returns:

bool - True on success

Raises :

xmlrpclib.Fault

GetConfig(*args, **kwargs)[source]

Build config for a client by calling BuildConfiguration().

Parameters:address (tuple) – Client (address, port) pair
Returns:lxml.etree._Element - The full configuration document for the client
Raises :xmlrpclib.Fault
GetDecisionList(*args, **kwargs)[source]

Get the decision list for the client with GetDecisions().

Parameters:address (tuple) – Client (address, port) pair
Returns:list of decision tuples
Raises :xmlrpclib.Fault
GetDecisions(metadata, mode)[source]

Get the decision list for a client.

Parameters:
Returns:

list of Decision tuples (<entry tag>, <entry name>)

GetProbes(*args, **kwargs)[source]

Fetch probes for the client.

Parameters:address (tuple) – Client (address, port) pair
Returns:lxml.etree._Element - XML tree describing probes for this client
Raises :xmlrpclib.Fault
GetStructures(obj, *args, **kwargs)[source]

Get all structures (i.e., bundles) for the given client

Parameters:metadata (Bcfg2.Server.Plugins.Metadata.ClientMetadata) – Client metadata to get structures for
Returns:list of lxml.etree._Element objects
HandleEvent(event)[source]

Handle a change in the Bcfg2 config file.

Parameters:event (Bcfg2.Server.FileMonitor.Event) – The event to handle
RecvProbeData(*args, **kwargs)[source]

Receive probe data from clients.

Parameters:address (tuple) – Client (address, port) pair
Returns:bool - True on success
Raises :xmlrpclib.Fault
RecvStats(*args, **kwargs)[source]

Act on statistics upload with process_statistics().

Parameters:address (tuple) – Client (address, port) pair
Returns:bool - True on success
Raises :xmlrpclib.Fault
block_for_fam_events(handle_events=False)[source]

Block until all fam events have been handleed, optionally handling events as well. (Setting handle_events=True is useful for local server cores that don’t spawn an event handling thread.)

build_metadata(obj, *args, **kwargs)[source]

Build initial client metadata for a client

Parameters:client_name (string) – The name of the client to build metadata for
Returns:Bcfg2.Server.Plugins.Metadata.ClientMetadata
cfile = None

Path to bcfg2.conf

check_acls(obj, *args, **kwargs)[source]

Check client IP address and metadata object against all Bcfg2.Server.Plugin.interfaces.ClientACLs plugins. If any ACL plugin denies access, then access is denied. ACLs are checked in two phases: First, with the client IP address; and second, with the client metadata object. This lets an ACL interface do a quick rejection based on IP before metadata is ever built.

Parameters:
  • address (tuple of (<ip address>, <port>)) – The address pair of the client to check ACLs for
  • rmi – The fully-qualified name of the RPC call
  • rmi – string
Returns:

bool

client_run_hook(hook, metadata)[source]

Invoke hooks from Bcfg2.Server.Plugin.interfaces.ClientRunHooks plugins for a given stage.

Parameters:
critical_error(message)[source]

Log an error with its traceback and return an XML-RPC fault to the client.

Parameters:message (string) – The message to log and return to the client
Raises :xmlrpclib.Fault
database_available[source]

True if the database is configured and available, False otherwise.

db_write_lock = None

RLock to be held on writes to the backend db

debug_flag = None

Used to keep track of the current debug state of the core.

expire_metadata_cache(_, hostnames=None)

Expire the metadata cache for one or all clients

Parameters:hostnames (None or list of strings) – A list of hostnames to expire the metadata cache for or None. If None the cache of all clients will be expired.
fam = None

The Bcfg2.Server.FileMonitor.FileMonitor object used by the core to monitor for Bcfg2 data changes.

fam_thread = None

The FAM threading.Thread, _file_monitor_thread()

get_statistics(_)[source]

Get current statistics about component execution from Bcfg2.Server.Statistics.stats.

Returns:dict - The statistics data as returned by Bcfg2.Server.Statistics.Statistics.display()
init_plugin(plugin)[source]

Import and instantiate a single plugin. The plugin is stored to plugins.

Parameters:plugin (type) – The plugin class to load.
Returns:None
listMethods(address)[source]

List all exposed methods, including plugin RMI.

Parameters:address (tuple) – Client (address, port) pair
Returns:list of exposed method names
load_plugins()[source]

Load all plugins, setting Bcfg2.Server.Core.BaseCore.plugins and Bcfg2.Server.Core.BaseCore.metadata as side effects. This does not start plugin threads; that is done later, in Bcfg2.Server.Core.BaseCore.run()

lock = None

A threading.Lock() for use by Bcfg2.Server.FileMonitor.FileMonitor.handle_event_set()

logger = None

A logging.Logger object for use by the core

metadata = None

The Metadata plugin

metadata_cache = None

A Bcfg2.Server.Cache.Cache object for caching client metadata

metadata_cache_mode[source]

Get the client metadata_cache mode. Options are off, initial, cautious, aggressive, on (synonym for cautious). See Server-side Caching for more details.

methodHelp(address, method_name)[source]

Get help from the docstring of an exposed method

Parameters:
  • address (tuple) – Client (address, port) pair
  • method_name (string) – The name of the method to get help on
Returns:

string - The help message from the method’s docstring

name = 'Core'

The name of this server core. This can be overridden by core implementations to provide a more specific name.

perflog_thread = None

The threading.Thread that reports performance statistics to syslog.

plugin_blacklist = None

Blacklist of plugins that conflict with enabled plugins. If two plugins are loaded that conflict with each other, the first one loaded wins.

plugins = None

Dict of plugins that are enabled. Keys are the plugin names (just the plugin name, in the correct case; e.g., “Cfg”, not “Bcfg2.Server.Plugins.Cfg”), and values are plugin objects.

plugins_by_type(base_cls)[source]

Return a list of loaded plugins that match the passed type.

The returned list is sorted in ascending order by the plugins’ sort_order value. The Bcfg2.Server.Plugin.base.Plugin.sort_order defaults to 500, but can be overridden by individual plugins. Plugins with the same numerical sort_order value are sorted in alphabetical order by their name.

Parameters:base_cls (type) – The base plugin interface class to match (see Bcfg2.Server.Plugin.interfaces)
Returns:list of Bcfg2.Server.Plugin.base.Plugin objects
process_statistics(client_name, statistics)[source]

Process uploaded statistics for client.

Parameters:
  • client_name (string) – The name of the client to process statistics for
  • statistics (lxml.etree._Element) – The statistics document to process
resolve_client(obj, *args, **kwargs)[source]

Given a client address, get the client hostname and optionally metadata.

Parameters:
Returns:

tuple - If metadata is False, returns (<canonical hostname>, None); if metadata is True, returns (<canonical hostname>, <client metadata object>)

revision = None

Revision of the Bcfg2 specification. This will be sent to the client in the configuration, and can be set by a Bcfg2.Server.Plugin.interfaces.Version plugin.

run()[source]

Run the server core. This calls _run(), starts the fam_thread, and calls _block(), but note that it is the responsibility of the server core implementation to call shutdown() under normal operation. This also handles creation of the directory containing the pidfile, if necessary.

set_core_debug(_, debug)[source]

Explicity set debug status of the server core

Parameters:debug (bool or string) – The new debug status. This can either be a boolean, or a string describing the state (e.g., “true” or “false”; case-insensitive)
Returns:bool - The new debug state of the FAM
set_debug(address, debug)[source]

Explicitly set debug status of the FAM and all plugins

Parameters:
  • address (tuple) – Client (address, hostname) pair
  • debug (bool or string) – The new debug status. This can either be a boolean, or a string describing the state (e.g., “true” or “false”; case-insensitive)
Returns:

bool - The new debug state

set_fam_debug(address, debug)[source]

Explicitly set debug status of the FAM

Parameters:debug (bool or string) – The new debug status of the FAM. This can either be a boolean, or a string describing the state (e.g., “true” or “false”; case-insensitive)
Returns:bool - The new debug state of the FAM
shutdown(*args, **kwargs)[source]

Perform plugin and FAM shutdown tasks.

terminate = None

Threading event to signal worker threads (e.g., fam_thread) to shutdown

toggle_core_debug(address)[source]

Toggle debug status of the server core

Parameters:address (tuple) – Client (address, hostname) pair
Returns:bool - The new debug state of the FAM
toggle_debug(address)[source]

Toggle debug status of the FAM and all plugins

Parameters:address (tuple) – Client (address, port) pair
Returns:bool - The new debug state of the FAM
toggle_fam_debug(address)[source]

Toggle debug status of the FAM

Returns:bool - The new debug state of the FAM
validate_goals(obj, *args, **kwargs)[source]

Checks that the config matches the goals enforced by Bcfg2.Server.Plugin.interfaces.GoalValidator plugins by calling Bcfg2.Server.Plugin.interfaces.GoalValidator.validate_goals().

Parameters:
validate_structures(obj, *args, **kwargs)[source]

Checks the data structures by calling the Bcfg2.Server.Plugin.interfaces.StructureValidator.validate_structures() method of Bcfg2.Server.Plugin.interfaces.StructureValidator plugins.

Parameters:
exception Bcfg2.Server.Core.CoreInitError[source]

Bases: exceptions.Exception

Raised when the server core cannot be initialized.

class Bcfg2.Server.Core.DefaultACL(core)[source]

Bases: Bcfg2.Server.Plugin.base.Plugin, Bcfg2.Server.Plugin.interfaces.ClientACLs

Default ACL ‘plugin’ that provides security by default. This is only loaded if no other ClientACLs plugin is enabled.

class Bcfg2.Server.Core.NetworkCore[source]

Bases: Bcfg2.Server.Core.Core

A server core that actually listens on the network, can be daemonized, etc.

_run()

Start up the server; this method should return immediately. This must be overridden by a core implementation.

_block()

Enter the infinite loop. This method should not return until the server is killed. This must be overridden by a core implementation.

_daemonize()[source]

Daemonize the server and write the pidfile. This must be overridden by a core implementation.

authenticate(cert, user, password, address)[source]

Authenticate a client connection with Bcfg2.Server.Plugin.interfaces.Metadata.AuthenticateConnection().

Parameters:
  • cert (dict) – an x509 certificate
  • user (string) – The username of the user trying to authenticate
  • password (string) – The password supplied by the client
  • address (tuple) – An address pair of (<ip address>, <port>)
Returns:

bool - True if the authenticate succeeds, False otherwise

ca = None

The CA that signed the server cert

run()[source]

Run the server core. This calls _daemonize() before calling Bcfg2.Server.Core.Core.run() to run the server core.

exception Bcfg2.Server.Core.NoExposedMethod[source]

Bases: exceptions.Exception

Raised when an XML-RPC method is called, but there is no method exposed with the given name.

Bcfg2.Server.Core.close_db_connection(func)[source]

Decorator that closes the Django database connection at the end of the function. This should decorate any exposed function that might open a database connection.

Bcfg2.Server.Core.exposed(func)[source]

Decorator that sets the exposed attribute of a function to True expose it via XML-RPC. This currently works for both the builtin and CherryPy cores, although if other cores are added this may need to be made a core-specific function.

Parameters:func (callable) – The function to decorate
Returns:callable - the decorated function
Bcfg2.Server.Core.sort_xml(node, key=None)[source]

Recursively sort an XML document in a deterministic fashion. This shouldn’t be used to perform a useful sort, merely to put XML in a deterministic, replicable order. The document is sorted in-place.

Parameters:
  • node (lxml.etree._Element or lxml.etree.ElementTree) – The root node of the XML tree to sort
  • key (callable) – The key to sort by
Returns:

None

Core Implementations

Builtin Core

The builtin server core consists of the core implementation (Bcfg2.Server.BuiltinCore.Core) and the XML-RPC server implementation (Bcfg2.Server.SSLServer).

Core

The core of the builtin Bcfg2 server.

class Bcfg2.Server.BuiltinCore.BuiltinCore[source]

Bases: Bcfg2.Server.Core.NetworkCore

The built-in server core

_run()[source]

Create server to start the server listening.

_block()[source]

Enter the blocking infinite loop.

_daemonize()[source]

Open context to drop privileges, write the PID file, and daemonize the server core.

context = None

The daemon.DaemonContext used to drop privileges, write the PID file (with PidFile), and daemonize this core.

server = None

The Bcfg2.Server.SSLServer.XMLRPCServer instance powering this server core

XML-RPC Server

Bcfg2 SSL server used by the builtin server core (Bcfg2.Server.BuiltinCore). This needs to be documented better.

class Bcfg2.Server.SSLServer.SSLServer(listen_all, server_address, RequestHandlerClass, keyfile=None, certfile=None, reqCert=False, ca=None, timeout=None, protocol='xmlrpc/tlsv1')[source]

Bases: SocketServer.TCPServer, object

TCP server supporting SSL encryption.

Parameters:
  • listen_all (bool) – Listen on all interfaces
  • server_address – Address to bind to the server
  • RequestHandlerClass – Request handler used by TCP server
  • keyfile (string) – Full path to SSL encryption key file
  • certfile (string) – Full path to SSL certificate file
  • reqCert (bool) – Require client to present certificate
  • ca (string) – Full path to SSL CA that signed the key and cert
  • timeout – Timeout for non-blocking request handling
  • protocol (string) – The protocol to serve. Supported values are xmlrpc/ssl and xmlrpc/tlsv1.
exception Bcfg2.Server.SSLServer.XMLRPCACLCheckException[source]

Bases: exceptions.Exception

Raised when ACL checks fail on an RPC request

class Bcfg2.Server.SSLServer.XMLRPCDispatcher(allow_none, encoding)[source]

Bases: SimpleXMLRPCServer.SimpleXMLRPCDispatcher

An XML-RPC dispatcher.

class Bcfg2.Server.SSLServer.XMLRPCRequestHandler(*args, **kwargs)[source]

Bases: SimpleXMLRPCServer.SimpleXMLRPCRequestHandler

XML-RPC request handler.

Adds support for HTTP authentication.

parse_request()[source]

Extends parse_request.

Optionally check HTTP authentication when parsing.

class Bcfg2.Server.SSLServer.XMLRPCServer(listen_all, server_address, RequestHandlerClass=None, keyfile=None, certfile=None, ca=None, protocol='xmlrpc/tlsv1', timeout=10, logRequests=False, register=True, allow_none=True, encoding=None)[source]

Bases: SocketServer.ThreadingMixIn, Bcfg2.Server.SSLServer.SSLServer, Bcfg2.Server.SSLServer.XMLRPCDispatcher, object

Component XMLRPCServer.

Parameters:
  • listen_all (bool) – Listen on all interfaces
  • server_address – Address to bind to the server
  • RequestHandlerClass – request handler used by TCP server
  • keyfile (string) – Full path to SSL encryption key file
  • certfile (string) – Full path to SSL certificate file
  • ca (string) – Full path to SSL CA that signed the key and cert
  • logRequests (bool) – Log all requests
  • register (bool) – Presence should be reported to service-location
  • allow_none (bool) – Allow None values in XML-RPC
  • encoding – Encoding to use for XML-RPC
ping(*args)[source]

Echo response.

serve_forever()[source]

Serve single requests until (self.serve == False).

shutdown()[source]

Signal that automatic service should stop.

Multiprocessing Core

The multiprocessing server core is a reimplementation of the Bcfg2.Server.BuiltinCore that uses the Python multiprocessing library to offload work to multiple child processes. As such, it requires Python 2.6+.

The parent communicates with the children over multiprocessing.Queue objects via a Bcfg2.Server.MultiprocessingCore.RPCQueue object.

A method being called via the RPCQueue must be exposed by the child by decorating it with Bcfg2.Server.Core.exposed().

class Bcfg2.Server.MultiprocessingCore.ChildCore(name, rpc_q, terminate)[source]

Bases: Bcfg2.Server.Core.Core

A child process for Bcfg2.MultiprocessingCore.Core. This core builds configurations from a given multiprocessing.Pipe. Note that this is a full-fledged server core; the only input it gets from the parent process is the hostnames of clients to render. All other state comes from the FAM. However, this core only is used to render configs; it doesn’t handle anything else (authentication, probes, etc.) because those are all much faster. There’s no reason that it couldn’t handle those, though, if the pipe communication “protocol” were made more robust.

Parameters:
  • name (string) – The name of this child
  • read_q (multiprocessing.Queue) – The queue the child will read from for RPC communications from the parent process.
  • write_q (multiprocessing.Queue) – The queue the child will write the results of RPC calls to.
  • terminate (multiprocessing.Event) – An event that flags ChildCore objects to shut themselves down.
GetConfig(client)[source]

Render the configuration for a client

expire_cache(*tags, **kwargs)[source]

Expire cached data

name = None

The name of this child

poll_wait = 3.0

How long to wait while polling for new RPC commands. This doesn’t affect the speed with which a command is processed, but setting it too high will result in longer shutdown times, since we only check for the termination event from the main process every poll_wait seconds.

rpc_q = None

The queue used for RPC communication

terminate = None

The multiprocessing.Event that will be monitored to determine when this child should shut down.

class Bcfg2.Server.MultiprocessingCore.DualEvent(threading_event=None, multiprocessing_event=None)[source]

Bases: object

DualEvent is a clone of threading.Event that internally implements both threading.Event and multiprocessing.Event.

clear()[source]

Reset the internal flag to false.

isSet()

Return true if and only if the internal flag is true.

is_set()[source]

Return true if and only if the internal flag is true.

set()[source]

Set the internal flag to true.

wait(timeout=None)[source]

Block until the internal flag is true, or until the optional timeout occurs.

class Bcfg2.Server.MultiprocessingCore.MultiprocessingCore[source]

Bases: Bcfg2.Server.BuiltinCore.BuiltinCore

A multiprocessing core that delegates building the actual client configurations to Bcfg2.Server.MultiprocessingCore.ChildCore objects. The parent process doesn’t build any children itself; all calls to GetConfig() are delegated to children. All other calls are handled by the parent process.

available_children = None

A queue that keeps track of which children are available to render a configuration. A child is popped from the queue when it starts to render a config, then it’s pushed back on when it’s done. This lets us use a blocking call to Queue.Queue.get() when waiting for an available child.

cache_dispatch(tags, exact, _)[source]

Publish cache expiration events to child nodes.

children = None

An iterator that each child will be taken from in sequence, to provide a round-robin distribution of render requests

pipes = None

A dict of child name -> one end of the multiprocessing.Pipe object used to communicate with that child. (The child is given the other end of the Pipe.)

rpc_q = None

A Bcfg2.Server.MultiprocessingCore.RPCQueue object used to send or publish commands to children.

shutdown_timeout = 10.0

How long to wait for a child process to shut down cleanly before it is terminated.

terminate = None

The flag that indicates when to stop child threads and processes

class Bcfg2.Server.MultiprocessingCore.RPCQueue[source]

Bases: Bcfg2.Logger.Debuggable

An implementation of a multiprocessing.Queue designed for several additional use patterns:

  • Random-access reads, based on a key that identifies the data;
  • Publish-subscribe, where a datum is sent to all hosts.

The subscribers can deal with this as a normal Queue with no special handling.

add_subscriber(name)[source]

Add a subscriber to the queue. This returns the multiprocessing.Queue object that the subscriber should read from.

close()[source]

Close queues and connections.

publish(method, args=None, kwargs=None)[source]

Publish an RPC call to the queue for consumption by all subscribers.

rpc(dest, method, args=None, kwargs=None)[source]

Make an RPC call to the named subscriber, expecting a response. This opens a multiprocessing.connection.Listener and passes the Listener address to the child as part of the RPC call, so that the child can connect to the Listener to submit its results.

CherryPy Core

The core of the CherryPy-powered server.

class Bcfg2.Server.CherrypyCore.CherrypyCore[source]

Bases: Bcfg2.Server.Core.NetworkCore

The CherryPy-based server core.

_run()[source]

Start the server listening.

_block()[source]

Enter the blocking infinite server loop. Bcfg2.Server.Core.NetworkCore.shutdown() is called on exit by a subscription on the top-level CherryPy engine.

_daemonize()[source]

Drop privileges, daemonize with cherrypy.process.plugins.Daemonizer and write a PID file with cherrypy.process.plugins.PIDFile.

default(*args, **params)[source]

Handle all XML-RPC calls. It was necessary to make enough changes to the stock CherryPy cherrypy._cptools.XMLRPCController to support plugin RMI and prepending the client address that we just rewrote it. It clearly wasn’t written with inheritance in mind.

do_authn()[source]

Perform authentication by calling Bcfg2.Server.Core.NetworkCore.authenticate(). This is implemented as a CherryPy tool.

rmi = None

List of exposed plugin RMI

Bcfg2.Server.CherrypyCore.on_error(*args, **kwargs)[source]

CherryPy error handler that handles xmlrpclib.Fault objects and so allows for the possibility of returning proper error codes. This obviates the need to use cherrypy.lib.xmlrpc.on_error(), the builtin CherryPy xmlrpc tool, which does not handle xmlrpclib.Fault objects and returns the same error code for every error.

Table Of Contents

Previous topic

Python Compatibility

Next topic

Documentation

This Page