You will first need to install the Python Mock Module and Python Nose modules. You can then run the existing tests with the following:
You should see output something like the following:
..................................................
----------------------------------------------------------------------
Ran 50 tests in 0.121s
OK
Unit tests are also run by Travis-CI, a free continuous integration service, at http://travis-ci.org/#!/Bcfg2/bcfg2/
Travis-CI runs the unit tests in a virtual environment, so to emulate that testing environment as closely as possible you can also use a virtual environment. To do so, you must have virtualenv installed.
There are two ways to test: Either with just the bare essential packages installed, or with optional packages installed as well. (Optional packages are things like Genshi; you can run Bcfg2 with them or without them.) For completeness, the tests should be run in both manners. (On Python 3, almost none of the optional packages are available, so it can only be run with just the required packages.) To install the optional packages, set:
export WITH_OPTIONAL_DEPS=yes
This flag tells the install script to install optional dependencies as well as requirements.
This assumes that you will create a virtual environment in ~/venvs/, and that the Bcfg2 source tree is cloned into ~/bcfg2.
First, create a new virtual environment and activate it:
cd ~/venvs
virtualenv travis
source travis/bin/activate
Get the test suite from bcfg2:
cp -R ~/bcfg2/* ~/venvs/travis/
Next, you must install prerequisite packages that are required to build some of the required Python packages, and some optional packages that are much easier to install from binary (rather than from source). If you are running on Ubuntu (the platform Travis-CI runs on) and have sudo, you can simply run:
testsuite/before_install.sh
If not, you will need to examine testsuite/before_install.sh and install the packages manually. The equivalent for Fedora, for instance, would be:
sudo yum -y update
sudo yum -y install swig pylint libxml2
if [[ "$WITH_OPTIONAL_DEPS" == "yes" ]]; then
sudo yum -y install libselinux-python pylibacl python-inotify \
PyYAML
fi
You could install these requirements using pip, but you’ll likely need to install a great many development packages required to compile them.
Next, install required Python packages:
testsuite/install.sh
Install Bcfg2 itself to the virtualenv:
pip install -e .
Now you can run tests:
nosetests testsuite
Bcfg2 makes extremely heavy use of object inheritance, which can make it challenging at times to write reusable tests. For instance, when writing tests for the base Bcfg2.Server.Plugin.base.Plugin class, which all Bcfg2 Plugins inherit from via the Plugin interfaces, yielding several levels of often-multiple inheritance. To make this easier, our unit tests adhere to several design considerations:
Our test objects should have inheritance trees that mirror the inheritance trees of their tested objects. For instance, the Bcfg2.Server.Plugins.Metadata.Metadata class definition is:
class Metadata(Bcfg2.Server.Plugin.Metadata,
Bcfg2.Server.Plugin.Statistics,
Bcfg2.Server.Plugin.DatabaseBacked):
Consequently, the TestMetadata class definition is:
class TestMetadata(TestPlugin.TestMetadata,
TestPlugin.TestStatistics,
TestPlugin.TestDatabaseBacked):
Note
The test object names are abbreviated because of the system of relative imports in the testsuite tree, described below.
This gives us a large number of tests basically “for free”: all core Bcfg2.Server.Plugin.interfaces.Metadata, Bcfg2.Server.Plugin.interfaces.Statistics, and Bcfg2.Server.Plugin.helpers.DatabaseBacked functionality is automatically tested on the Metadata class, which gives the test writer a lot of free functionality and also an easy list of which tests must be overridden to provide tests appropriate for the Metadata class implementation.
Additionally, a test class should have a class variable that describes the class that is being tested, and tests in that class should use that class variable to instantate the tested object. For instance, the test for Bcfg2.Server.Plugin.helpers.DirectoryBacked looks like this:
class TestDirectoryBacked(Bcfg2TestCase):
test_obj = DirectoryBacked
...
def test_child_interface(self):
""" ensure that the child object has the correct interface """
self.assertTrue(hasattr(self.test_obj.__child__, "HandleEvent"))
Then test objects that inherit from TestDirectoryBacked can override that object, and the test_child_interface test (e.g.) will still work. For example:
class TestPropDirectoryBacked(TestDirectoryBacked):
test_obj = PropDirectoryBacked
Finally, each test class must also provide a get_obj method that takes no required arguments and produces an instance of test_obj. All test methods must use self.get_obj() to instantiate an object to be tested.
An object that does not inherit from any other tested Bcfg2 objects should inherit from testsuite.common.Bcfg2TestCase, described below.
In order to reuse test code and allow for test inheritance, each test module should add all parent module paths to its sys.path. For instance, assuming a test in testsuite/Testsrc/Testlib/TestServer/TestPlugins/TestMetadata.py, the following paths should be added to sys.path:
testsuite
testsuite/Testsrc
testsuite/Testsrc/Testlib
testsuite/Testsrc/Testlib/TestServer
testsuite/Testsrc/Testlib/TestServer/TestPlugins
This must be done because Python 2.4, one of our target platforms, does not support relative imports. An easy way to do this is to add the following snippet to the top of each test file:
import os
import sys
# add all parent testsuite directories to sys.path to allow (most)
# relative imports in python 2.4
path = os.path.dirname(__file__)
while path != "/":
if os.path.basename(path).lower().startswith("test"):
sys.path.append(path)
if os.path.basename(path) == "testsuite":
break
path = os.path.dirname(path)
In addition, each new directory created in testsuite must contain an empty __init__.py.
This will allow you, within TestMetadata.py, to import common test code and the parent objects the TestMetadata class will inherit from:
from common import inPy3k, call, builtins, u, can_skip, \
skip, skipIf, skipUnless, Bcfg2TestCase, DBModelTestCase, syncdb, \
patchIf, datastore
from TestPlugin import TestXMLFileBacked, TestMetadata as _TestMetadata, \
TestStatistics, TestDatabaseBacked
The Python Mock Module provides a patch decorator that can be used to replace tested objects with Mock objects. This is wonderful and necessary, but due to differences in the way various versions of Python and Python Mock handle object scope, it’s not always reliable when combined with our system of test object inheritance. Consequently, you should follow these rules when considering whether to use patch:
As an example of the last rule, assume you are writing tests for Bcfg2.Server.Plugin.helpers.FileBacked. Bcfg2.Server.Plugin.helpers.FileBacked.HandleEvent() calls Bcfg2.Server.Plugin.helpers.FileBacked.Index(), so we need to mock the Index function. This is the wrong way to do that:
class TestFileBacked(Bcfg2TestCase):
@patch("%s.open" % builtins)
@patch("Bcfg2.Server.Plugin.helpers.FileBacked.Index")
def test_HandleEvent(self, mock_Index, mock_open):
...
Tests that inherit from TestFileBacked will not reliably patch the correct Index function. Instead, assign the object to be mocked directly:
class TestFileBacked(Bcfg2TestCase):
@patch("%s.open" % builtins)
def test_HandleEvent(self, mock_open):
fb = self.get_obj()
fb.Index = Mock()
Note
@patch decorations are evaluated at compile-time, so a workaround like this does not work:
class TestFileBacked(Bcfg2TestCase):
@patch("%s.open" % builtins)
@patch("%s.%s.Index" % (self.test_obj.__module__,
self.test_obj.__name))
def test_HandleEvent(self, mock_Index, mock_open):
...
But see below about patching objects before instantiation.
In some cases, you will need to patch an object in order to instantiate it. For instance, consider Bcfg2.Server.Plugin.helpers.DirectoryBacked, which attempts to set a file access monitor watch when it is instantiated. This won’t work during unit testing, so we have to patch Bcfg2.Server.Plugin.helpers.DirectoryBacked.add_directory_monitor() in order to successfully instantiate a DirectoryBacked object. In order to do that, we need to patch the object being tested, which is a variable, but we need to evaluate the patch at run-time, not at compile time, in order to deal with inheritance. This can be done with a @patch decorator on an inner function, e.g.:
class TestDirectoryBacked(Bcfg2TestCase):
test_obj = DirectoryBacked
def test__init(self):
@patch("%s.%s.add_directory_monitor" % (self.test_obj.__module__,
self.test_obj.__name__))
def inner(mock_add_monitor):
db = self.test_obj(datastore, Mock())
mock_add_monitor.assert_called_with('')
inner()
inner() is patched when test__init() is called, and so @patch() is called with the module and the name of the object being tested as defined by the test object (i.e., not as defined by the parent object). If this is not done, then the patch will be applied at compile-time and add_directory_monitor will be patched on the DirectoryBacked class instead of on the class to be tested.
Some of our older unit tests do not follow these rules religiously, so as more tests are written that inherit from larger portions of the testsuite tree they may need to be refactored.
In order to make the system of inheritance we implement possible, we must follow these naming conventions fairly religiously.
In order to make testing easier and more consistent, we provide a number of convenience functions, variables, and classes, for a wide variety of reasons. To import this module, first set up Relative Imports and then simply do:
from common import *
Bases: unittest.case.TestCase
Base TestCase class that inherits from unittest.TestCase. This class does a few things:
Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.
Bases: testsuite.common.Bcfg2TestCase
Test case class for Django database models
Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.
The XInclude namespace in a format suitable for use in XPath expressions
The XInclude namespace name
The name of the builtin module, for mocking Python builtins. In Python 2, this is __builtin__, in Python 3 builtins. To patch a builtin, you must do something like:
@patch("%s.open" % open)
def test_something(self, mock_open):
...
Whether or not skipping tests is natively supported by unittest. If it isn’t, then we have to make tests that would be skipped succeed instead.
The path to the Bcfg2 specification root for the tests. Using the root directory exposes a lot of potential problems with building paths.
Whether or not the tests are being run on Python 3.
Bases: object
Decorator class to perform conditional patching. This is necessary because some libraries might not be installed (e.g., selinux, pylibacl), and patching will barf on that. Other workarounds are not available to us; e.g., context managers aren’t in python 2.4, and using inner functions doesn’t work because python 2.6 parses all decorators at compile-time, not at run-time, so decorating inner functions does not prevent the decorators from being run.
| Parameters: |
|
|---|
The type of compiled regular expression objects
alias of SRE_Pattern
skip decorator from unittest.skip()
skipIf decorator from unittest.skipIf()
skipUnless decorator from unittest.skipUnless()
Given an instance of a DBModelTestCase object, sync and clean the database