keystone/doc/source/contributor/testing-keystone.rst

Testing Keystone

Running Tests

Before running tests, you should have tox installed and available in your environment (in addition to the other external dependencies in dev-environment):

$ pip install tox

Note

You may need to perform both the above operation and the next inside a python virtualenv, or prefix the above command with sudo, depending on your preference.

To execute the full suite of tests maintained within Keystone, simply run:

$ tox

This iterates over multiple configuration variations, and uses external projects to do light integration testing to verify the Identity API against other projects.

Note

The first time you run tox, it will take additional time to build virtualenvs. You can later use the -r option with tox to rebuild your virtualenv in a similar manner.

To run tests for one or more specific test environments (for example, the most common configuration of Python 2.7 and PEP-8), list the environments with the -e option, separated by spaces:

$ tox -e py27,pep8

See tox.ini for the full list of available test environments.

Running with PDB

Using PDB breakpoints with tox and testr normally doesn't work since the tests just fail with a BdbQuit exception rather than stopping at the breakpoint.

To run with PDB breakpoints during testing, use the debug tox environment rather than py27. Here's an example, passing the name of a test since you'll normally only want to run the test that hits your breakpoint:

$ tox -e debug keystone.tests.unit.test_auth.AuthWithToken.test_belongs_to

For reference, the debug tox environment implements the instructions here: https://wiki.openstack.org/wiki/Testr#Debugging_.28pdb.29_Tests

Disabling Stream Capture

The stdout, stderr and log messages generated during a test are captured and in the event of a test failure those streams will be printed to the terminal along with the traceback. The data is discarded for passing tests.

Each stream has an environment variable that can be used to force captured data to be discarded even if the test fails: OS_STDOUT_CAPTURE for stdout, OS_STDERR_CAPTURE for stderr and OS_LOG_CAPTURE for logging. If the value of the environment variable is not one of (True, true, 1, yes) the stream will be discarded. All three variables default to 1.

For example, to discard logging data during a test run:

$ OS_LOG_CAPTURE=0 tox -e py27

Building the Documentation

The documentation is generated with Sphinx using the tox command. To create HTML docs and man pages:

$ tox -e docs

The results are in the doc/build/html and doc/build/man directories respectively.

Tests Structure

Not all of the tests in the keystone/tests/unit directory are strictly unit tests. Keystone intentionally includes tests that run the service locally and drives the entire configuration to achieve basic functional testing.

For the functional tests, an in-memory key-value store or in-memory SQLite database is used to keep the tests fast.

Within the tests directory, the general structure of the backend tests is a basic set of tests represented under a test class, and then subclasses of those tests under other classes with different configurations to drive different backends through the APIs. To add tests covering all drivers, update the base test class in test_backend.py.

Note

The structure of backend testing is in transition, migrating from having all classes in a single file (test_backend.py) to one where there is a directory structure to reduce the size of the test files. See:

• keystone.tests.unit.backend.role
• keystone.tests.unit.backend.domain_config

To add new drivers, subclass the base class at test_backend.py (look towards test_backend_sql.py for examples) and update the configuration of the test class in setUp().

For example, test_backend.py has a sequence of tests under the class ~keystone.tests.unit.test_backend.IdentityTests that will work with the default drivers as configured in this project's etc/ directory. test_backend_sql.py subclasses those tests, changing the configuration by overriding with configuration files stored in the tests/unit/config_files directory aimed at enabling the SQL backend for the Identity module.

keystone.tests.unit.test_v2_keystoneclient.ClientDrivenTestCase uses the installed python-keystoneclient, verifying it against a temporarily running local keystone instance to explicitly verify basic functional testing across the API.

Testing Schema Migrations

The application of schema migrations can be tested using SQLAlchemy Migrate’s built-in test runner, one migration at a time.

Warning

This may leave your database in an inconsistent state; attempt this in non-production environments only!

This is useful for testing the next migration in sequence (both forward & backward) in a database under version control:

$ python keystone/common/sql/migrate_repo/manage.py test \
--url=sqlite:///test.db \
--repository=keystone/common/sql/migrate_repo/

This command references to a SQLite database (test.db) to be used. Depending on the migration, this command alone does not make assertions as to the integrity of your data during migration.

LDAP Tests

LDAP has a fake backend that performs rudimentary operations. If you are building more significant LDAP functionality, you should test against a live LDAP server. Devstack has an option to set up a directory server for Keystone to use. Add ldap to the ENABLED_SERVICES environment variable, and set environment variables KEYSTONE_IDENTITY_BACKEND=ldap and KEYSTONE_CLEAR_LDAP=yes in your localrc file.

The unit tests can be run against a live server with keystone/tests/unit/test_ldap_livetest.py and keystone/tests/unit/test_ldap_pool_livetest.py. The default password is test but if you have installed devstack with a different LDAP password, modify the file keystone/tests/unit/config_files/backend_liveldap.conf and keystone/tests/unit/config_files/backend_pool_liveldap.conf to reflect your password.

Note

To run the live tests you need to set the environment variable ENABLE_LDAP_LIVE_TEST to a non-negative value.

"Work in progress" Tests

Work in progress (WIP) tests are very useful in a variety of situations including:

• During a TDD process they can be used to add tests to a review while they are not yet working and will not cause test failures. (They should be removed before the final merge.)
• Often bug reports include small snippets of code to show broken behaviors. Some of these can be converted into WIP tests that can later be worked on by a developer. This allows us to take code that can be used to catch bug regressions and commit it before any code is written.

The keystone.tests.unit.utils.wip decorator can be used to mark a test as WIP. A WIP test will always be run. If the test fails then a TestSkipped exception is raised because we expect the test to fail. We do not pass the test in this case so that it doesn't count toward the number of successfully run tests. If the test passes an AssertionError exception is raised so that the developer knows they made the test pass. This is a reminder to remove the decorator.

The ~keystone.tests.unit.utils.wip decorator requires that the author provides a message. This message is important because it will tell other developers why this test is marked as a work in progress. Reviewers will require that these messages are descriptive and accurate.

Note

The ~keystone.tests.unit.utils.wip decorator is not a replacement for skipping tests.

@wip('waiting on bug #000000')
def test():
    pass

Note

Another strategy is to not use the wip decorator and instead show how the code currently incorrectly works. Which strategy is chosen is up to the developer.

API & Scenario Tests

Keystone provides API and scenario tests via a tempest plugin located at ~keystone.keystone_tempest_plugin. This tempest plugin is mainly intended for specific scenarios that require a special deployment, such as the tests for the Federated Identity feature. For the deployment of these scenarios, keystone also provides a devstack plugin.

For example, to setup a working federated environment, add the following lines in your devstack local.conf` file:

[[local|localrc]]
enable_plugin keystone git://git.openstack.org/openstack/keystone
enable_service keystone-saml2-federation

Finally, to run keystone's API and scenario tests, deploy tempest with devstack (using the configuration above) and then run the following command from the tempest directory:

tox -e all-plugin -- keystone_tempest_plugin

Note

Most of keystone's API tests are implemented in tempest and it is usually the correct place to add new tests.

Writing new API & Scenario Tests

When writing tests for the keystone tempest plugin, we should follow the official tempest guidelines, details about the guidelines can be found at the tempest coding guide. There are also specific guides for the API and scenario tests: Tempest Field Guide to API tests and Tempest Field Guide to Scenario tests.

The keystone tempest plugin also provides a base class. For most cases, the tests should inherit from it: keystone_tempest_plugin.tests.base.BaseIdentityTest. This class already setups the identity API version and is the container of all API services clients. New API services clients keystone_tempest_plugin.services (which are used to communicate with the REST API from the services) should also be added to this class. For example, below we have a snippet from the tests at :pykeystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.py.

class IdentityProvidersTest(base.BaseIdentityTest):

...

def _create_idp(self, idp_id, idp_ref):
    idp = self.idps_client.create_identity_provider(
        idp_id, **idp_ref)['identity_provider']
    self.addCleanup(
        self.idps_client.delete_identity_provider, idp_id)
    return idp

@decorators.idempotent_id('09450910-b816-4150-8513-a2fd4628a0c3')
def test_identity_provider_create(self):
    idp_id = data_utils.rand_uuid_hex()
    idp_ref = fixtures.idp_ref()
    idp = self._create_idp(idp_id, idp_ref)

    # The identity provider is disabled by default
    idp_ref['enabled'] = False

    # The remote_ids attribute should be set to an empty list by default
    idp_ref['remote_ids'] = []

    self._assert_identity_provider_attributes(idp, idp_id, idp_ref)

The test class extends keystone_tempest_plugin.tests.base.BaseIdentityTest. Also, the _create_idp method calls keystone's API using the idps_client, which is an instance from. keystone_tempest_plugin.tests.services.identity.v3.identity_providers_client.IdentityProvidersClient.

Additionally, to illustrate the construction of a new test class, below we have a snippet from the scenario test that checks the complete federated authentication workflow ( :pykeystone_tempest_plugin.tests.scenario.test_federated_authentication.py). In the test setup, all of the needed resources are created using the API service clients. Since it is a scenario test, it is common to need some customized settings that will come from the environment (in this case, from the devstack plugin) - these settings are collected in the _setup_settings method.

class TestSaml2EcpFederatedAuthentication(base.BaseIdentityTest):

...

def _setup_settings(self):
    self.idp_id = CONF.fed_scenario.idp_id
    self.idp_url = CONF.fed_scenario.idp_ecp_url
    self.keystone_v3_endpoint = CONF.identity.uri_v3
    self.password = CONF.fed_scenario.idp_password
    self.protocol_id = CONF.fed_scenario.protocol_id
    self.username = CONF.fed_scenario.idp_username

...

def setUp(self):
    super(TestSaml2EcpFederatedAuthentication, self).setUp()
    self._setup_settings()

    # Reset client's session to avoid getting garbage from another runs
    self.saml2_client.reset_session()

    # Setup identity provider, mapping and protocol
    self._setup_idp()
    self._setup_mapping()
    self._setup_protocol()

Finally, the tests perform the complete workflow of the feature, asserting its correctness in each step:

def _request_unscoped_token(self):
    resp = self.saml2_client.send_service_provider_request(
        self.keystone_v3_endpoint, self.idp_id, self.protocol_id)
    self.assertEqual(http_client.OK, resp.status_code)
    saml2_authn_request = etree.XML(resp.content)

    relay_state = self._str_from_xml(
        saml2_authn_request, self.ECP_RELAY_STATE)
    sp_consumer_url = self._str_from_xml(
        saml2_authn_request, self.ECP_SERVICE_PROVIDER_CONSUMER_URL)

    # Perform the authn request to the identity provider
    resp = self.saml2_client.send_identity_provider_authn_request(
        saml2_authn_request, self.idp_url, self.username, self.password)
    self.assertEqual(http_client.OK, resp.status_code)
    saml2_idp_authn_response = etree.XML(resp.content)

    idp_consumer_url = self._str_from_xml(
        saml2_idp_authn_response, self.ECP_IDP_CONSUMER_URL)

    # Assert that both saml2_authn_request and saml2_idp_authn_response
    # have the same consumer URL.
    self.assertEqual(sp_consumer_url, idp_consumer_url)

    ...


@testtools.skipUnless(CONF.identity_feature_enabled.federation,
                      "Federated Identity feature not enabled")
def test_request_unscoped_token(self):
    self._request_unscoped_token()

Notice that the test_request_unscoped_token test only executes if the the federation feature flag is enabled.

Note

For each patch submitted upstream, all of the tests from the keystone tempest plugin are executed in the gate-keystone-dsvm-functional-v3-only-* job.