Often exceptions caused by the Request Library are either:
* Impacting, in which case they are consistent
* Non-Impacting, in which case they are sporadic and cause false
failures
In tests that do API testing using the HTTP plugin, there is a use
case for retrying Request Library exceptions as some of these
failures can be caused by issues unrelated to tests and the system
under test. This PR provides a mechanism to configure how many of
these errors a test runner is willing to ignore.
Change-Id: I2d15095ad435b151df56f0a23f068f0fae662c37
* Cleans up some unittests/makes Brew more testable
* Cleans up __repr__ output on some classes
* Removes unneeded imports
Change-Id: Ib33bdcc21c2f554f2312f9ed92de8e500e8356f2
- unit test captures stdout of print_results
- mock out init to get past all the argparse requirements
Change-Id: Ifec5dff1a96288289b73ea270964165cc67b6b1a
* Generates suites from test-containing classes,
unittest.TestSuite-derived classes (Fixtures), and
opencafe DatasetList classes as defined in a flat text file
called a 'brew' (runfile).
* Adds brew package in cafe.drivers.unittest
* Adds arguments, parser and runner modules in brew
package.
* Adds examples directory in repository root directory
* Adds example brewfile in examples directory
* Adds new 'cafe-brew' entry point to setup.py
* Adds unittests for brew package
Change-Id: I379c5b2c4bdf17bc49ce055ef4600b38a72a1140
* Added README explaining the paramiko transition.
* NOTE: The sshv2 plugin will eventually transition
to utilizing paramkio>2 once plugins are external
to opencafe.
Change-Id: Iab3249ce31503367ed862114b71c460d5eb1b208
This PR prevents users of paramiko<2.0.0 from running into an
ImportError while trying to import the cryptography library. This comes
with no functional changes and should remediate any ImportError's users
of paramiko<2.0.0 would be getting.
Change-Id: I6c45bdbf288c4cef5d5e5fb055276d9d3978e8b8
Paramiko 2.0 changed the paramiko dependancy to require cryptography
which can cause issues with OpenCafe installations of the SSH plugin.
Change-Id: Ied6793e0a42e1d7de6a15db19e9883399d3301d6
This PR updates the SSHV2 client to have the fix the issue
outlined in
https://github.com/paramiko/paramiko/issues/735#issuecomment-216083986
where a new library added by paramiko 2.0.0 (cryptography) causes a
threadlocking issue where the SSHV2 client is unable to function when
connecting on import or through the unittest runner.
This PR should not affect any functionality users may have but may
affect users who have not yet installed the cryptography library. To
install the correct cryptography library it is recommended to upgrade
paramiko using `pip install --upgrade paramiko`
Change-Id: Iabcce8a66b5958b117c45977708cab777dc90a39
Modified json_report.py to include new metrics
Modified xml_report.py to include new metrics
Modified SummarizeResults to handle new metrics
Added metatests for compile_results method
Change-Id: Iecd23d519e7cbaf4887f64b3d362942d39f50006
Test setUp methods that do not have a 'super()' call will
fail and cause the entire suite to fail despite tests
passing. Runner failures should, whenever possible, not
interfere with test results.
This is to add a try/except around getting the '_duration'
value for tests. If a test uses a setUp() method without super
it will recieve 'NaN' as a 'duration' and the user will
recieve a warning in console output but will not recieve
failures.
This is considered a safety mechanism intended to provide
test writers time to update their tests using 'setUp()' to
correctly call super().
Additionally added some newlines around the warning so that it is
very clear that there is an issue.
Change-Id: I32d2352ff7081b84383835bcc8812e37735c68d2
* Adds the pathos_multiprocess plugin which serves only to
install the multiprocess package. If installed, it will
be used by all unittest runners instead of the default
multiprocessing library.
Change-Id: Id18ee56d3fb1466f038aaa8c6fd9e2d890847f9e
This change uses 'not __eq__' to implement __ne__ preventing
execeptions from being raised when obj passed in does not have
__dict__. Example are instances of tuple, string, int, float,
bool...
Change-Id: Idf82eeeb865a121d5c81c5654e7c21e71e555238
If a DataSetList is passed into a test that does not generate or
contain data the current behavior is that the test suites passes.
This proposal is to change that behavior. If a DSL cannot generate
data the test suite as a whole should still run but it should
fail. Otherwise teams using tooling to track and manage test results
will have false positives with mysteriously missing tests.
Added Metatests to check basic functionality.
Added Engine config section for unittest driver.
Change-Id: I3430e9573c3e99b80f25e7577709144d49c8eae2