Fixes:
1) Crashes in runner and file_utils
2) Binary strings being read in as payloads
Updates:
1) Clarified error messages in parser
2) Confusing variable names in test cases vs issues
Adds:
1) A `syntribos root` CLI sub command to display the current syntribos root dir
Change-Id: I22edf7a1f3d39724522aee88d08b00d299b67248
This change:
1) rewrites the runner to spawn a thread pool for each template
and assigns a worker for each test case
2) makes the output colorized by default
3) makes minor changes to the output
Change-Id: I49906f5daaa339ca9429913680203c762a0ad9fe
Syntribos now allows the user to specify variables in their request
templates by reading from a meta.json file. This is part 1 of 3 of
the full effort, dealing primarily with the template parser itself.
Change-Id: Id41d331f595cd3bc32f085ef49cb5d1b16779a5c
Fixes a bug in BTC that causes syntribos to send the same init_req over
and over, meaning that only the first template specified actually gets
executed.
Change-Id: I9df3de8ae8ac305d67428afc9c029d2145788b53
Closes-Bug: #1635747
Wrote empty debug test as a replacement for dry_run. dry_run no longer executes
any test code, or send any requests. Rather, it serves only to check that the
template file and all external calls inside the template are parsed and run
correctly.
Change-Id: I1cd817476c9ca4bd27b013dcdbd0cbce334847c4
1) results are now printed at the end of the run or on interrupt, not after
every template
2) made test_id's globally unique
3) print number of errors next to number of failures
4) number of failures printed after next to each progress bar and reported
in total at end of run now match up
Change-Id: I28617c64218ab632c1a35083ef9351874054a26d
Adding a simple counter based id to tests to track them in debug
log and results log. Also, this patch improves the result output
like adding a progress bar and uniform test names..
Change-Id: Ib83181b25a0c18c7993f491cde98d73555b01404
- This removes FuzzRequest as a type of RequestObject. All requests are
now RequestObjects, and are parsed by the
syntribos.clients.http.parser
- Fuzzing a request is now done via
syntribos.tests.fuzz.datagen.fuzz_request()
- Moved _remove_attr_names and _remove_braces to the RequestObject
- Added unittests for fuzz datagen, http client models
Change-Id: Ib589c34ad80da58daab875d7383210d22d82d764
Adding metadata, stats on http requests and response. Also, adding
checks and signals used by a test to the debug log.
Change-Id: I3b85d50804f1071383dfa4b9a07235926018338c
This updates the BTC/BFTC class methods based on the changes we agreed
on in our 7/13 meeting.
Details here: https://etherpad.openstack.org/p/syntribos-planning
Change-Id: I05e426ee1832385ec42d64ba930caea9ddd5374f
Currently, no signals are printed with the output, so now, the slugs of signals
associated with each issue are now printed in the results output. We may
revisit this to add strengths and tags should they be needed in the future.
Change-Id: Ib58127a9d6bb296b9731668e957c15e5c2ef71f4
There are differences between requestslib's PreparedRequest and
Syntribos' RequestObject, which can cause problems when they are used
inconsistently.
This is a first pass at a fix, editing BaseTestCase's send_init_request
to add the body to the test.init_req object.
Change-Id: I6f71fb52bca2f1cccdbdbce5f081431a0750db66
Partial-Bug: #1604187
- Merges arguments.py and and existing config.py files into one file
- Removes dependencies on cafe.* config file / CLI parsing
- Adds "register_opts" to BTC to allow Tests to specify config options
- Moves us completely from cclogging to Python logging
Change-Id: I0d4a84563d54307c94c0064be429919f9d91d67b
- Removed unnecessary code from BTC
- Moved some components from BFTC to BTC
- Start using signals for all tests
- Renamed several variables in BTC for clarity
- Cleaned up BaseAuthTestCase somewhat
Change-Id: I3efc44b33aa4416e1f9853910485a8c5703a9057
CORS header `Access-Control-Allow-Origin` should be a whitelist
of URIs. This testcase checks if a wildcard is set instead.
Partially implements: blueprint test-headers
Change-Id: I0f16a3f0843b6c828d40994dd1be4f4c86455371
Extended the extend_class() method from BaseTestCase in BaseFuzzTestCase, as
only BaseFuzzTestCase should have fuzz_string and param_path as parameters to
the method.
Change-Id: If39629caed9d9659bb4c7c39eb8199b25610b5b1
We are currently using block quotes for license comments, which means
that Sphinx pulls this into every module, which is bad. Hash-prefixed
comments for licenses seem to be the standard for OpenStack as well.
Change-Id: Iabc60228c1201a960f672e1b0c7ab3c5312c9d8e
Implements: blueprint docstring-add-to-framework
This PR adds docstrings to a number of important components of
Syntribos, and adds this documentation to our Sphinx doc structure. It
also removes copyrights from __init__.py files with no other content,
in line with OpenStack style guidelines.
Set 2: Fixed PEP8 failure.
Change-Id: Ic57b31f451ec3ecf7f5b308da4544f808c9c9a5d
Implements: blueprint docstring-add-to-framework
Changes to the framework are only found in runner.py and base.py.
Tests are now no longer written as lists of assertions, which before meant that
it was impossible to access any variables used in the test itself after the
test was over. Instead, the register_issue method now adds the issue to
cls.failures, and will throw an AssertionError if there are any failures
reported to the test runner. This will allow us to do things like put the
failure string into the Issue description.
Change-Id: Ic3ca2ec48a6e1d99d56e605a2e0d0dc89158bc72
Implements: bp/framework-issue-creation
Registering an issue and testing that issue should be universal to
all test cases, not just fuzz test cases.
Change-Id: Idd09819475f68d99a8d0d491daac4e03e286a321
Andreas Jaeger