Adding a script to generate README.rst from docs

The script readme.py will generate a README.rst file
from syntribos.docs.source. and add it to syntribos/. location.

Edited a few doc files to reflect the present status of the
project, also generated a README.rst from them.

Change-Id: Ida67b2486b8e1e6e6f694a00e41f00ffc825b3f7
This commit is contained in:
Rahul Nair 2016-08-22 13:58:10 -05:00
parent 96cb9bb438
commit c3be7d56c5
11 changed files with 304 additions and 174 deletions

View File

@ -61,7 +61,7 @@ however it supports installation and execution on Windows. But it has
not been tested yet.
Installation
------------
============
Syntribos can be `installed with
pip <https://pypi.python.org/pypi/pip>`__ from the git repository.
@ -74,13 +74,12 @@ pip <https://pypi.python.org/pypi/pip>`__ from the git repository.
$ cd syntribos
$ pip install . --upgrade
Configuration
-------------
=============
This is the basic structure of a Syntribos configuration file.
All config files should have a section ```[Syntribos]``` and a
```[user]``` section, ```[logging]``` is optional.
All config files should have the section ``[syntribos]`` and a
``[user]`` section, the ``[logging]`` section is optional.
::
@ -105,18 +104,19 @@ All config files should have a section ```[Syntribos]``` and a
log_dir=<location_to_save_debug_logs>
Testing the Keystone API
Testing Keystone API
::
.
You can modify the file to add your user credentials, log, template
directory etc.
A sample config file is given in ``examples/configs/keystone.conf``.
Copy this file to a location of your choice (default file path for
configuration file is: ``~/.syntribos/syntribos.conf``) and update the
necessary fields like user credentials, log, template directory etc.
::
$ vi examples/configs/keystone.config
$ vi examples/configs/keystone.conf
::
[syntribos]
# As keystone is being tested in the example, enter your
@ -125,8 +125,6 @@ directory etc.
# Set payload and templates path
templates=<location_of_templates_dir/file>
payload_dir=<location_of_payload_dir>
# Optional, api version if required.
#version=v2
[user]
#
@ -135,12 +133,15 @@ directory etc.
endpoint=http://localhost:5000
username=<yourusername>
password=<yourpassword>
# Optional, only needed if Keystone V3 API is used
# Optional, unless Keystone V3 API is used
#user_id=<youruserid>
# Optional, api version if required.
#version=v2.0
#[alt_user]
#
# Used for cross auth tests (-t AUTH_WITH_SOMEONE_ELSE_TOKEN)
# Optional, only used for cross auth tests (-t AUTH)
#
endpoint=http://localhost:5000
@ -160,7 +161,8 @@ directory etc.
http_request_compression=True
Another example, to test a keystone v3 API, use a configuration like
Another example, to test a keystone v3 API, use a configuration
similar to the one given below.
::
@ -182,86 +184,80 @@ Another example, to test a keystone v3 API, use a configuration like
To test any other project, just change the endpoint URI under
```[syntribos]``` to point to the API and also modify the user
credentials if needed. The endpoint URI in the ```[syntribos]```
``[syntribos]`` to point to the API and also modify the user
credentials if needed. The endpoint URI in the ``[syntribos]``
section is the one being tested by Syntribos and the endpoint URI in
```[user]``` section is just used to get an AUTH_TOKEN.
``[user]`` section is just used to get an AUTH_TOKEN.
Syntribos Commands
------------------
===================
Below are the set of commands that should be specified while
using Syntribos.
### run
- **run**
This command runs Syntribos with the given config options
::
$ syntribos --config-file keystone.conf -t SQL run
- **dry_run**
This command runs Syntribos with the given config options
This command prepares all the test cases that would be executed by
the ```run``` command based on the configuration options passed to
Syntribos, but simply prints their details to the screen instead
of actually running them.
::
::
$ syntribos --config-file keystone.config -t SQL run
$ syntribos --config-file keystone.conf -t SQL dry_run
### dry_run
This command prepares all the test cases that would be executed by
the ```run``` command based on the configuration options passed to
Syntribos, but simply prints their details to the screen instead
of actually running them.
::
$ syntribos --config-file keystone.config -t SQL dry_run
- **list_tests**
### list_tests
This command will list the names and description of all the tests
that can be executed by the ```run``` command.
::
$ syntribos --config-file keystone.conf list_tests
This command will list the names and description of all the tests
that can be executed by the ```run``` command.
::
$ syntribos --config-file keystone.config list_tests
All these commands will only work if the config file or directory is
specified.
All these commands will only work if a configuration file
is specified.
Running Syntribos
-----------------
=================
To run Syntribos against all the available tests, just specify command
```syntribos``` with the configuration file without specifying any
test type.
To run Syntribos against all the available tests, just specify the
command ``syntribos`` with the configuration file without specifying
any test type.
::
$ syntribos --config-file keystone.config run
####Fuzzy-matching test names
Fuzzy-matching test names
--------------------------
It is possible to limit Syntribos to run a specific test type using
the ```-t``` flag.
For example,
the ``-t`` flag..
::
$ syntribos --config-file keystone.config -t SQL run
This will match all tests that contain the string 'SQL' in their name,
This will match all tests that contain ``SQL`` in their name
like SQL_INJECTION_HEADERS, SQL_INJECTION_BODY etc.
Syntribos Logging
-----------------
Syntribos logging
=================
(**This section will be updated shortly**)
Syntribos takes advantage of the OpenCafe logging facility. Logs are
@ -355,13 +351,14 @@ above:
2016-05-19 16:11:52,083: INFO: root: ========================================================
Basic Syntribos Test Anatomy
----------------------------
============================
**Test Types**
The tests included at release time include LDAP injection, SQL
injection, integer overflow, buffer overflow, Cross Origin
Resource Sharing(CORS) wild card vulnerability and SSL.
injection, integer overflow, command injection, XML external entity,
reflected cross-site scripting, Cross Origin Resource Sharing (CORS)
wildcard and SSL.
In order to run a specific test, simply use the ``-t, --test-types``
option and provide `syntribos` with a keyword or keywords to match from
@ -371,26 +368,32 @@ For SQL injection tests, use:
::
$ syntribos --config-file keystone.config -t SQL run
$ syntribos --config-file keystone.conf -t SQL
For SQL injection tests against the template body only, use:
Another example, to run SQL injection tests against the template body only, use:
::
$ syntribos --config-file keystone.config -t SQL_INJECTION_BODY run
$ syntribos --config-file keystone.conf -t SQL_INJECTION_BODY
For all tests against HTTP headers only, use:
::
$ syntribos --config-file keystone.config -t HEADERS run
$ syntribos --config-file keystone.conf -t HEADERS
**Call External**
Syntribos template files can be supplemented with variable data, or data
retrieved from external sources. This is handled using 'extensions.'
Extensions are found in ``syntribos/syntribos/extensions/`` .
Extensions are found in ``syntribos/extensions/`` .
Calls to extensions are made in this form:
::
CALL_EXTERNAL|{extension dot path}:{function}:{arguments}
One example packaged with Syntribos enables the tester to obtain an auth
token from keystone/identity. The code is located in
@ -403,7 +406,7 @@ To use this extension, you can add the following to your template file:
X-Auth-Token: CALL_EXTERNAL|syntribos.extensions.identity.client:get_token_v3:["user"]|
The "user" string indicates the data from the configuration file we
added in ``configs/keystone.config``
added in ``examples/configs/keystone.conf``
Another example is found in ``random_data/client.py`` . This returns a
UUID when random but unique data is needed. This can be used in place of
@ -411,7 +414,7 @@ usernames when fuzzing a create user call.
::
"username": "CALL_EXTERNAL|syntribos.extensions.random_data.client:get_uuid:[]|",
"username": "CALL_EXTERNAL|syntribos.extensions.random_data.client:get_uuid:[]|"
The extension function can return one value or be used as a generator if
you want it to change for each test.
@ -425,12 +428,12 @@ follows:
::
"ACTION_FIELD:id": "1a16f348-c8d5-42ec-a474-b1cdf78cf40f",
"ACTION_FIELD:id": "1a16f348-c8d5-42ec-a474-b1cdf78cf40f"
The ID provided will remain static for every test.
Executing Unittests
-------------------
Executing unittests
===================
Navigate to the ``syntribos`` root directory
@ -438,7 +441,26 @@ Navigate to the ``syntribos`` root directory
$ python -m unittest discover syntribos/ -p ut_*.py
.. _Apache license: https://github.com/openstack/syntribos/blob/master/LICENSE
.. _Launchpad project: https://launchpad.net/syntribos
.. _Blueprints: https://blueprints.launchpad.net/syntribos
.. _Bugs: https://bugs.launchpad.net/syntribos
Contributing Guidelines
========================
1. Follow all the `OpenStack Style Guidelines <http://docs.openstack.org/developer/hacking/>`__
(e.g. PEP8, Py3 compatibility)
2. All new classes/functions should have appropriate docstrings in
`RST format <https://pythonhosted.org/an_example_pypi_project/sphinx.html>`__
3. All new code should have appropriate unittests (place them in the
``tests/unit`` folder)
Anyone wanting to contribute to OpenStack must follow
`the OpenStack development workflow <http://docs.openstack.org/infra/manual/developers.html#development-workflow>`__
All changes should be submitted through the code review process in Gerrit
described above. All pull requests on Github will be closed/ignored.
Bugs should be filed on the `Syntribos launchpad site <https://bugs.launchpad.net/syntribos>`__,
and not on Github. All Github issues will be closed/ignored.
Breaking changes, feature requests, and other non prioritized work should
first be submitted as a blueprint `here <https://blueprints.launchpad.net/syntribos>`__
for review.

61
doc/source/about.rst Normal file
View File

@ -0,0 +1,61 @@
Syntribos, An Automated API Security Testing Tool
=================================================
::
Syntribos
xxxxxxx
x xxxxxxxxxxxxx x
x xxxxxxxxxxx x
xxxxxxxxx
x xxxxxxx x
xxxxx
x xxx x
x
xxxxxxxxxxxxxxx xxxxxxxxxxxxxxx
xxxxxxxxxxxxx xxxxxxxxxxxxx
xxxxxxxxxxx xxxxxxxxxxx
xxxxxxxxx xxxxxxxxx
xxxxxx xxxxxx
xxx xxx
x x
x
=== Automated API Scanning ===
.. image:: https://img.shields.io/pypi/v/syntribos.svg
:target: https://pypi.python.org/pypi/syntribos/
:alt: Latest Version
.. image:: https://img.shields.io/pypi/dm/syntribos.svg
:target: https://pypi.python.org/pypi/syntribos/
:alt: Downloads
Syntribos is an automated API security testing tool that is maintained by
members of the `OpenStack Security Project <https://wiki.openstack.org/wiki/Security>`__.
Given a simple configuration file and an example HTTP request, Syntribos
can replace any API URL, URL parameter, HTTP header and request body
field with a given set of strings. This is similar to Burp Proxy's
Intruder sniper attack, but Syntribos iterates through each position
automatically. Syntribos aims to automatically detect common security
defects such as SQL injection, LDAP injection, buffer overflow, etc. In
addition, Syntribos can be used to help identifying new security defects
by fuzzing.
Syntribos has the capability to test any API, but is designed with
`OpenStack <http://http://www.openstack.org/>`__ applications in mind.
**Details**
* Free software: `Apache license`_
* `Launchpad project`_
* `Blueprints`_
* `Bugs`_
Supported Operating Systems
---------------------------
Syntribos has been developed primarily in Linux and Mac environments,
however it supports installation and execution on Windows. But it has
not been tested yet.

View File

@ -1,43 +1,41 @@
Syntribos Commands
------------------
===================
Below are the set of commands that should be specified while
using Syntribos.
### run
- **run**
This command runs Syntribos with the given config options
::
$ syntribos --config-file keystone.conf -t SQL run
- **dry_run**
This command runs Syntribos with the given config options
This command prepares all the test cases that would be executed by
the ```run``` command based on the configuration options passed to
Syntribos, but simply prints their details to the screen instead
of actually running them.
::
::
$ syntribos --config-file keystone.config -t SQL run
$ syntribos --config-file keystone.conf -t SQL dry_run
### dry_run
This command prepares all the test cases that would be executed by
the ```run``` command based on the configuration options passed to
Syntribos, but simply prints their details to the screen instead
of actually running them.
::
$ syntribos --config-file keystone.config -t SQL dry_run
- **list_tests**
### list_tests
This command will list the names and description of all the tests
that can be executed by the ```run``` command.
::
$ syntribos --config-file keystone.conf list_tests
This command will list the names and description of all the tests
that can be executed by the ```run``` command.
::
$ syntribos --config-file keystone.config list_tests
All these commands will only work if the config file or directory is
specified.
All these commands will only work if a configuration file
is specified.

View File

@ -2,8 +2,8 @@ Configuration
=============
This is the basic structure of a Syntribos configuration file.
All config files should have a section ```[Syntribos]``` and a
```[user]``` section, ```[logging]``` is optional.
All config files should have the section ``[syntribos]`` and a
``[user]`` section, the ``[logging]`` section is optional.
::
@ -28,18 +28,27 @@ All config files should have a section ```[Syntribos]``` and a
log_dir=<location_to_save_debug_logs>
Testing the Keystone API
To test any project, just update the endpoint URI under
``[syntribos]`` to point to the API and also modify the user
credentials if needed. The endpoint URI in the ``[syntribos]``
section is the one being tested by Syntribos and the endpoint URI in
``[user]`` section is just used to get an AUTH_TOKEN.
::
.
You can modify the file to add your user credentials, log, template
directory etc.
Testing Keystone API
--------------------
A sample config file is given in ``examples/configs/keystone.conf``.
Copy this file to a location of your choice (default file path for
configuration file is: ``~/.syntribos/syntribos.conf``) and update the
necessary fields like user credentials, log, template directory etc.
::
$ vi examples/configs/keystone.config
$ vi examples/configs/keystone.conf
::
[syntribos]
# As keystone is being tested in the example, enter your
@ -48,8 +57,6 @@ directory etc.
# Set payload and templates path
templates=<location_of_templates_dir/file>
payload_dir=<location_of_payload_dir>
# Optional, api version if required.
#version=v2
[user]
#
@ -60,10 +67,13 @@ directory etc.
password=<yourpassword>
# Optional, only needed if Keystone V3 API is used
#user_id=<youruserid>
# Optional, api version if required.
#version=v2.0
#[alt_user]
#
# Used for cross auth tests (-t AUTH_WITH_SOMEONE_ELSE_TOKEN)
# Optional, Used for cross auth tests (-t AUTH)
#
endpoint=http://localhost:5000
@ -81,32 +91,3 @@ directory etc.
# Optional, compresses http_request_content,
# if you don't want this, set this option to False.
http_request_compression=True
Another example, to test a keystone v3 API, use a configuration like
::
[syntribos]
endpoint=http://localhost:5000
templates=<location_of_template_dir/file>
payload_dir=<location_of_payloads data>
[user]
endpoint=http://localhost:5000
username=<username>
password=<password>
domain_name=default
domain_id=default
[logging]
log_dir=<location_to_store_log_files>
http_request_compression=True
To test any other project, just change the endpoint URI under
```[syntribos]``` to point to the API and also modify the user
credentials if needed. The endpoint URI in the ```[syntribos]```
section is the one being tested by Syntribos and the endpoint URI in
```[user]``` section is just used to get an AUTH_TOKEN.

View File

@ -1,5 +1,5 @@
Contributing Guidelines
=================================
========================
1. Follow all the `OpenStack Style Guidelines <http://docs.openstack.org/developer/hacking/>`__
(e.g. PEP8, Py3 compatibility)
@ -17,6 +17,6 @@ described above. All pull requests on Github will be closed/ignored.
Bugs should be filed on the `Syntribos launchpad site <https://bugs.launchpad.net/syntribos>`__,
and not on Github. All Github issues will be closed/ignored.
Breaking changes, feature requests, and other unprioritized work should first be
submitted as a blueprint `here <https://blueprints.launchpad.net/syntribos>`__
Breaking changes, feature requests, and other non prioritized work should
first be submitted as a blueprint `here <https://blueprints.launchpad.net/syntribos>`__
for review.

View File

@ -21,8 +21,10 @@ Index
.. toctree::
:maxdepth: 1
about
installation
configuration
commands
running
logging
test.anatomy

View File

@ -1,27 +1,24 @@
Running Syntribos
=================
To run Syntribos against all the available tests, just specify command
```syntribos``` with the configuration file without specifying any
test type.
To run Syntribos against all the available tests, just specify the
command ``syntribos`` with the configuration file without specifying
any test type.
::
$ syntribos --config-file keystone.config run
####Fuzzy-matching test names
Fuzzy-matching test names
--------------------------
It is possible to limit Syntribos to run a specific test type using
the ```-t``` flag..
For example,
the ``-t`` flag..
::
$ syntribos --config-file keystone.config -t SQL run
This will match all tests that contain the string 'SQL' in their name,
This will match all tests that contain ``SQL`` in their name
like SQL_INJECTION_HEADERS, SQL_INJECTION_BODY etc.

View File

@ -1,36 +1,39 @@
Basic Syntribos Test Anatomy
============================
**Test Types**
Test Types
----------
The tests included at release time include LDAP injection, SQL
injection, integer overflow, command injection, XML external entity,
reflected cross-site scripting, Cross Origin Resource Sharing (CORS)
wildcard and Ssl.
wildcard and SSL.
In order to run a specific test, simply use the ``-t, --test-types``
option and provide `syntribos` with a keyword or keywords to match from
the test files located in ``syntribos/tests/fuzz/``.
the test files located in ``syntribos/tests/``.
For SQL injection tests, use:
::
$ syntribos --config-file keystone.config -t SQL
$ syntribos --config-file keystone.conf -t SQL
Another example, to run SQL injection tests against the template body only, use:
::
$ syntribos --config-file keystone.config -t SQL_INJECTION_BODY
$ syntribos --config-file keystone.conf -t SQL_INJECTION_BODY
For all tests against HTTP headers only, use:
::
$ syntribos --config-file keystone.config -t HEADERS
$ syntribos --config-file keystone.conf -t HEADERS
**Call External**
Call External
-------------
Syntribos template files can be supplemented with variable data, or data
retrieved from external sources. This is handled using 'extensions.'
@ -54,7 +57,7 @@ To use this extension, you can add the following to your template file:
X-Auth-Token: CALL_EXTERNAL|syntribos.extensions.identity.client:get_token_v3:["user"]|
The "user" string indicates the data from the configuration file we
added in ``examples/configs/keystone.config``
added in ``examples/configs/keystone.conf``
Another example is found in ``random_data/client.py`` . This returns a
UUID when random but unique data is needed. This can be used in place of
@ -67,7 +70,9 @@ usernames when fuzzing a create user call.
The extension function can return one value or be used as a generator if
you want it to change for each test.
**Action Field**
Action Field
------------
While Syntribos is designed to test all fields in a request, it can also
ignore specific fields through the use of Action Fields. If you want to

View File

@ -1,8 +1,22 @@
Executing unittests
===================
Navigate to the ``syntribos`` root directory
To execute unittests automatically, navigate to the ``syntribos`` root
directory and install the test requirements.
::
$ pip install -r test-requirements.txt
Now, run
::
$ python -m unittest discover syntribos/ -p ut_*.py
$ python -m unittest discover tests/unit -p "test_*.py"
Also, if you have configured tox you could also do
::
$ tox -e py27
This will run all the unittests and give you a result output
containing the status and coverage details of each test.

View File

@ -9,8 +9,6 @@ endpoint=http://localhost:5000
# Set payload and templates path
templates=<location_of_templates_dir/file>
payload_dir=<location_of_payload_dir>
# Optional, api version if required.
#version=v2
[user]
#
@ -19,14 +17,16 @@ payload_dir=<location_of_payload_dir>
endpoint=http://localhost:5000
username=<yourusername>
password=<yourpassword>
# Optional, only if keystone V3 API is used
# Optional, unless keystone V3 API is used
#user_id=<youruserid>
#domain_name=<domain_name>
#domain_id=<domain_id>
# Optional, api version if required.
#version=v2.0
#[alt_user]
#
# Used for cross auth tests (-t AUTH_WITH_SOMEONE_ELSE_TOKEN)
# Optional, only used for cross auth tests (-t AUTH)
#
#endpoint=http://localhost:5000
#username=<alt_username>

50
scripts/readme.py Normal file
View File

@ -0,0 +1,50 @@
# Copyright 2016 Intel
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
def find_docs():
"""Yields files as per the whitelist."""
loc = "../doc/source/{}.rst"
whitelist = [
"about", "installation",
"configuration", "commands",
"running", "logging",
"test.anatomy", "unittests",
"contributing"]
for fname in whitelist:
fpath = loc.format(fname)
if os.path.isfile(fpath):
yield fpath
def concat_docs():
"""Concatinates files yielded by the generator `find_docs`."""
outfile = "../README.rst"
if not os.path.isfile(outfile):
print("../README.rst not found, exiting!")
exit(1)
with open(outfile, 'w') as readme_handle:
for doc in find_docs():
with open(doc, 'r') as doc_handle:
for line in doc_handle:
readme_handle.write(line)
readme_handle.write("\n")
if __name__ == '__main__':
"""Generate README.rst from docs."""
concat_docs()
print("\nREADME.rst created!\n")