Commit Graph

406 Commits

Author SHA1 Message Date
Le'Sage, Oded (ol7435) 870b20b34f implement support for heat's get_file function
Enhances Shaker's heat client to support templates which use Heat's
get_file function to pass locally available files to the heat engine via
the API request

Change-Id: I51d3bf3ebcfe09833bbe15f96d36fc8ee5926bf5
2021-04-27 16:21:30 -05:00
Le'Sage, Oded (ol7435) a2adff395f fix shaker-spot execution
A previous commit introduced a feature to specify the directory the
agent uses for shell class execution via a cfg entry, agent_dir.

However this addition introduced an error when using shaker-spot:
  oslo_config.cfg.NoSuchOptError: no such option agent_dir in group
  [DEFAULT]

This commit fixes shaker-spot while still supporting the agent_dir
feature

Change-Id: I7cfb87ba406e177121e7914225857937a19502e4
2021-03-04 15:31:18 -06:00
Dinesh Yadav 159bd35d2f Enhance Shaker to support the shell class execution after reboot
agent instance.

Issue: Some of our complex test requires to reboot compute nodes. In
that scenario, if agent instances are on those computes then those
instances will also be rebooted and running shaker test(s) will be
stopped.

At this time this commit is specifically for shaker's shell class
execution as it writes execution script in /tmp directory and we
know that if system/instance is being rebooted then whatever /tmp
directory contains that will be wiped out. So, if user has a choice
to copy that execution script in some specific directory which is
not being impacted (i.e /mnt, /opt etc) after reboot then through
custom test scripts above condition can be handled

Change-Id: I41f824561e36329d07afc9e798eab44b72aeb01a
2021-01-25 15:19:01 +00:00
Ilya Shakhat d259b94322 Resolve job failures
* Pin versions of dependencies for py<3.6
* Use Ubuntu Bionic for intergration jobs

Change-Id: Idb5095e93df9e8261b670d1cc639b2bab4939980
2021-01-25 17:39:22 +04:00
Oded Le'Sage 68266af6c7 update master/slave terms
This commit replaces all master/slave references with primary/minion in
an effort to be more socially aware and inclusive

Change-Id: I3d5db71de6506d458707403e8f8c5e0493b78f4f
2020-07-15 16:40:32 -05:00
DeJaeger, Darren (dd118r) e4fff5fcc5 Add psuedo lazy pirate stragey to agent, with socket timeouts
This adds a pseudo lazy pirate strategy to the agent, with
configurable socket timeouts and retry counts.

Change-Id: I57bd09c33d071f1cc975e00b9e2deeb715f19bd6
2020-06-19 20:25:20 +00:00
Ilya Shakhat 9d2ccc4e59 Resolve code style issue reported by a new version of flake
Change-Id: If4c4a460a95851b3ed09a68a240704a70e7d12a2
2020-06-04 11:29:03 +04:00
Ilya Shakhat 9168689007 Resolve py27 and pep8 job failures
1. Cap version of stestr for py27 (the latest version is py3 only).

2. Flake8 is runtime dependency of diskimage-builder, currently they use
version >= 3.6.0, so Shaker must conform to this version too.

Failures found by the newer version of flake8:
 * W605 invalid escape sequence -- issues are fixed since they
   correspond to syntax warnings from py36.
 * E305 expected 2 blank lines after class or function definition --
   issues are fixed (older version didn't restrict spaces between
   functions and `if __name__ == "__main__":` block).
 * W504 line break after binary operator -- the rule is ignored since it
   contradicts the code style of the whole project.

Change-Id: I7fa53cfb6b654f5d93df30441a6eb2d47714d479
2020-04-05 15:49:53 +00:00
Ilya Shakhat cd540090dd Cap requirements for Python version below 3.6
OpenStack drops support for py27 starting with Ussuri release.
All new libraries are tested against py36 and py37 only,
older versions support py27 and py35. This patch caps dependency's
versions for py<3.6 and keeps them open for py>=3.6.

Change-Id: I1edc4893d2699529385fabd2d6ac25ccef18b2d9
2020-03-09 19:22:17 +04:00
Ilya Shakhat 737b019a3a Update documentation
Change-Id: Id2a831e8a245778bdea508fd818b47c8ecd0f8e7
2020-03-08 21:53:04 +04:00
Ilya Shakhat 55833e5b2b Add a job to mirror the code from OpenDev to Github
Change-Id: Ic58256dbb9e257781b02f0a19c06b73eddde787a
2020-03-08 02:03:52 +04:00
Oded Le'Sage d4c226b9d3 cleanup per cfg when heat stack creation fails
If a heat stack fails creation (over quota, no more ips, resource
issue, etc) it does not get deleted even when cleanup_on_exit = true

Heat stacks should be deleted per the cleanup_on_exit cfg setting

Failure to cleanup is occuring because the stack_id is curently
empty when the exception is raised.

Change-Id: I1be7b82a4178d20ab267ccc1ec96051a6e8c7120
2020-02-21 10:24:30 -06:00
Oded Le'Sage 539e376978 Update hung process/thread approach
The previous commit attempted to prevent a hung process/thread by using
a "tcp ping" to determine if the server_endpoint was reachable before
deploying the heat stack and starting the heartbeat thread.

While this approach works well in theory we're finding that in practice
in a live K8 environment we're seeing a lot of random errors:
[Errno 111] Connection refused: ConnectionRefusedError

There could be numerous reasons for these random connection errors but
ZMQ has retry logic which should overcome these problems.

This commit updates the sockets used in ZMQ to add a timeout (a padded
value of agent_loss_timeout). While this does not prevent the
creation of a heat stack and heartbeat thread that might never respond,
it does solve the initial problem of having stuck process/threads and
getting a clean exit

Change-Id: I8193c72120b459c2a18d780d9f8799e8df592e20
2020-02-11 12:13:58 -06:00
Oded Le'Sage 639760a0fa Prevent hung process/threads when server_endpoint is not reachable
This commit is to address an issue that can cause Shaker to get
stuck indefinately waiting for a zeroMQ message that will never
arrive.

To reproduce set server_endpoint to 10.0.0.1:8080 (assuming
nothing is listening and/or this address is unreachable).
Shaker will go through launching the heartbeat thread, and
creating the heat stack, but then will be stuck at:
Waiting for quorum of agents:
If debug is turned on, there are no heartbeats coming through.

In order to fix this situation this commit sends 3 "tcp pings"
to the server_endpoint in order to test it's response before
the heartbeat thread and heat stack are created.

Change-Id: Ifcfacfb383e7553b53c5b2a20ad39ceccb833c8f
2020-02-05 09:39:14 -06:00
Sai Sindhur Malleni b948585b71 Upgrade pip and setuptools in CentOS images
Change-Id: I51b638357530ac7551a2a34add39c14204c28891
2020-01-07 12:49:37 -05:00
Sai Sindhur Malleni 569d166515 Fix CentOS7 shaker image build failures
Ignore installed packages and reinstall them.

Change-Id: Ia5b78a3333adb0873be81ede40ea1bcea3e502f0
2020-01-02 13:25:09 -05:00
Ilya Shakhat 01dbc4ffd6 Sync version of hacking library with diskimage-builder
Hacking is runtime dependency of diskimage-builder and its version
is higher than the one used by Shaker. This patch aligns version
that Shaker uses for code style check.

Change-Id: I44ade3ea99ffc165b91e44e5e62a8682912f3de2
2019-10-20 17:41:22 +04:00
Sean M. Collins 766c53b372 Make environment to create_stack have a default value
dc5d0ce added this new parameter to create_stack but did not thoroughly
handle all callers to create_stack to add this parameter, which causes
`shaker-image-builder` to crash with a stacktrace

https://bugs.launchpad.net/shaker/+bug/1848368

Change-Id: I91424852a17558e6a593d138ea4926c30359af2d
2019-10-17 07:20:36 +00:00
Ilya Shakhat 1dcddf8358 Fix failures in pep8 job caused by flake8>=2.6.0
Diskimage-builder depends on hacking library, which in turn depends
on flake8<2.7.0,>=2.6.0, so shaker should follow the current version.

Change-Id: I5d4eccc0d5f5bfc7e21edaad060634b3e53b60e8
2019-10-17 00:34:01 +04:00
Ilya Shakhat 39942a642c Update project links and package metadata
Change-Id: I7134ade3e8575bee474ce579c0ebd9959d48a2d8
2019-09-09 11:00:56 +02:00
Oded Le'Sage 242992bf9a Update cleanup cfg parameter to a more appropriate name
The current cfg cleanup_on_error is a bit of a misnomer, since cleanup
is not always dependent on there being an error.

This commit updates the cleanup cfg parameter to better reflect when
cleanup actually happens - cleanup_on_exit

Change-Id: I984e4b660705be8afa50a1e8605102832fd802dd
2019-08-20 10:30:26 -05:00
Zuul 873772fa7f Merge "Add support for heat stack related configurations" 2019-08-16 15:25:18 +00:00
Oded Le'Sage 8385ae1b97 Add support for heat stack related configurations
This commit enhances Shaker to allow:
  - specifying a custom stack name, instead of always using a uniquely
    generated one. This helps with tracking test stacks in CICD and/or
    allowing users to be more descriptive when looking at running heat
    stacks
  - reusing an existing test stack, rather than re-stacking. This
    is to allow the following scenario: Execute scenario A (setting
    cleanup_on_error = false) to establish baseline data. Perform some
    update in Openstack (config change, cert rotation, etc). Reuse the
    existing VMs from the baseline test to re-execute scenario A to
    help ensure the update had no impact.

Change-Id: Ifbdd332a44ca54f06cc81d9018ea5cea26c32416
2019-08-16 09:13:56 -05:00
Ilya Shakhat 5d8ad2fa6e Remove redundant lines in zuul config
openstack-tox-docs is part of docs-on-readthedocs template, so remove
it from check/gate - it's really redundant.

Change-Id: I1c20f5d6a8ac06debbc375be4c5319e84257e1ae
2019-05-29 14:41:05 +00:00
Ilya Shakhat b48c6d731a Pin version of Sphinx
Change-Id: Iaeb13731d9dd50845357557ee58c08d27036b877
2019-05-29 14:36:49 +02:00
OpenDev Sysadmins 66d1bc035d OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:25:34 +00:00
Ian Wienand ecd88a3967 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I37f57d6c8b5c86ea87247dc0220c6030993633ec
2019-03-25 11:38:38 +00:00
Ilya Shakhat 093931ff2d Replace py35 job with py36
Change-Id: Ie7fe417558b4505e7ab8f10c4214c4efb07ca5dc
2019-03-25 09:54:08 +01:00
Ilya Shakhat 3845b7b403 Run tests on Python 3.7
Change-Id: I89aa5dab72f2b0abb9dfa82694429c840de989e6
2019-02-19 20:30:28 +00:00
Zuul db5f9ca053 Merge "Adjust new config options to use dash instead of underscore" 2018-12-10 21:20:06 +00:00
Sphicas, Phil (ps3910) 3d1d37b96a Add support for best-effort accommodations
Adds a new accommodation instruction 'best_effort' that allows the
deployment to proceed even if the number of available compute nodes
is less than what was requested.

For example, this will create a single pair of agents on separate
compute nodes if possible, but will fall back to creating them on
the same compute node if only one is available:

accommodation: pair, single_room, best_effort, compute_nodes: 2

Change-Id: I1986f550ac0dc934bfa5db84b40a6419d6608c7e
Closes-Bug: 1807630
2018-12-10 12:15:03 -08:00
Ilya Shakhat 6bddae1810 Adjust new config options to use dash instead of underscore
There are 3 recently added config options that have underscores instead
of dashes:
 * custom_user_opts (added in I96b6e578eb59813e5e0c8a2fe7a14c5ecc369be7)
 * scenario_availability_zone and scenario_compute_nodes
   (both added in I5459150a0eac9bed6a6a62a126bd9ec0648941fe)
The rest of options are declared with words separated by dashes. This patch
makes name style consistent across the project. The change affects CLI only.

Change-Id: I9d799827a36682fbe7ae73b22bcbaaf2f05afd6e
2018-12-10 10:22:24 +01:00
Sphicas, Phil (ps3910) 249ad453a4 Allow granular host selection
This commit is to allow the use of the following syntax in the
scenario_availability_zone parameter:

scenario_availability_zone = availability_zone:compute_host

Change-Id: I5b93d1083ce019e1d61199ad6d7b8b478a5678b9
Closes-Bug: 1807278
2018-12-06 16:28:46 -08:00
huang.zhiping a07b246d02 Change openstack-dev to openstack-discuss
Mailinglists have been updated. Openstack-discuss replaces openstack-dev.

Change-Id: I200e38ce26ed2861c30995170bccab52145b16df
2018-12-04 11:09:19 +00:00
Zuul ad1aa16355 Merge "Add Zuul job to build Docker image" 2018-11-09 09:28:31 +00:00
Ilya Shakhat 4b041b2b38 Add Zuul job to build Docker image
The job runs in gate pipeline. It creates a new image
and pushes it into performa/shaker repo on Dockerhub.

Change-Id: I85d26a03c812657138b2ceb2a0496ceb7d015577
2018-11-09 09:31:01 +01:00
Oded Le'Sage af4af421ae Make {{ unique }} available in env files
Issue: For certain tests we need to create long strings that use the
stack's name. Currently we have no way to inject the stack name into the
env file to build these necessary strings as parameters to the
templates.

This commit allows users to reference the dynamic stack name in the env
file the same way it's referenced in the Heat template by using {{
unique }}

Change-Id: I817189ac6353e34fe7cad84bbaf484c30b6b97b2
2018-11-08 16:30:41 -06:00
Oded Le'Sage 5a25825b73 Enhance Iperf executor to accept "unmapped" args
Issue: The current iperf executor is setup nicely to handle the
majority, let's say 90%, of the test cases a user might want to run.
However, it still leaves a bit to be desired for the remaining 10%. For
example we use a custom qcow2 image for our tests  and the verison of
iperf in this image (and the latest iperf which shaker doesn't have) you
can specify --sctp as an argument to run this type of traffic. This is
not possible with the current executor, but most users may not want or
care about this ability.

This commit proposes to extend the current executor to accept any
argument as part of an "advanced" mode, rather than mapping every
argument that may or may not used or exists only in certain versions.
This allows different version of iperf (e.g. --sctp capable) of
executing a supported argument without having to directly add support
for this new argument into the executor

I know that any argument could be passed to iperf using the script
executor, but this has the downside of not providing the nice graphs.
I believe this commit is a good compromise between general
useability (pre-mapping arguments) and granular control (using any
agrument that is not mapped). This will also be useful for other
arguments that may not be widely used, or are newly introduced, such
as --zerocopy or --omit along with other sctp related argument like
--xbind or --nstreams.

This would be for advanced users familiar with iperf and would not
impact current tests or general use. The only "safety net" provided is
not allowing mutually exclusive arguments with time to be specified with
the time argument. Since time is currently added by default, even if we
were to create as mapping for say --blockcount(-k), we would have to
handle removing the time argument for a correct run. Arguments that
don't mix will lead to an iperf error, but this error is correctly
logged and handled. Overriding time if a mutually exclusive argument is
used seems like a fair way to handle the situation while allowing other
possible argument combinations.

Change-Id: I072278d0ab3912aac375605dc36987a7f8c20461
2018-11-07 16:10:57 -06:00
Zuul fa8efac93b Merge "Enhance Shaker to support custom user options" 2018-11-07 10:10:42 +00:00
Oded Le'Sage b592dd9325 Enhance Shaker to support custom user options
Issue: Some of our more complex tests require injecting "custom" data
into the heat stacks in order to run correctly. For example in order to
run certain Contrail based tests we need to set "contrail-asn:(some
asn number)". The asn number varies between cloud deployments, so we
dynamically set this field in a Heat environment file. However this is
a very specific field that other Shaker users might not need or
understand it, so this commit allows for a more generic approach.

This commit aims to let the user specify "custom" user defined options
similar to the matrix config parameter. Since this opts is not
directly referenced anywhere in code it's main use is to inject data
into heat environment files. The YAML format allows flexibility for any
object the user might want to add during their scenario
run or it can be left blank without any issues.

Change-Id: I96b6e578eb59813e5e0c8a2fe7a14c5ecc369be7
2018-11-06 16:53:26 -06:00
Oded Le'Sage 5f84721040 Fix bug with interactive report and SLA failures
This commit is to fix a bug where a test will fail due to the SLA not
being met but the interactive html report shows 'success'. This fixes
interactive html report to show 'sla fail'.

Change-Id: Ief30bf5125013805e3aef75f74a892e71c41ea6e
Closes-Bug: 1801972
2018-11-06 10:41:56 -06:00
Oded Le'Sage abe9fbd877 Enhance Shaker to provide the ability to define support templates
Issue: At AT&T we have large complex test stacks that make putting
everything into a single heat template and environment file very
cumbersome. Large monolithic templates make it harder to debug failures,
maintain, extend, and organize these tests. In order to solve this issue
we have enhanced Shaker to support specifying support templates with
environment files.

This commit enhances Shaker to add the ability to define
support_templates with env_files in test definitions.

Support templates spin up "support type" resources before the actual
test stack is spun up. This could range from networks, to volumes, to
anything Heat can create. The support resources do not have any reliance
on resources created in the test stack, they set up a "foundation" for
the test stack. The test stack can then reference these resources by
name. (e.g. assume they exist by the time the test stack is spun up)

While the example provided with this commit is simple, and the support
networks that get created are not directly used in the test, it
shows the basic principles of how support templates work.

As a real-world example and to give an idea of the complexity this
enhancement is trying to solve, we have a test definition that looks
like this:

  support_templates:
  -
    Base:
      template: templates/module_1_base.yaml
      env_file: env/module_1_base.env
  -
   SI_L2:
      template: templates/module_2_si_l2.yaml
      env_file: env/module_2_si_l2.env
  -
   SI_L3:
      template: templates/module_3_si_l3.yaml
      env_file: env/module_3_si_l3.env

  template: templates/module_4_master_servant.yaml
  env_file: env/module_4_master_servant.env

The first support stack (module_1) sets up some "base" network
resources. This stack provides some network resources used by the SI_L2
and SI_L3 support stacks.

SI_L2 is a support stack with 2 VMs that do Contrail service chaining
on an L2 network.

SI_L3 is a support stack with 2 VMs that do Contrail service chaining
on an L3 network.

Then the test stack (module 4) gets spun up on N amount of computes and
runs traffic across the SI_L2 and SI_L3 service chained networks.

After the test run all stacks are cleaned up

Using the concept of support stacks allows us to beter organize and
maintain our complex tests and allows for faster debugging due to the
"layered" nature of the setup.

Support templates also allow us to spin up more Shaker test threads
that use the same support templates simultaneously to better simulate
real-world network traffic. It also reduces the set up time of certain
tests we have since the support stacks already exist.

This enhancement does not alter existing Shaker functionality and
is fully backwards compatible.

Change-Id: Ife51bc55874c6ec4faac221bab8f9f0eea175fdc
2018-11-05 11:12:14 -06:00
Ilya Shakhat f5c3a09535 Do not list test scenarios in CLI help
Scenarios under `test/` subfolder are used for integration testing
or as samples. Skip them in CLI help and keep only production-ready.

Change-Id: I69a428480055c825aa3b590f02b4c09b346a6012
2018-11-05 16:05:32 +01:00
Zuul 296cfb1344 Merge "Enhance Shaker to support basic heat environment files" 2018-10-30 09:51:57 +00:00
Oded Le'Sage dc5d0ce4ce Enhance Shaker to support basic heat environment files
Issue: at AT&T we have large complex heat stacks that all use the basic
parameter functionality found in environment heat files. Using
environment files for heat stacks is a common standard practice that
improves reusability and portability of heat stacks. We would like
to have our Shaker tests also use parameters in environment heat files.

This commit adds the ability to define an environment file that gets
passed to the heat client during stack creation in deploy_from_hot
method

This will allow us to improve the reusability of our test stacks and
improve portability when dealing with large amount of sites. While I
have provided a sample/test template that demonstrates a simple/basic
use case, we're currently using more complex patterns.

For example several of our cloud deployments use Contrail, in those
sites we need to define networks such as:
  aic_trunk_vlan_net_name: 'default-domain:{{ CONF.os_project_name }}:{{
CONF.os_project_name }}_trunk_vlan_net'

Let's assume  we have 100 sites, which all have different unique project
names usually based on region, and a centrally located repo
containing all of our Shaker tests. Rather than changing X amount
of Shaker tests times 100 sites times X projects, we can define an
environment file with the setting above, deploy the "test bundle"
(everything for the test ready to go) as-is and then based on the
shaker.cfg input (every site has one) we can be sure that we're
referencing a valid network and have a successful test.

We also have a lot of existing VNF stacks that all make use of heat
environment files, having the ability to use environment files in
Shaker greatly speeds up our ability to translate those VNFs into
Shaker based tests.

Heat environment files make everything cleaner and better organized.
It may not be apparent with a simple example, but it definitely makes a
difference when dealing with large complex heat stacks.

Change-Id: I857aa292bc89f494b5731c2b9b54742b1e2236f1
2018-10-29 10:25:53 -05:00
Ilya Shakhat 10b919127a Ignore errors from server console retrieval
Server console is used to get errors that may occur
during server boot. Those errors are used to give hints
on critical failures inside VM.

However it should not be critical not to get server console.

Change-Id: Ibf13111a85b8967475dda710830acb212c7d6796
Closes-Bug: #1800158
2018-10-29 00:49:05 +04:00
Ilya Shakhat 2f9335d07a Revert "Report presence of any records with errors"
This reverts commit 0b10da4cb6.

With the original patch Shaker reports errors even if scenario
was completed successfully.

Change-Id: I6c2c26be19c20be8c5e135b64fed6379e249bfe7
Closes-Bug: #1800194
2018-10-28 19:11:38 +04:00
Zuul b300583ebb Merge "Enhance Shaker to use flavor metadata matching with host aggregates" 2018-10-25 08:59:26 +00:00
Oded Le'Sage f4276baa19 Enhance Shaker to use flavor metadata matching with host aggregates
Problem: We have large compute deployments (400+) but some of these
computes have metadata set because they offer special hardware.For this
example let's assume 100 of these computes have the metdata dpdk=true.

There is currently no way in Shaker to choose a flavor that will only
run on a subset of all available computes to run hardware specific tests

It is not feasible or practical to isolate, reconfigure, or create an
availbility_zone or host aggregate for these 100 computes.

Nova has a feature to address situations like this by using
aggregate_instance_extra_specs in a flavor.

This commit enhances Shaker to make use of the Nova feature to match a
flavor with a host aggregate to ensure VMs are only spun up on computes
that can support that flavor.

Using this enhancement users could create a shaker-dpdk flavor with
metadata aggregate_instance_extra_specs:dpdk=true. When running the
function get_available_compute_nodes Shaker will only take into account
the computes which have the metadata dpdk:true as matching/available
computes. This will allow users to run tests on all dpdk computes
only, just by specifying the correct flavor.

If a flavor with no metadata is chosen, the same behavior (use all
computes) that shaker uses today would apply.

Change-Id: If04e90fc23959a2ff5c1a51a1bd19f40cec5cd2b
2018-10-24 11:11:59 -05:00
Zuul 1e7a67d19e Merge "Add zuul job with integration testing" 2018-10-23 09:47:14 +00:00