Commit Graph

40 Commits

Author SHA1 Message Date
Andrey Kurilin 98b326f0e4 [ci] Fix several rally task arguments
There are several invalid cased that Rally ignored before and that will
become an validation error soon.

* transmitting None where dict is expected
* setting 'name' of entities. It is restricted thing, since rally
  generates pseudo-random names that can be filtered by celanup
  mechanism. Currently, Rally overrides 'name' params silently, but it
  will become an error soon.

Change-Id: Icef60dc4db70a3058251909cf485cd8e8424cb16
2020-05-06 14:56:27 +03:00
Bence Romsics 5e0a5d41c7 Rally task definition for port binding scenario
Change-Id: I112e477f95958c6235dbc42da06ebc45b236e8f6
Depends-On: https://review.opendev.org/662781
Partial-Bug: #1833674
2019-07-08 13:25:46 +02:00
Bence Romsics 7be9c10679 Allow VM booting rally scenarios to time out
Not long ago we enabled a rally scenario booting VMs in the neutron gate
so we can collect osprofiler reports about it. The rally scenario we
re-used was only triggered by changes in the rally-openstack repo so I
could not collect data about its failure rate. Now that it's running
frequently in the neutron gate this scenario actually seems to be quite
unstable (usually timing out while waiting for the VM to get to ACTIVE):

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:\"rally.exceptions.TimeoutException: Rally tired waiting\" AND build_name:\"neutron-rally-task\" AND voting:1&from=864000s

Since we only want to run this scenario for the osprofiler report
we can get rid of the gate instability by allowing a 100% failure rate
in the scenario SLA.

Change-Id: Ied354e8242274c8eeb26909e29afbe6d41662bfc
Related-Change: https://review.opendev.org/662804
2019-06-17 13:59:24 +02:00
Bence Romsics ccef17605d Run nova's VM boot rally scenario in the neutron gate
As discussed on the neutron_performance meeting [1] we'd like to run
a rally scenario in the neutron gate that boots a vm. Combined with
Slawek's work to integrate osprofiler with rally [2] we should start
to get some insight into neutron's vif plugging performance even if
that's not something directly exposed and measurable through the API.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/latest.log.html#t2019-06-03T16:12:17
[2] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006769.html

Change-Id: I7e2c19e8cdb62cb85046500f862e8dbd42306225
2019-06-06 10:56:34 +02:00
Zuul 9492d40bfe Merge "rally-jobs: Add FloatingIP workloads: assoc/dissoc" 2019-03-01 21:24:43 +00:00
Zuul 44a07e8217 Merge "Enhance rally-job for sec group rules to creates multiple rules" 2019-02-16 05:51:46 +00:00
Sai Sindhur Malleni 3c2a23ba94 Remove trunk rally scenario from plugins
Since the trunk scenario is now present in rally-openstack
tree, it is no longer needed in neutron tree.

Change-Id: I27c9f0baed267ca8bd181d34842b9d5cc03ab846
2019-02-13 13:09:33 -05:00
Bence Romsics 031a6e5246 rally-jobs: Add FloatingIP workloads: assoc/dissoc
As discussed on the neutron-performance meeting [1] floating IP operations
should be covered in the neutron gate rally job. As I set out to write
the missing tests I discovered that some of them were already written
by rally folks. For example see [2]. Keep in mind the test code has been
relocated since to the rally-openstack repository. For example see [3].
They were later added to neutron's rally job definition [4].

Here we add a one more scenario to exercise the floating IP operations
left uncovered by the above changes: associate and dissociate.

[1] http://eavesdrop.openstack.org/meetings/neutron_performance/2018/neutron_performance.2018-10-22-16.02.log.html#l-143
[2] https://review.openstack.org/225223
[3] https://github.com/openstack/rally-openstack/blob/4e4bfc5/rally_openstack/scenarios/neutron/network.py
[4] https://review.openstack.org/620952

Change-Id: Id2e27258f2e9a35bb8123854e098d0822458b314
Depends-On: https://review.openstack.org/624036
2019-02-13 09:30:20 +01:00
Sai Sindhur Malleni 8362c0a050 Enhance rally-job for sec group rules to creates multiple rules
This commit makes use of recently merged functionality in rally
to create multiple security group rules per security group and list
them. This will help us identify API performance regressions with
respect to security group rules.

Change-Id: I92ac9785d2403c00b873e68f30062f9796b9ac8b
2019-02-12 14:42:00 -05:00
Sai Sindhur Malleni a6422ea8f9 Add Security Group scenarios to rally-jobs
This commit adds security group related rally scenarios to CI.

Change-Id: I11fd3a6d30f743dbeea923407bc72753ba488da4
2019-02-01 15:22:18 -05:00
Juha Kosonen 35d89fad1b rally-jobs: Set floating network as a parameter
The option to define the name of floating network enables more
flexibility for using the task file as is.

Change-Id: Icaf9ca6b337a500d3f76521f94244f1932d0e09b
Signed-off-by: Juha Kosonen <juha.kosonen@nokia.com>
2018-12-17 16:31:10 +02:00
Bence Romsics e294845541 rally-jobs: Add FloatingIP workloads
As discussed on the neutron-performance meeting [1] floating IP
operations should be covered in the neutron gate rally job.
As I set out to write the missing tests I discovered that some of it
was already written by rally folks. For example see [2].
In neutron we only need to add them to the rally job definition.
Keep in mind the test code has been relocated since to the
rally-openstack repository. For example see [3].

[1] http://eavesdrop.openstack.org/meetings/neutron_performance/2018/neutron_performance.2018-10-22-16.02.log.html#l-143
[2] https://review.openstack.org/225223
[3] https://github.com/openstack/rally-openstack/blob/4e4bfc5/rally_openstack/scenarios/neutron/network.py

Change-Id: Icec7011cb293f1c92d968ef60efe7468ea4631fb
2018-11-29 17:10:05 +01:00
Andrey Kurilin 3e83fa1dc7 [rally] Port custom plugins to use proper code
Rally team moved OpenStack plugins under separate repository and in-tree
code is deprecated now and will be removed soon.

This patch changes several imports to use the latest available code.

Change-Id: I901ceb685e75d905578135fdf9f1b08ba3ea7223
2018-05-18 00:25:35 +03:00
Andrey Kurilin 49e3b37e28 [ci][rally] make the job inherit the right parent
Rally team finally added native Zuul V3 jobs (with a bunch of separate
roles and etc) and for simplification of maintainance, it would be nice
to use them.

Change-Id: I755e776a7c24e1bcdf144d7af071a52633aeb94d
2018-05-16 17:05:43 +03:00
Sławek Kapłoński d8914056ad Switch rally trunk scenario to use new rally validators
Validators like:
* rally.task.validation.required_openstack
* rally.task.validation.required_services
are deprecated in Rally 0.10.0 [1].

Instead of calling it directly they should be used via
new decorator 'rally.common.validation.add' and this commit
switches it to use validators in new way.

[1] https://tinyurl.com/y94sfct2

Change-Id: I2883eb94e2532a10160305b283e2d64b93443909
2018-04-26 14:27:02 +02:00
Ihar Hrachyshka 661247a617 Switch rally job from q-* to neutron-* service names
When at least one service named as q-* is present in ENABLED_SERVICES,
then devstack utilizes lib/neutron-legacy to configure services
regardless of how other services are deployed (e.g. with lib/neutron).
This breaks deployment using lib/neutron.

Switching to new names doesn't change anything substantial because
devstack plugin equally handles both variants. It allows to use new
devstack neutron library though.

Change-Id: Id0d35523651131766a70e78bf130205c1c63acd5
2018-01-09 12:12:48 -08:00
Andrey Kurilin 93b8a3301c [rally] Port input task to the new format.
The new task format was introduced recently. It unifies different
sections and tries to make the things a bit simpler.

Rally task consists of subtasks. Their amount should be at least one.
The subtask is a group of workloads. Soon, it will be possible to define
a single SLA for all workloads in a subtask and even more - use once
executed contexts for workloads (i.e. create temporary users, network not
for each workload but for a group of them).

The workload is a combination of different plugins to be executed for a
test. The most important are Scenario plugin (what will be executed in
each iteration), runner (how the load should be generated) and contexts
(what resources should be precreated before the workload).

One scenario with different runners/contexts can create different load.
To distinguish them the new section "description" of workload was
introduced. It allows to add a custom description for a workload which
will be dispayed in the report files. In case of missing "description"
section, the description of scenario will be taken.

Also, I need to mention that "failure_rate: 0" SLA is a default now, so
there is no need to specify it.

Change-Id: If99e8c722d9ccb18b8b9d7e12214e76e483a2016
2017-11-02 12:28:29 +02:00
Swaminathan Vasudevan fdd29a207a Fix the link to the rally docs in README.rst
The links to the rally docs are invalid.
This patch fixes it.

Change-Id: I8713a8cdf317d385e770528111b0ac376f89391f
Closes-Bug: #1718819
2017-09-26 21:11:03 +00:00
Ihar Hrachyshka 8f49038829 rally: switch to new format for context name
Since I9d3bafa075631a3f48cbd3627a4cc1a5a859cce2 in rally, platform
should be part of context name (context@platform).

Otherwise, a warning message (or even breakage because of
I10ac687f9f420dcf0d907b51d5d9303f68d35719) may be triggered.

Change-Id: I6f84282d22d13d36dcba221ab9d94c3fda95f130
2017-09-22 07:27:24 -07:00
Kevin Benton e97ee8a972 Reduce rally sub-port count
This test is executed 4 times and creating 1000 Neutron ports
just to run this scenario 4 times results in this job taking
15 minutes to complete its iterations.

This cuts the count in half to 125 per execution to cut the run
time in half and ensure we don't get to close to the gate timeout.

This was done once before as part of
817a19c4b9 but unfortunately being
part of another SLA change resulted in it being reverted as part
of the SLA change.

Change-Id: I61466d87b002252efc163cbb5d03eafc5d4da3fb
2017-08-10 11:21:03 -07:00
Jenkins 6ca3ccc209 Merge "Revert "Rally: decrease SLA for avg list of ports and nets"" 2017-04-17 23:02:41 +00:00
Ihar Hrachyshka 7d54652e13 Revert "Rally: decrease SLA for avg list of ports and nets"
This reverts commit 817a19c4b9.

This apparently hits us in gate where I can see at least one
failure with 5.44 secs taken for a scenario.

Change-Id: Ied0516686e86167a0c7b1d480eb1db2789f7cada
2017-04-17 15:45:34 +00:00
Jenkins b49764cdc7 Merge "Rally: decrease SLA for avg list of ports and nets" 2017-04-17 07:03:59 +00:00
Ihar Hrachyshka d28a33a898 Fixed validation of create_and_list_trunk_subports rally scenario
New validation now enforces that times is >= concurrency because times
is total number of *runs*, not the number of concurrent *run sets*.

Change-Id: I454ef821e00bd5123a9640f472ad4b034dbec75e
Closes-Bug: #1680580
2017-04-06 12:30:15 -07:00
Kevin Benton 817a19c4b9 Rally: decrease SLA for avg list of ports and nets
In a normal gate run these are returning in 2 seconds each
on average. Let's reduce the SLA of these from 15 to 5 now
to help prevent future performance regressions in this area.

Change-Id: Iae174c95d214c83d6726a3d3bd339dde7886af4f
2017-03-18 01:24:49 +00:00
Kevin Benton f82667e8eb Trim rally scenario sizes
During a normal run, the top three scenarios account for slightly
more than a half hour of runtime. Sample numbers:

Scenario                                    Load Duration Full Duration
NeutronNetworks.create_and_update_subnets       562.189   1,182.400
NeutronTrunks.create_and_list_trunk_subports    427.475    600.721
NeutronNetworks.create_and_list_ports           310.167    540.144

This patch reduces the resources created by each of the 3 by 75%. This
should save us an additional ~20 minutes during a normal gate run, which
should change our window for timeout from approximately a half hour on a
normal node to about 50 minutes.

This additional buffer should hopefully be enough to reduce the failure
rate for the rally job when it gets scheduled to a slow node.

Change-Id: I923b625f7dd3ebf794b6a9e097f5ed12ce446bb5
2017-03-15 04:27:42 +00:00
Kevin Benton 8976e42e30 rally trunk port list
Change-Id: Idbbe08bdef83c60fad49f49996a74fd999914fe8
2016-12-10 06:15:34 -08:00
Kevin Benton d2c292e5dc Add a trunk rally test
This adds a basic rally scenario to create a trunk
with a bunch of subports so we can keep an eye on the
performance of the trunk API.

Change-Id: I12aaf6121b677e9696131601b3539a7091e2858c
2016-11-14 23:59:56 +00:00
Armando Migliaccio f5e30abcc5 Add rally hook to neutron devstack plugin
This allows us to configure neutron when running the rally job in
the gate. This effort stems from patch [1]. Blame Kevin for not
wanting to squash the two together.

[1] I12aaf6121b677e9696131601b3539a7091e2858c
Change-Id: I006957784ac7900021bcfee57cbc83b5a6c533c4
2016-11-10 07:44:28 +00:00
Kevin Benton 16901e8472 Reduce rally executions of create_and_list_ports
The previous configuration of the task was taking up
to a half hour to run. Between this task and the others,
it was eating up all of our gate time, leaving no room
to add new jobs.

This reduces it 5 times in the number of runs, from 40
to 8. This still gives us a reasonable number to get an
average from, especially since each run creates 100 ports.

Change-Id: I0955e44df1a9e072c58fdacc337121b8621132df
2016-11-05 15:00:19 -06:00
Jenkins 00197acd99 Merge "Increase subnet count for rally test" 2016-10-26 11:37:52 +00:00
Jenkins bc0146e350 Merge "Lower concurrency on rally port job" 2016-10-05 02:28:44 +00:00
Kevin Benton 648e079f6b Increase subnet count for rally test
Test concurrency of subnet create/delete inside of a tenant.

Change-Id: I36e5ed927859d228911cd64c5c6228eeca9fa81c
2016-09-20 02:10:35 -07:00
Kevin Benton a15d259b11 Lower concurrency on rally port job
Previous runs are showing that creating ports under this high load
ends up taking >5 secs per port on average. Lets set to 4, which is
still double the api_worker count in some cases for the current gate.

Change-Id: I05a0d28f5b035684e07288825f5b704a843dc9d7
2016-09-20 02:08:21 -07:00
Kevin Benton 933af3fdac Add to rally quotas to handle worst case quota race
Quoting the quota devref:
"""
For a reservation to be successful, the total amount of resources requested,
plus the total amount of resources reserved, plus the total amount of resources
already stored in the database should not exceed the project's quota limit.
"""

This means that in the absolute worst case scenario with 20 concurrent
workers, 19 could have made reservations, committed resources, but not
yet cleared their reservation. Because of the outstanding reservation
and the resources created by the 19 workers, they will all be
double-counted until their reservation is cleared (or it expires).

This adjusts the rally scenarios to handle the double-count for
concurrency.

Related-Bug: #1623390
Change-Id: I4808a92e7e6067aeeb62fc3b3d7f7ac71b179c44
2016-09-19 22:11:15 -07:00
Kevin Benton a5d7b1ab51 Increase rally network/port count and add quotas
This increases the rally port and network count to 100
and enables quotas to exercise the quota engine to
better simulate a real system.

Additionally, it reduces the SLA requirements because of
regressions that have snuck in throughout the cycle. As
they are fixed these should be reduced back down.

Change-Id: I042d64245b1e4486334996d834ad31561613fa50
2016-08-18 10:51:46 +00:00
Kevin Benton 52fcdb4297 Increase ports per network and add SLA for rally
Increase the ports per network in the create and list ports
test. This also adds a max average time SLA so we can catch
regressions in performance.

Change-Id: I2e7e3fd7406db77c8e44dce2ab0b4594ff6f2db9
2016-04-01 23:09:19 +00:00
Alexander Maretskiy 28c95da3f5 Changes in rally-jobs/README.rst
Changes:
  * fix broken links
  * add link to release notes

Change-Id: Ie42b246aac2fd4f6de8cd8bcf28e5ff2906ffd32
2015-06-04 10:34:15 -04:00
Boris Pavlovic 6c61cf7af7 Run more Rally benchmark on every patch
* Since 24 Nov 2014 we added a lot of Neutron benchmarks

Running more Neutron related benchmarks in Neutron gate allows
to avoid performance regressions and races.

* Neutron benchmarks are described here:
https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/neutron/network.py
It's quite simple code be free to take a look.

* All changes in concurrency and times are related to optimization
of duration/usefulness

* To get description of benchmarks use:

   rally info find NeutronNetworks.create_and_update_networks

New benchmarks:

- NeutronNetworks.create_and_update_networks

- NeutronNetworks.create_and_delete_networks

- NeutronNetworks.create_and_update_subnets

- NeutronNetworks.create_and_delete_subnets

- NeutronNetworks.create_and_update_routers

- NeutronNetworks.create_and_delete_routers

- NeutronNetworks.create_and_list_routers

- NeutronNetworks.create_and_update_ports

- NeutronNetworks.create_and_delete_ports

- NeutronNetworks.create_and_list_ports

- Quotas.neutron_update

related bug: #bug 1419723
Change-Id: Ie3c84e057fc96c0f35ad77b7297c564442ebcf10
2015-03-18 10:38:44 +00:00
Boris Pavlovic fbd42439ca Update rally-jobs files
*) Rename rally-scenarios that is quite misleading to rally-jobs.
   rally-jobs makes much more sense, cause it actually contains files
   related to rally job

*) Update rally-jobs/README.rst to add more info

*) Update rally-jobs/plugins/README.rst to expaling plugins

*) Add new directory rally-jobs/extra, this directory is copy pasted
   in gates and can be used for files that are required by some of
   benchmarks

Change-Id: I6d0c0435a4bb4658ddf4adb871bc36ab8c157f3e
2014-11-24 23:21:11 +04:00