Commit Graph

2389 Commits

Author SHA1 Message Date
Mehdi Abaakouk (sileht) 9f720ecdd7 Revert "Revert "Retire project""
This reverts commit bc81f1c699.

Change-Id: I7a7bc3deeadd094d7a42b47b16cde2e8a8805a24
2017-06-05 17:04:30 +00:00
gordon chung bc81f1c699 Revert "Retire project"
not ready yet.

This reverts commit 35c7613e65.

Change-Id: I381323562fe8233c088c04128742b3c237c3dca1
2017-06-05 13:24:27 +00:00
Julien Danjou 35c7613e65 Retire project
Change-Id: Ic8651e5bdef3468d02e05f7d196cc09cd2c0de8d
Depends-On: I260058e48198013f35c638cc3d78ef335637f298
2017-05-19 19:26:07 +02:00
Jenkins 91656b0a43 Merge "travis: fix travis-ci docker image" 2017-05-19 15:45:07 +00:00
Jenkins 4317d3a890 Merge "carbonara: reduce the number of array copy" 2017-05-19 15:45:01 +00:00
Jenkins 176c1332f6 Merge "Remove oslo.utils dependency" 2017-05-19 14:57:54 +00:00
Jenkins 934ad03612 Merge "wsgi: use pbr wsgi binary" 2017-05-19 14:33:50 +00:00
Jenkins 9633937202 Merge "carbonara: rewrite lock debug statement" 2017-05-19 14:19:48 +00:00
Mehdi Abaakouk 86694d656c travis: fix travis-ci docker image
A couple of fix for the docker image used by travis:

* locale package is no more installed by default
* liberasurecode-dev is required, now
* sphinx >= 1.6.0 have broken sphinx-versioning
* Don't use sphinx math module

Change-Id: Iba06d0c4667e2a11495fb25375de7152b2b02597
2017-05-19 14:20:09 +02:00
Mehdi Abaakouk 6377e25bdc wsgi: use pbr wsgi binary
We don't need to provide the app.wsgi script anymore.
pbr provides it within the gnocchi-api binary

Change-Id: I6e16607128849a18b9e6eb1bc5558d1c9df64775
2017-05-19 07:29:42 +02:00
gord chung 8a2f36cbae add release note for sacks
Change-Id: I0c0f190f556b36579a3afea63dc79fa7b1a9f443
2017-05-19 03:06:04 +00:00
Julien Danjou cf762fbb3d Remove oslo.utils dependency
Change-Id: I5d1a027f4e2d550ab8d27ca0cf9bc61c586f2eed
2017-05-18 23:36:28 +02:00
Julien Danjou 11e5668e69 carbonara: rewrite lock debug statement
There's no lock anymore so this should not be logged like that. instead, we
only LOG.debug if we successfully processed the log. The time is not useful
anymore and since log are timestamped, it can be computed if needed anyway.

Change-Id: I40d38985f7a4f3368aeeec56d2b592f64882d1f7
2017-05-18 23:36:28 +02:00
Jenkins c4183fe4fa Merge "rest: do not duplicate non-existing resource ids" 2017-05-18 20:30:07 +00:00
Jenkins d7d422ab06 Merge "ceph: store measures in omap values" 2017-05-18 20:03:01 +00:00
Jenkins 121e5cc3a7 Merge "ceph: async write new measures" 2017-05-18 20:02:55 +00:00
Jenkins b02387794b Merge "drop metric lock" 2017-05-18 19:07:15 +00:00
Jenkins 38df0f67f0 Merge "don't lock on delete" 2017-05-18 19:07:09 +00:00
Jenkins b47fa1b1c5 Merge "sack-changer cli" 2017-05-18 19:03:28 +00:00
Jenkins 735c59bd69 Merge "configurable sacks framework" 2017-05-18 19:03:23 +00:00
Jenkins 62e140317d Merge "configurable hashring replicas" 2017-05-18 19:03:17 +00:00
Jenkins 7dea926e98 Merge "Stop using oslo_utils.netutils" 2017-05-18 18:47:21 +00:00
Jenkins 6aa5491203 Merge "implement hashring partitioning" 2017-05-18 18:34:15 +00:00
Jenkins 8f14739350 Merge "fix workload logging" 2017-05-18 17:33:53 +00:00
Jenkins 6ab74b988a Merge "Stop using oslo_utils.fnmatch" 2017-05-18 17:20:12 +00:00
Jenkins eaf46226ef Merge "ceph: fix errno_to_exception" 2017-05-18 17:19:44 +00:00
Jenkins 0ea87f2e1a Merge "redis: fix CLIENT_LIST_ARGS parsing" 2017-05-18 16:37:57 +00:00
Jenkins f9e4aa2566 Merge "doc: move storage plan to running" 2017-05-18 16:37:50 +00:00
Julien Danjou bbf27ad09f Stop using oslo_utils.netutils
The urlsplit provided does not provide anymore value than the builtin one in
this case.

Change-Id: I4a40c35b7123b7760c354ccfd7e1a5a0c156becc
2017-05-18 16:52:35 +02:00
Julien Danjou 26f87a457e Stop using oslo_utils.fnmatch
This was a work around for old buggy Python 2.7 versions. Everybody will have
to upgrade!

Change-Id: Id105e2e8c1fab7af49c11b323db64886e57eade6
2017-05-18 16:52:35 +02:00
Jenkins ffbb446f3f Merge "Stop relying on dictutils" 2017-05-18 09:18:12 +00:00
Jenkins c80f869db6 Merge "Replace oslo_utils.units.M by its value" 2017-05-18 09:00:56 +00:00
Julien Danjou 8dc03b9aca redis: fix CLIENT_LIST_ARGS parsing
The entire list must be taken, not only the last argument.

Change-Id: I9b5b9336250cb5292cdd6c7ca4964bfe186575d5
Closes-Bug: #1691542
2017-05-17 23:31:28 +02:00
Julien Danjou 906951633b Fix Carbonara compression with Cradox and LZ4 >= 0.9.2
Change-Id: Ifca3c175146887db153b2f5bb780a532e8bc6d48
2017-05-17 17:41:19 +02:00
gord chung 3332206969 ceph: store measures in omap values
ceph sucks for small objects; POSTs to ceph incoming are always
small objects.

this changes logic to store new measures as omap values instead of
objects. this allows us to write to 'memory' aka leveldb/rocksdb
instead of disk.

this does not change the durability agreement since we are already
storing object keys in omap so if omap fails, we will lose link
to objects regardless if on disk or not.

using local 20OSD ceph cluster, with 18 metricd. this is:
- ~2x faster than aio_write patch to POST
- ~2x faster than aio_write patch to process
- ~3x faster than no aio_write patch to POST
- ~3x faster than no aio_write patch to process
- ~3.5x faster than 3.1 to POST
- something a lot faster than 3.1 to process (no idea why)

Change-Id: I4bae365955fdbafe4ad837596490774c42bc5251
2017-05-16 15:39:09 +00:00
Julien Danjou 4444130a6a Stop relying on dictutils
Change-Id: Ida27c6ae1f6b5e5cd9a763c596054ccda2f9a38f
2017-05-16 11:23:34 +02:00
Julien Danjou 45954c26c8 Replace oslo_utils.units.M by its value
Really?

Change-Id: Ibcf02f81c6148a0958a4924a1b322acf3d92959e
2017-05-16 11:21:02 +02:00
Mehdi Abaakouk e35e526ad6 ceph: fix errno_to_exception
We are using an internal API of rados.

This change removes its usage.

Closes-bug: #1690876
Change-Id: Ia1304395748f54fed0995406b9bc7260af8f5e84
2017-05-16 05:02:42 +00:00
gord chung 60f7e33bd9 fix workload logging
accidentally show total metrics in last sack as total metrics
processed.

Change-Id: I5c95857ab35ee0d2ba0932be9634853075851bb1
2017-05-15 22:53:33 +00:00
gord chung 63fa106d78 ceph: async write new measures
we should use async writes since we're not using threads anymore.

testing on small ceph cluster and posting 20metrics/POST results in
~2x better write performance.

Change-Id: Ic451e6ea874a2a3695d3d148b1c05ecc54aa473b
2017-05-15 17:13:08 +00:00
Julien Danjou bca589a557 doc: move storage plan to running
makes more sense

Change-Id: Id5eb6d15cf521b6ade14d37ebcb648f4489cb63c
2017-05-15 18:14:14 +02:00
gord chung 5c18a8d4fe drop metric lock
lock sack when refresh param is given. check first if there's
anything to be refreshed rather than blindly blocking.

Change-Id: I945db03e80450d35877427fce9c12675891e016d
2017-05-15 16:00:51 +00:00
gord chung 3099bdbebf don't lock on delete
we don't need to lock metric on delete but rather we only need to
check if sack is being processed. either:

1) sack is locked, so there's a chance that metric is being processed.
therefore we skip.
2) sack is unlocked, so there's no chance that a concurrent process
will start processing metric as the indexer already says it's already
deleted and no other process can see it as not-deleted.

Change-Id: I8df135621dfabc3d17733a3577d0ea60b30e83e4
2017-05-15 13:26:45 +00:00
gord chung 5b83fe64cc sack-changer cli
support ability to change sacks via command line. will set new
sack value and clear any old sacks (if applicable). backlog must
be empty to change value.

Change-Id: Icf74b081e4cfaaaa607a5b9c684cbad4b8ecc006
2017-05-15 13:26:32 +00:00
gord chung 1bacea9416 configurable sacks framework
this adds framework to configure sack size without corruption.

- default to 128 sacks
- add note on how to calculate how many sacks to set

actual ability to change value done in subsequent patch.

Change-Id: I389c1a7ca9b3fe39b3716782e85073796ad26333
2017-05-15 13:23:38 +00:00
gord chung b21b878f36 configurable hashring replicas
as a trade off between less load on coordination backend vs higher
worker utilisation.

Change-Id: I679e79edf857be3660a80e54e039ded60e8431b5
2017-05-15 13:23:29 +00:00
gord chung e82460117a implement hashring partitioning
support hashring partitioning if user wants to reduce potential
locking load by sacrificing potential throughput. if hashring is not
supported or does not assign jobs, we just default to entire set of
sacks. if fails to set up partitioning, try again.

Change-Id: I1439fb3cdb171ce57ce7887857aa4789fe8f0d9c
2017-05-15 13:23:18 +00:00
Julien Danjou f8b18df786 doc: fix typo and URL in collectd page
Change-Id: I9743f7f5f7e61173416f29478178567393f38355
2017-05-13 13:11:34 +02:00
gord chung 25cc9d7a03 s3: push to multiple sacks
enable s3 to push to multiple sacks and handle them accordingly

Change-Id: If68b4ef0c172f824b86b120092feb936a1c2fda2
2017-05-10 20:36:52 +00:00
gord chung 85b41d699a swift: push to multiple sacks
enable swift to push to multiple sacks and handle them accordingly

Change-Id: I0550961960805505d85919f30bfa1677dc390345
2017-05-10 20:36:35 +00:00