Commit Graph

24 Commits

Author SHA1 Message Date
Martin Chacon Piza 91022f3e80 Remove project content on master branch
This is step 2b of repository deprecation process as described in [1].

[1] https://docs.openstack.org/project-team-guide/repository.html#step-2b-remove-project-content

Change-Id: Ie446c1cd447789d189a9e723fff9ee783dd3cf4f
2021-02-22 14:57:15 +01:00
Witold Bedyk 7d40e5d5c2 Fix default policy rules registration
The constants used for configuring default policy rules were initialized
before configuration file was parsed. As the result the configuration
options in 'roles_middleware' group didn't have effect on registered
policy rules. The behavior can be reproduced in devstack plugin where
monasca-log-agent using 'monasca-agent' role is not authorized to send
logs to the API.

The change postpones the initilization of these constants. Additionally
obsolete 'roles' filter was removed from the paste pipelines.

Change-Id: I0ca38f2cc7d63b9b47eedb304ba7b00a94816f9a
Story: 2001233
Task: 24250
2018-08-03 10:29:06 +02:00
Tomasz Trębski 71f0bdfb8c Enable UWSGI support in monasca-log-api
Community has set a community wide goal
in Pike cycle: "Control Plane API endpoints deployment via WSGI" [1].
monasca-log-api is already capable of being deployed under mod-wsgi but
not under uwsgi. This commits enables that feature. Until Queens release
API will hence support:

* gunicorn
* mod_wsgi (remove in Queens)
* uwsgi (preffered)

Refs:
[1]:
https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html

Story: 2001464
Task: 6180

Change-Id: Ie4a94c9c2e166915c3f4a1783dc3527f4eb62f29
2018-01-16 14:36:45 +01:00
Tomasz Trębski 9c163622e2 Migrate log integration test to ZuulV3
- define .zuul.yaml to contain new job definitions
- move gate hooks to appropriate location

Story: 2001238
Task: 5763
Task: 5766

Depends-On: I263b1a92a9ec7b5e38d9c1dc8b02e214102a92c2
Change-Id: Icfb32df7e083c2c58c89fa3d2fe9a3eacfda6beb
2017-10-24 08:15:57 +00:00
Tomasz Trębski f55f34c9c4 Refactor project tree
Old project tree had everything mixed up
in different locations. Not to mention
that actual REST controllers resided in
package that's name did not reflect the content.

Reorganized the tree to be more predictable
and easier to maintain.

Change-Id: Ic9acc80a4cf7416394702d43141c96953c03beea
2017-07-27 22:02:48 +00:00
Thomas Bechtold d243ed35d0 Use oslo-config-generator
It is common for OpenStack projects to use oslo-config-generator to generate
a sample config file which contains all available configuration options.

Work items:

* removed etc/monasca/log-api.conf (it is autogenerated)
* keep all config in one place to mimic the configuration file
* added configuration files to documentation

Story: 2000970
Task: 4121

Change-Id: I8777ed6cd38739e87b19be248b9c120e84626ad1
2017-07-05 09:44:35 +00:00
Witold Bedyk 9bb918197e Fix cross-tenant logs submission
* add 'delegate_roles' configuration option

Change-Id: If4952b84536ef058d91f6ee2332076dc448d97bd
2017-06-03 14:19:52 +02:00
Tomasz Trębski 5e7ebdd9c5 Composite paste for monasca-log-api
Following commit introduces using composite
paste file to describe application pipelines.
With the help of composite feature it was possible to
define 4 distinct supapplications, each having different
pipeline setup:

* version_app => no keystoneauth, simple data return with error trap
* healthcheck_app => no keystoneauth, simple data return with error trap
* api_v2 => as it was, added error_trap though
* api_v3 => as it was, added error_trap though

Following approach opens up the possibilities of modelling an API
in far more flexible way and removes the need of any hacking inside
the codebase.

Note:
Keeps backward compatibility with older codebase.

Extra:
* removed dirty hack around skipping certain request.path
in keystonemiddleware (now described in paste file)
* removed 404 when issuing ```curl api_host:api_port/```. Request
is redirected down to VersionApp

Depends-On: I0323eacb5cbba8418550e3322189104c35cf4c36
Change-Id: I873e1376665c8cf66c8ee503975324f9b93ddb45
2017-06-02 14:20:43 +00:00
Tomasz Trębski f00e8c9012 Integration with oslo.context
Similar to other openstack projects,
monasca-log-api should provide information
about request's context in the log information.

This is done by:
* implementing custom Request that creates context (oslo.context)
* changing logging configuration to use ContextLogFormatter

Additionally, since information like tenant-id can be
retrieved from the context, modified resource code to use
that.

Change-Id: I992a2f4de64b54a705cc2bafdb3e83de0b1691c7
2017-01-09 05:36:03 +00:00
Tomasz Trębski 900e32ad8e Adjust monasca-log-api devstack to Xenial
Following change allows to run monasca-log-api
devstack plugin on the Xenial environment.
That was implemented with the help of:
* start/stop of processes handles through run_process
  and stop_process & no services files
* setting up environment using devstack functions
** monasca-log-api virtual env (USE_VENV)
** installing monasca-log-api dependencies from git
** saving data in keystone catalog with Openstack CLI

Additionally added method that downloads tarballs
only if they are not found in the system.

Change-Id: I08b2ddbe17b2c7899056a63a5633070ae9a2c2db
2016-12-07 10:18:09 +00:00
wachowskij feb6897cbe Enable size-based log rotation for monasca-log-api
By default log file is rotated after reaching size of 100MB and 5 files
are archived.

Closes-Bug: 1621329
Related Change-Id: Iac7f29b3909354f75e5d3264ef2c987c9f3e2aec
Change-Id: I8d7e071e899eca1247b2db56ea348bb8775d68c3
2016-09-14 08:59:07 +02:00
Jenkins e3a4f3e038 Merge "Change Monasca Port" 2016-07-15 06:06:10 +00:00
Tomasz Trębski c3bb9b9eb4 Monitoring for monasca-log-api
Following commit adds monitoring
for monasca-log-api with metrics:
* monasca.log.in_logs - amount of logs that API has received
* monasca.log.in_logs_rejected - size of received logs in bytes
* monasca.log.processing_time_ms - log-api time to process received logs
* monasca.log.out_logs - amount of logs pubslished to kafka
* monasca.log.out_logs_lost - amount of logs that were lost (critical errors)
* monasca.log.publish_time_ms - time log-api needed to pubslish logs
* monasca.log.in_bulks_rejected - amount of rejected bulk requests
* monasca.log.out_logs_truncated_bytes - amount of truncated bytes from messages

Change-Id: Ib4165fe128e87b356415da8423f536d393c89f01
2016-07-14 08:08:58 +02:00
Matteus Silva 80655f935e Change Monasca Port
Actually, the port for log api is 5607.

Change-Id: I1b0d7128fe1fb62843a21e6aab4966ecc77d74e6
2016-07-13 19:41:21 -03:00
Artur Basiak 310e3053df Kafka message max size
Added new variable max_message_size to config file.
Do not use the same variable to validate envelope and payload
size.

Change-Id: I1552ef88a5b54ad8d7b530627f2c54cededd370e
2016-05-23 07:50:53 +02:00
Tomasz Trębski 81413d4d52 Moving unversioned endpoints
Healthcheck and Versions are endpoints
that are not available under either
v2.0 and v3.0. Changes includes:
- package reornigazation
- tests adjustments

Change-Id: Ib6bdd4695b5ebf57f27fc8bd311222bb216dd470
2016-03-11 08:21:47 +00:00
Roland Hochmuth 9b93fc8864 Add implementation of /v3.0/logs API
Implemented specification proposal for batch support at,
https://review.openstack.org/#/c/273058/

Note, if you want to use this in the monasca-vagrant environment
it should work. I've modified the value of the field kafka_url in
etc/monasca/monasca_log_api.conf to use the kafka server in the
mini-mon vm.

After you've deployed monasca-vagrant using "vagrant up"
ssh into mini-mon "vagrant ssh mini-mon" and then create a new
topic for logs which can be done running the command:

/opt/kafka/bin/kafka-topics.sh --create -zookeeper localhost:2181
--replication-factor 1  --partitions 128 --topic logs

To watch the log messages at the console in the mini-mon VM

/opt/kafka/bin/kafka-console-consumer.sh
--zookeeper localhost:2181 --topic logs

Change-Id: I4247d1824a237ecbe4db878e72485938f40a31c3
2016-02-24 23:09:39 -07:00
Tomasz Trębski 412892aed2 Adding healthcheck
Healthcheck allows to verify if:
 - API is up and running
 - Kafka, that monasca-log-api sends data to, is up and running
 and an expected topic can be found there.

Other:
- added documentation entries

Change-Id: I316c1d9518cfed37119f11c326c071bfbfc7658e
2016-02-22 12:49:21 +01:00
Tomasz Trębski 33e9d7aaa3 Security update for monasca-log-api
Created middleware to authorize access
to log api. Only configured roles (i.e. default) can access
the api. Also middleware detect if the request would
come from monasca-log-agent.

Summary:
- middleware added (logic + tests)
- extended documentation

Additionaly:
- added better tox processing
- added minimum coverage

Change-Id: Ic848bfa3a8552887661f8223078efe3a4bca5c37
2015-12-17 11:27:12 +01:00
Tomasz Trębski 1feaa74013 Validating message size
Received request's content should be validated if
its size does not exceed allowed value. Bytes refers
to byte size of the object instead of amount
of characters. This change is required in order
to reject those meesages that couldn't be processed
by Kafka queue.

Changed:
- added payload size validation to Python
- added validation of message size that it sent to Kafka
- reworked validation of message size in Java

Change-Id: I2acc647550d7c851a5715a7cf44f749db1f54d7b
2015-12-10 12:29:58 +01:00
Tomasz Trębski dc89cde325 Syncing API version
Python implementation should have the same API
version as Java implementation.

Change-Id: I2ea33eb37076ec662c883e7afaf8617fb3dcf857
2015-12-04 06:42:58 +00:00
Witold Bedyk 13653d21ec Remove not used config variable
Change-Id: I0c2ac84749b179a7f532db395c735a1a8a1af1e6
2015-11-24 17:05:23 +01:00
Tomasz Trębski f07a38e388 Fixing issues spotted in dev environment
- removed own kafka abstraction in favour of monasca-common
- removed monasca keystone context filter, not actually used
- changed URI of logs endpoints to /v1.0/log/single

Change-Id: Iaceabdce2b2862451cfe63d2a612577d7710022b
2015-11-24 11:36:08 +01:00
Tomasz Trębski e921fd506c monasca-log-api python
- single log message with rest api
- parsing / validation for data
- configuration
- bootstrapping
- tox
- unit tests

Change-Id: I7386b3500ee9097383a573bf915da55ce2ff881f
2015-10-08 10:38:12 +02:00