The constants used for configuring default policy rules were initialized
before configuration file was parsed. As the result the configuration
options in 'roles_middleware' group didn't have effect on registered
policy rules. The behavior can be reproduced in devstack plugin where
monasca-log-agent using 'monasca-agent' role is not authorized to send
logs to the API.
The change postpones the initilization of these constants. Additionally
obsolete 'roles' filter was removed from the paste pipelines.
Change-Id: I0ca38f2cc7d63b9b47eedb304ba7b00a94816f9a
Story: 2001233
Task: 24250
Community has set a community wide goal
in Pike cycle: "Control Plane API endpoints deployment via WSGI" [1].
monasca-log-api is already capable of being deployed under mod-wsgi but
not under uwsgi. This commits enables that feature. Until Queens release
API will hence support:
* gunicorn
* mod_wsgi (remove in Queens)
* uwsgi (preffered)
Refs:
[1]:
https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html
Story: 2001464
Task: 6180
Change-Id: Ie4a94c9c2e166915c3f4a1783dc3527f4eb62f29
Old project tree had everything mixed up
in different locations. Not to mention
that actual REST controllers resided in
package that's name did not reflect the content.
Reorganized the tree to be more predictable
and easier to maintain.
Change-Id: Ic9acc80a4cf7416394702d43141c96953c03beea
It is common for OpenStack projects to use oslo-config-generator to generate
a sample config file which contains all available configuration options.
Work items:
* removed etc/monasca/log-api.conf (it is autogenerated)
* keep all config in one place to mimic the configuration file
* added configuration files to documentation
Story: 2000970
Task: 4121
Change-Id: I8777ed6cd38739e87b19be248b9c120e84626ad1
Following commit introduces using composite
paste file to describe application pipelines.
With the help of composite feature it was possible to
define 4 distinct supapplications, each having different
pipeline setup:
* version_app => no keystoneauth, simple data return with error trap
* healthcheck_app => no keystoneauth, simple data return with error trap
* api_v2 => as it was, added error_trap though
* api_v3 => as it was, added error_trap though
Following approach opens up the possibilities of modelling an API
in far more flexible way and removes the need of any hacking inside
the codebase.
Note:
Keeps backward compatibility with older codebase.
Extra:
* removed dirty hack around skipping certain request.path
in keystonemiddleware (now described in paste file)
* removed 404 when issuing ```curl api_host:api_port/```. Request
is redirected down to VersionApp
Depends-On: I0323eacb5cbba8418550e3322189104c35cf4c36
Change-Id: I873e1376665c8cf66c8ee503975324f9b93ddb45
Similar to other openstack projects,
monasca-log-api should provide information
about request's context in the log information.
This is done by:
* implementing custom Request that creates context (oslo.context)
* changing logging configuration to use ContextLogFormatter
Additionally, since information like tenant-id can be
retrieved from the context, modified resource code to use
that.
Change-Id: I992a2f4de64b54a705cc2bafdb3e83de0b1691c7
Following change allows to run monasca-log-api
devstack plugin on the Xenial environment.
That was implemented with the help of:
* start/stop of processes handles through run_process
and stop_process & no services files
* setting up environment using devstack functions
** monasca-log-api virtual env (USE_VENV)
** installing monasca-log-api dependencies from git
** saving data in keystone catalog with Openstack CLI
Additionally added method that downloads tarballs
only if they are not found in the system.
Change-Id: I08b2ddbe17b2c7899056a63a5633070ae9a2c2db
By default log file is rotated after reaching size of 100MB and 5 files
are archived.
Closes-Bug: 1621329
Related Change-Id: Iac7f29b3909354f75e5d3264ef2c987c9f3e2aec
Change-Id: I8d7e071e899eca1247b2db56ea348bb8775d68c3
Following commit adds monitoring
for monasca-log-api with metrics:
* monasca.log.in_logs - amount of logs that API has received
* monasca.log.in_logs_rejected - size of received logs in bytes
* monasca.log.processing_time_ms - log-api time to process received logs
* monasca.log.out_logs - amount of logs pubslished to kafka
* monasca.log.out_logs_lost - amount of logs that were lost (critical errors)
* monasca.log.publish_time_ms - time log-api needed to pubslish logs
* monasca.log.in_bulks_rejected - amount of rejected bulk requests
* monasca.log.out_logs_truncated_bytes - amount of truncated bytes from messages
Change-Id: Ib4165fe128e87b356415da8423f536d393c89f01
Added new variable max_message_size to config file.
Do not use the same variable to validate envelope and payload
size.
Change-Id: I1552ef88a5b54ad8d7b530627f2c54cededd370e
Healthcheck and Versions are endpoints
that are not available under either
v2.0 and v3.0. Changes includes:
- package reornigazation
- tests adjustments
Change-Id: Ib6bdd4695b5ebf57f27fc8bd311222bb216dd470
Implemented specification proposal for batch support at,
https://review.openstack.org/#/c/273058/
Note, if you want to use this in the monasca-vagrant environment
it should work. I've modified the value of the field kafka_url in
etc/monasca/monasca_log_api.conf to use the kafka server in the
mini-mon vm.
After you've deployed monasca-vagrant using "vagrant up"
ssh into mini-mon "vagrant ssh mini-mon" and then create a new
topic for logs which can be done running the command:
/opt/kafka/bin/kafka-topics.sh --create -zookeeper localhost:2181
--replication-factor 1 --partitions 128 --topic logs
To watch the log messages at the console in the mini-mon VM
/opt/kafka/bin/kafka-console-consumer.sh
--zookeeper localhost:2181 --topic logs
Change-Id: I4247d1824a237ecbe4db878e72485938f40a31c3
Healthcheck allows to verify if:
- API is up and running
- Kafka, that monasca-log-api sends data to, is up and running
and an expected topic can be found there.
Other:
- added documentation entries
Change-Id: I316c1d9518cfed37119f11c326c071bfbfc7658e
Created middleware to authorize access
to log api. Only configured roles (i.e. default) can access
the api. Also middleware detect if the request would
come from monasca-log-agent.
Summary:
- middleware added (logic + tests)
- extended documentation
Additionaly:
- added better tox processing
- added minimum coverage
Change-Id: Ic848bfa3a8552887661f8223078efe3a4bca5c37
Received request's content should be validated if
its size does not exceed allowed value. Bytes refers
to byte size of the object instead of amount
of characters. This change is required in order
to reject those meesages that couldn't be processed
by Kafka queue.
Changed:
- added payload size validation to Python
- added validation of message size that it sent to Kafka
- reworked validation of message size in Java
Change-Id: I2acc647550d7c851a5715a7cf44f749db1f54d7b
- removed own kafka abstraction in favour of monasca-common
- removed monasca keystone context filter, not actually used
- changed URI of logs endpoints to /v1.0/log/single
Change-Id: Iaceabdce2b2862451cfe63d2a612577d7710022b
- single log message with rest api
- parsing / validation for data
- configuration
- bootstrapping
- tox
- unit tests
Change-Id: I7386b3500ee9097383a573bf915da55ce2ff881f