The endpoint_id option is required because it is the key to look up
the target endpoint from Keystone catalog. This ensures more sensible
error is shown in case endpoint id is not set, instead of trying to
look up endpoint by 'None'.
Change-Id: I43859dec08040130e4e5fa6ad65b74be21489aff
In change I22234e0bb6b3a1cecb29a6b99a3afcd02ffdbf5f added a
get_registered_limits() interface but missed making it work with the
Enforcer cache because caching was implemented in a higher layer,
_get_limit() and get_registered_limits() could not make use of
_get_limit().
This moves the caching logic to where limits are retrieved from
keystone, so that get_registered_limits() will use caching and any
interfaces that might be added in the future will be able to benefit
from using the cache.
Closes-Bug: #1964848
Change-Id: I28cdd4d3f4927b94acea3309b36999850cf2ee2a
This fixes how the opts data is formatted to fix auto-generation of
config file using oslo-config-generator with oslo.limit entry point.
Closes-Bug: #1962406
Change-Id: I176a7b2b8e2edb50659783a2139c3cc13e8f2f53
We currently have a public Enforcer interface for getting limits and
calculating usage but it is not yet possible to retrieve only limits
without calculating usage.
While working on unified limits support in nova, we realized we need a
way to get limits only. In nova there are legacy APIs for showing quota
limits and initially we will provide compat by proxying to keystone to
get the limits.
This adds public interfaces for getting limits to Enforcer.
Related to blueprint unified-limits-nova
Change-Id: I22234e0bb6b3a1cecb29a6b99a3afcd02ffdbf5f
This allows a caller to pass None for the project_id if it only wants
it to check the registered limit for a given resource. This is useful
for non-project-scoped resourced where we just want to make sure some
global limit hasn't been exceeded. This would also be relevant for
resources that are created by system-scoped users, such as host
aggregates.
Change-Id: I5fea0143b6a96b5f79bc273961e3e284a260e25e
The calculate_usage interface was added recently to allow consumers
to probe limits and usage without requiring the enforce behavior
workflow. If a limit was passed to it that was not registered in
keystone, get_project_limits() would raise a ProjectOverLimit
exception itself to abort the process immediately, providing the
"unregistered means zero" behavior. This works fine for the enforce
workflow, but not the calculate one.
This changes get_project_limits() to just return a zero limit for
a missing one, which will be considered by the enforce workflow in
the same way, keeping the existing behavior. It will merely be
reported by the calculate workflow, which is the desired change.
Change-Id: Iaab1f0d5eb0da9a667267537d86f6c70bc8db51d
This adds caching of resource limits for an Enforcer in order to
improve performance when repeated limit checks are needed. The cache
lasts the lifetime of an Enforcer and is enabled by default. It can be
disabled by passing cache=False when instantiating an Enforcer.
One usage pattern for a caching Enforcer would be to create an Enforcer
per service request so that the caching lives only as long as the
request.
Change-Id: I8e43dceec76aecd2b2ae23a137e56519efe29777
Currently the oslo.config.opts entry_points provided by oslo.limit
doesn't include options for auth plugins, thus the parameters to
define credentials like username, password and etc are not picked up by
oslo-config-generator.
This adds the options for auth plugin options to the entry point so
that the auth parameters are included by the generated config files
and users can easily find the parameters to define the required user
credential.
Note that keystoneauth provides several plugins but this change covers
only password plugins, assuming the password authentication is most
popularly used.
Change-Id: Ib440f58b589076677be9e90dd960cd4459e63746
This adds a fixture that can be used by consuming projects to
simulate a set of limits in keystone, without requiring actual
keystone. Currently, consumers have to mock oslo.limit internals (at
least) in order to do testing.
Change-Id: If72050e90ca8b03e26d128c7bbcef6bbea92b501
In multiple situations, it is necessary to be able to probe the limits
set for a project without actually enforcing. Examples:
1. Exposing a usage API where we want to not only report the current
usage, but the limit as well. Otherwise clients have to do their
own calls to keystone and correlation to get a single integer
limit value, which we should be able to expose for them.
2. When checking quota as part of a long-running process of consuming
an unbounded data stream, we need to be able to determine how much
quota remains so that we can stop the transfer if we exceed the
limit. Without this, we have to periodically call to keystone
during the transfer, which is expensive and could fail.
This patch adds a calculate_usage() method to the Enforcer which
calculates the usage using the enforcement model and returns a
mapping of resource names to namedtuples that contain limit and usage
information.
Change-Id: Ic0632cc5ec52aefb85a04f879651963bfa54dcbe
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I0d2b84e2ed627481118151b36386014ed0930452
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Taking the Nova work as an example, looking to add a basic flat enforcer
that meets Nova's needs.
The user of the Enforce provides a callback. You can see an example of
the callback in the unit tests:
def fake_callback(project_id, resources):
return {
"a": 5,
"b": 10,
"c": 0,
}
In the code where you want to check if you increase the amount of
resources that are being consumed, you can make this call:
enforcer = limit.Enforcer(fake_callback)
enforcer.enforce(project_id, {"a": 5})
The enforce function looks up the limits that apply for the given
project_id, uses the callback to count the current usage. The proposed
usage is then calculated be adding the delta and the current usage
together. This is compared to any limits that apply.
If you later want to check if you have races that mean you are
over your limit, you can do this call:
enforcer.enforce(project_id, {'a': 0})
Summary of key design points:
* single set of logic to enforce limits shared between all projects
that adopt unified limits
* interface should work for both flat and strict-two-level enforcement
* it is assumed that oslo.limit will control which type of enforcement
is being applied
* callback lists all resources that need counting
for the given project_id, in Nova this helps limit
the number of API calls made to placement
* allows to check if proposed additional usage means you are over
your limit, and also double check if the current usages means
you are over quota
* if the code is checking a resource where you do not have a
registered limit, we default to a limit of zero, i.e. no
resources can be created unless you set that registered limit
There will be an appropriate warning logged to help the operator
understand what needs to be setup in keystone.
This builds on various previous prototypes from:
Co-Authored-By: Lance Bragstad<lbragstad@gmail.com>
Co-Authored-By: wangxiyuan <wangxiyuan@huawei.com>
Change-Id: I294a922ea80af673291c991f07a4a203f25c289d
For a given resource find out the current limits for the project.
If there are no per project limits look up the registered limits
that act as a default for when there are no project limits.
Future patches will look at comparing these limits with the current and
proposed resource usage to enforce the limits.
Change-Id: Ia5ced4a46833b194f397ac936b99b0c9573b50d1
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Co-Authored-By: Lance Bragstad <lbragstad@gmail.com>
The enforcer needs to be able to determine which model to use and pass
information to it. This commit sets the model attribute on the
enforcer, adds a method to determine which model to use, and
defines a basic interface for enforcement models to use.
Change-Id: Id03d361b702c2ee0811f37ad23bb2b9a3171b1f7
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Co-Authored-By: Lance Bragstad <lbragstad@gmail.com>
Co-Authored-By: John Garbutt <john.garbutt@stackhpc.com>
This commit pulls the ksa-specific logic from a previous attempt:
Ibc35360dd4a952bbe63ee68ddafd5def0608949d
Subsequent patches will use this connection to query unified limit
information out of keystone.
Change-Id: Iae45df091e98904fd6a30db9923b141279fb2cc0
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
filter_resources doesn't make as much sense as resource_filters.
filter_resources almost sounds like a boolean, when the argument is
designed to contain a list of strings that represent resources to
filter.
Change-Id: Idc787052ac41744075d4a4860d06a3529fd271bb
The `enforce()` method is going to be the main entry point into
oslo.limit for services enforcing usage against unified limits.
Future patches will add methods for collecting information from
keystone, resolving callback functions for project resource usage, and
implement different enforcement models (flat, strict-two-level).
Change-Id: I3cca109213e6d3fad91160ebb632d15499690093
Originally, we wanted oslo.limit's Enforcer object to be a context
manager to make it easier to abstract logic to protect against
resource race conditions. Since verification logic isn't going to be
implemented right away, we can just focus on a single method for
enforcement instead, making it easier to adopt.
Subsequent patches are going to introduce a new enforcement interface
for oslo.limit.
Change-Id: I3a1e58cb18a498bca67f36c76d8e8c8c4fb80083
We want the initial implementation of oslo.limit to be as simple as
possible. Originally, we thought we were going to implement resource
verification, but that's something we can do later.
Resource verification is meant to protect against race conditions when
two clients claim the same amount of resources and the project is over
limit. Unfortunately, having services implement resource verification
requires them to use either a context manager or make multiple calls
to enforce limits and sprinkling those calls around existing business
logic. Since we don't know which approach we want to use to implement
this, let's just focus on enforcing limits and come back to resource
verification when we get more feedback from developers incorporating
this into their services.
Change-Id: I50f52189862bc3d109654e158cba381b156e70b6
The concept of having project claims and tacking them on to an
Enforcer object is too heavy-handed for what we need right now.
Instead, we can just use a dictionary with resource names as the key
and their respective usage/request limits as values.
Future patches will remove the context manager functionality from
oslo.limit, too. We can revisit the concept of a context manager to
implement additional functionality for verification and race
conditions when we have a for sure need for it.
The goal of this patch is to make the interface as simple as possible
so that it is easier for projects to adopt it.
Note: we're able to make these changes because we haven't released
oslo.limit, yet. We can make breaking changes up to version 1.0, so we
can easily come back to this approach up to that point.
Change-Id: I3f04ffc6d27208500b8e6409c3c30e6cb3147169
Added opts file
added [oslo_limit] group.
keystoneauth options and "endpoint_id" are in this group.
Change-Id: I31e8f7c9f97b7175a7580f720383228e6f355f9a
This commit builds the foundation for a context manager that can be
used for usage enforcement.
Subsequent patches will incorporate keystoneauth and perform calls
to keystone to verify project limits and registered limits,
incorporating them into the usage check.
Change-Id: I7c78c1979444a2a83189e2035a4ab72890abc5ab
When calculating usage of a resource against a project, there is some
common information that we need to know as an enforcement library.
This can be referred to as a "claim", which consists of the name of
the resource being claimed, the project claiming the resources in the
request, and the quantity being claimed.
This commit adds some basic plumbing and tests that introduce this
object. Subsequent patches will introduce a context manager that
performs usage calculation using ProjecClaim objects.
Change-Id: Iacfa39535c58a7c6b49ba01c5ebf6bbb5f3fe8d1
Takashi Kajinami