autopep8 is a code formating tool that makes python code pep8
compliant without changing everything. Unlike black it will
not radically change all code and the primary change to the
existing codebase is adding a new line after class level doc strings.
This change adds a new tox autopep8 env to manually run it on your
code before you submit a patch, it also adds autopep8 to pre-commit
so if you use pre-commit it will do it for you automatically.
This change runs autopep8 in diff mode with --exit-code in the pep8
tox env so it will fail if autopep8 would modify your code if run
in in-place mode. This allows use to gate on autopep8 not modifying
patches that are submited. This will ensure authorship of patches is
maintianed.
The intent of this change is to save the large amount of time we spend
on ensuring style guidlines are followed automatically to make it
simpler for both new and old contibutors to work on nova and save
time and effort for all involved.
Change-Id: Idd618d634cc70ae8d58fab32f322e75bfabefb9d
Once again, do what we did for '_LE' and '_LW' and remove the final
remnants of the log translation effort.
Change-Id: Id6cf7a9bfbe69d6d3e65303e62403d1db9188a84
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Now that the FilterScheduler is using a RequestSpec object,
we need to change the filters for backporting the object
into the legacy dicts before changing each of them.
Release Notes will be updated to mention that custom filters
need to be modified to either use a RequestSpec object or
primitive them into a dict.
Partially-Implements: blueprint request-spec-object-mitaka
UpgradeImpact - see the reno file attached.
Change-Id: I14f18c4507498b1d24a9318fafc97193861ca0b6
As discussed, reservation_id is a compatible layer for EC2 and thus is persisted,
but there are no filters which are using it. Since there are very little benefits
of giving it to the operator (because the operator can find a request by other ways)
and since it would mean we should persist that too in the RequestSpec object,
we prefer to remove that from the log information exposed to the operator.
Change-Id: Ie8f050a1b3653fe1a6d901b3e4020158d63b9b8a
Partially-Implements: blueprint request-spec-object-mitaka
When hosts are filtered for a request, and no host passes all the
filters, operators can only see the last filter run; they have no
visibility into which filter removed which host. This attempts to
improve this situation by tracking the hosts remaining after each
filter is run and the number of hosts removed. If no host is left, to
log that information along with the reservation ID and instance UUID
of the request. Since that can be a lot of information, it will call
LOG.info with just the counts for all but the final filter, which
will contain the last deleted host names; LOG.debug will output the
full record of each filter and the hosts remaining after each step.
This will better enable operators to determine why a particular host
was not selected.
Change-Id: I6173274a03bd8712eb4d63c7a998bb02abdecf4b
Blueprint: no-valid-host-reporting
Change the use of the name 'filter' to 'filter_' so that it doesn't
conflict with the builtin function 'filter'.
Change-Id: Iaa18aa54fbd8578c3b0d3313540393541c002720
Closes-Bug: #1459826
Convert the use of the incubated version of the log module
to the new oslo.log library.
Sync oslo-incubator modules to update their imports as well.
Co-Authored-By: Doug Hellmann <doug@doughellmann.com>
Change-Id: Ic4932e3f58191869c30bd07a010a6e9fdcb2a12c
Right now, filters/weighers are instantiated on every invocation of the
scheduler. This is both time consuming and unnecessary. In cases where
a filter/weigher tries to be smart and store/cache something in between
invocations this actually prohibits that.
This change make base filter/weigher functions take objects instead of
classes and then let schedulers create objects only once and then reuse
them.
This fixes a known bug in trusted_filter that tries to cache things.
Related to blueprint scheduler-optimization
Change-Id: I3174ab7968b51c43c0711033bac5d4bc30938b95
Closes-Bug: #1223450
oslo.i18n uses different marker functions to separate the
translatable messages into different catalogs, which the translation
teams can prioritize translating. For details, please refer to:
http://docs.openstack.org/developer/oslo.i18n/guidelines.html#guidelines-for-use-in-openstack
There were not marker fuctions some places in directory network.
This commit makes changes:
* Add missing marker functions
* Use ',' instead of '%' while adding variables to log messages
Change-Id: Iaebb239ef20a0da3df1e3552baf26f412d0fcdc0
oslo.i18n provides the i18n functions that were provided by
oslo-incubator's gettextutils module. Some tests that were
using internal details of the library were removed.
Change-Id: I44cfd5552e0dd86af21073419d31622f5fdb28e0
Our translation policy
(https://wiki.openstack.org/wiki/LoggingStandards#Log_Translation) calls
for not translating debug level logs. This is to help prioritize log
translation. Furthermore translation has a performance overhead, even if
the log isn't used (since nova doesn't support lazy translation yet).
This is the final patch ensuring that all fines in the nova directory
no longer have translations for debug messages.
Change-Id: Ib10f5826e2238cbb2d4315c48be570eb31b2ea55
If log level is set to INFO,only see a log message about nova-schedule
attempting to build n instances then a log message about setting the
state to error when the scheduler fails to schedule an instance due to
a filter returning 0 hosts.It may be useful to see which filterreturns 0
host for operator.
Close-bug: #1226334
Change-Id: I7564a8ab093895d1f9db357009aac294783a808b
Previous _ was monkey patched into builtins whenever
certain modules were imported. This removes that and
simply imports it when it is needed.
Change-Id: I0af2c6d8a230e94440d655d13cab9107ac20d13c
Currently the filter_all() method of each scheduler filter is run
once for each instance in a request, but for many filters the data
doesn't change during a request.
For example the data used by the AZ filter is pretty static, and
doesn't filter any more hosts on successive runs. However it is
fairly expensive to run on a large system.
Similarly the ServiceGroup information used by the compute filter
is cached as part of the host status, and doesn't need to be
evaluated more than once per request.
Only filters that are based on data that may change within a
request, for example resource consumption, need to be evaluated
for each instance.
This blueprint introduces a new attribute that allows filters to
declare that they only need to be run once per request. The default
behaviour is left so that a filter is evaluated once for each
instance in a request.
The following existing filters are changed to only evaluate once
per request:
AggregateInstanceExtraSpecsFilter
AggregateMultiTenancyIsolation
AggregateTypeAffinityFilter
AllHostsFilter
AvailabilityZoneFilter
ComputeCapabilitiesFilter
ComputeFilter
DifferentHostFilter
GroupAntiAffinityFilter
ImagePropertiesFilter
IsolatedHostsFilter
SameHostFilter
SimpleCIDRAffinityFilter
In addition the function that does the check is defined in the
filter base class in a way that a filter that wants to run
(for example) once for every 10 instances in a request could
overload the function and implement its own behaviour.
Implements: bp once-per-request-filters
Fixes: bug 1189557
Change-Id: Ia668f16414da86441323c58b1bbef5f88c81b90c
The code snippet can not provide the original target. In the
filters module, there is one line code will never be invoked
because of the code logic, so I modified it.
Fixes: bug 1186945
Change-Id: I3f2a662c28ab466b178268cae3502c40a1d22657
This adds filtering and weighing support to the cells scheduler.
Adds the following config options to the 'cells' group:
scheduler_filter_classes -- list of filter classes
scheduler_weight_classes -- list of weight classes
Adds a couple of weighing modules as defaults (which removes the random
cell selection):
ram_by_instance_type: Select cells with the most capacity for the
instance type being requested.
weight_offset: Allows modifying the DB to weight a particular cell
(useful for disabling a cell)
Adds a filter class (TargetCellFilter) that allows specifying a scheduler
hint to direct a build to a particular cell if you're an admin.
DocImpact
Implements blueprint cells-filter-scheduler
Change-Id: I027c5734e3d6134127fcd4dd28b8ff39047416dc
On a large system (>500 hosts) the amount of logging
information if everything is at DEBUG is vasts, due to
the number of hosts checked for each filter
However at the moment some useful information is only
available within the filter's debug entrys (such as
parts of the request_spec). Also instance uuids are
not logged making it hard to search for instance entries.
This set of changes allows it to be used in a very
large install with debug turned off for everything
except nova.filters
Specifically:
- Log.Info the instance_uuids at the start of schedule
- Log.debug the request_spec at the start of schedule
- Log.debug hosts in weighted order
- Log.info which host has been allocated to a specific
instance_uuid
- Log.debug how many hosts are returned by each filter
To get a count from each filter nova.filters had to be
changed to generate a list after each filter rather than
building a recursive generator object. Although this
new approach is less elegant it does provide simple and
useful insight into the behaviour of the filters, and s
the impact on execution time on a system with several
hundred hosts and 10 filters was negligible
Change-Id: Ibbdd14f87b1dfd3cee8bf3cf6388e40b474e530a
This changes the scheduling filters to use the code in nova/loadables.py
to locate filtering classes. This also creates some base functionality
in nova/filters.py to be used by both the host scheduler and the cells
scheduler.
The scheduler_available_filters default has changed to
'nova.scheduler.filters.all_filters' which is better named compared to
the old setting of 'standard_filters'. The old method is still
supported for those that have put it explicitly in their configs, but
it's marked as deprecated.
DocImpact
Change-Id: I84fdeafdba0275ab4b25f8857563bd7b1494bb69
Sean Mooney