The generate path does not contain the node UUID, causing conflicts.
Also make sure to always clean up any existing files first.
Change-Id: I30f948d64e7b87f33841dc22828db60338a62dd8
Currently, if the image download fails, there are no traces of the
error. This change adds logging and populates last_error.
Change-Id: I73ea2f94fb910daf21a5d4f52d6839aac3bad579
This is the first in a series of commits to add support for codespell. This is continuning the process completed in ironic-python-agent.
Future Commits will add a Tox Target, CI support and potentially a git-blame-ignore-revs file if their are lots of spelling mistakes that could clutter git blame.
Change-Id: Id328ff64c352e85b58181e9d9e35973a8706ab7a
This is a MVP of auto-discovery with no extra customization and no new
auto_discovered field from the spec.
Change-Id: I1528096aa08da6af4ac3c45b71d00e86947ed556
I've seen a situation where heartbeats managed to completely saturate
the conductor workers, so that no API requests could come through that
required interaction with the conductor (i.e. everything other than
reads). Add periodic tasks for a large (thousands) number of nodes, and
you get a completely locked up Ironic.
This change reserves 5% (configurable) of the threads for API requests.
This is done by splitting one executor into two, of which the latter is
only used by normal _spawn_worker calls and only when the former is
exhausted. This allows an operator to apply a remediation, e.g. abort
some deployments or outright power off some nodes.
Partial-Bug: #2038438
Change-Id: Iacc62d33ffccfc11694167ee2a7bc6aad82c1f2f
This patch allows to attach or detach a generic image as
virtual media device after a node has been provisioned.
Closes-Bug: #2033288
Change-Id: I97b68047d769f6fb686c53e89084b5874e02b8c7
In the agent token mechanism, restrictions exist when a an agent
token can be generated, and unfortunately this has to be done on
the conductor side involving a lock and a task because we need to
save the state of the node.
As such, we were in a situation where we were waiting on DB node
locking, which would prevent the agent from getting a node, and
potentially causing the lookup operation to fail, eventually.
We now quickly return NodeLocked which shouldn't cause the agent
any issues, although we need to improve error handling there as
well.
Change-Id: Ice335eed82b936753be99eedb16ceccf8a9a86a8
A huge list of initial work for service steps
* Adds service_step verb
* Adds service_step db/object/API field on the node object for the
status.
* Increments the API version to 1.87 for both changes.
* Increments the RPC API version to 1.57.
* Adds initial testing to facilitate ensurance that supplied steps
are passed through and executed upon.
Does not:
* Have tests for starting the agent ramdisk, although this is
relatively boiler plate.
* Have a collection of pre-decorated steps available for immediate
consumption.
Change-Id: I5b9dd928f24dff7877a4ab8dc7b743058cace994
FirmwareInterface base
New Config options [default]
- enabled_firmware_interfaces
- default_firmware_interface
New FirmwareInterface base with update method
Implementations of FirmwareInterface
- FakeFirmware (fake)
- NoFirmware (no-firmware)
New entrypoint ironic.hardware.interfaces.firmware
* fake and no-firmware
Api Controllers
- Updated: driver/node/utils/versions
- Created: firmware
Unit tests
api-ref for Node Firmware
Fake and Noop implementation for FirmwareInterface
Change-Id: Ib3b9cb22099819f97d5eab1e3f1b670cb91cbb25
* Updates API version to 1.85 to permit an ``unhold`` verb
* Adds the ``deploy hold`` and ``clean hold`` provision states
to the internal state machine.
* Adds on documentation on steps to help provide greater clarity
to Ironic's users on how to utilize steps. It should be noted
this documentation also includes the power state reserved step
names from the DPU functionality patch.
* Fixes the state machine diagram. Changes type to PNG as SVG
rendering is broken due to python libraries utilized for SVG
generation which do not work on more recent Python versions.
Change-Id: I34f58f4e77e7757b89247fd64f5fcde26f679453
This change creates all necessary parts to processing inspection data:
* New API /v1/continue_inspection
Depending on the API version, either behaves like the inspector's API
or (new version) adds the lookup functionality on top.
The lookup process is migrated from ironic-inspector with minor changes.
It takes MAC addresses, BMC addresses and (optionally) a node UUID and
tries to find a single node in INSPECTWAIT state that satisfies all
of these. Any failure results in HTTP 404.
To make lookup faster, the resolved BMC addresses are cached in advance.
* New RPC continue_inspection
Essentially, checks the provision state again and delegates to the
inspect interface.
* New inspect interface call continue_inspection
The base version does nothing. Since we don't yet have in-band
inspection in Ironic proper, the only actual implementation is added
to the existing "inspector" interface that works by doing a call
to ironic-inspector.
Story: #2010275
Task: #46208
Change-Id: Ia3f5bb9d1845d6b8fab30232a72b5a360a5a56d2
Prior to this fix, we have been unable to run the Metal3 CI job
with SQLAlchemy's internal autocommit setting enabled. However
that setting is deprecated and needs to be removed.
Investigating our DB queries and request patterns, we were able
to identify some queries which generally resulted in the
underlying task and lock being held longer because the output
was not actually returned, which is something we've generally
had to fix in some places previously. Doing some of these
changes did drastically reduce the number of errors encountered
with the Metal3 CI job, however it did not eliminate them
entirely.
Further investigation, we were able to determine that the underlying
issue we were encountering was when we had an external semi-random
reader, such as Metal3 polling endpoints, we could reach a situation
where we would be blocked from updating the database as to open a
write lock, we need the active readers not to be interacting with
the database, and with a random reader of sorts, the only realistic
option we have is to enable the Write Ahead Log[0]. We didn't have
to do this with SQLAlchemy previously because autocommit behavior
hid the complexities from us, but in order to move to SQLAlchemy
2.0, we do need to remove autocommit.
Additionally, adds two unit tests for get_node_with_token rpc
method, which apparently we missed or lost somewhere along the
way. Also, adds notes to two Database interactions to suggest
we look at them in the future as they may not be the most
efficient path forward.
[0]: https://www.sqlite.org/wal.html
Change-Id: Iebcc15fe202910b942b58fc004d077740ec61912
Fix the warnings that oslo.versionedobjects has been emitting for years
now.
Change-Id: I53bd78d8b70f276d2ea8569f0ab1e7ce04f52fea
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
... to reduce the already frightening size of ironic.conductor.manager
and make space for more inspection additions.
While here, fix up log messages for clarity and brevity.
Change-Id: I5196d58016ae094f17e0aad187a11d9cceaab04b
When cleaning fails, we power off the node, unless it has been running
a clean step already. This happens when aborting cleaning or on a boot
failure. This change makes sure that the power action does not wipe
the last_error field, resulting in a node with provision_state=CLEANFAIL
and last_error=None for several seconds. I've hit this in Metal3.
Also when aborting cleaning, make sure last_error is set during
the transition to CLEANFAIL, not when the clean up thread starts
running.
While here, make sure to log the current step in all cases, not only
when aborting a non-abortable step.
Change-Id: Id21dd7eb44dad149661ebe2d75a9b030aa70526f
Story: #2010603
Task: #47476
This change adds the capability for the ironic-conductor
and standalone service process to transmit timer and counter
metrics to the message bus notifier which may be consumed by
a ceilometer, ironic-prometheus-exporter, or other consumer of
metrics event data on to the message bus.
This functionality is not presently supported on dedicated API
services such as those running as an ``ironic-api`` application
process, or Ironic WSGI application. This is due to the lack of
an internal trigger mechanism to transmit the data in a metrics
update to the message bus and/or notifier plugin.
This change requires ironic-lib 5.4.0 to collect and ship metrics via
the message bus.
Depends-On: https://review.opendev.org/c/openstack/ironic-lib/+/865311
Change-Id: If6941f970241a22d96e06d88365f76edc4683364
The dynamically allocated console port for a node is saved
into database and reused on subsequent console operations.
In certain code path the port record cann't be trusted and
we should do a re-allocation.
This patch fixes the issue by ignores previous allocation
record. The extra cleanup in the takeover is not required
anymore and removed as well.
Change-Id: I1a07ea9b30a2c760af7a6a4e39f3ff227df28fff
Story: 2010489
Task: 47061
Follow-up to Ie174904420691be64ce6ca10bca3231f45a5bc58
which enables storage of inventory in Swift, but does not delete
the Swift entry when the node whose inventory is stored is deleted
Story: 2010275
Task: 46204
Change-Id: I74b19f7a42c1326d7ec04e6320176e81639ebfb4
One of the major changes in SQLAlchemy 2.0 is the removal
of autocommit support. It turns out Ironic was using this quite
aggressively without even really being aware of it.
* Moved the declaritive_base to ORM, as noted in the SQLAlchemy 2.0
changes[0].
* Console testing caused us to become aware of issues around locking
where session synchronization, when autocommit was enabled, was
defaulted to False. The result of this is that you could have two
sessions have different results, which could results on different
threads, and where one could still attempt to lock based upon prior
information. Inherently, while this basically worked, it was
also sort of broken behavior. This resulted in locking being
rewritten to use the style mandated in SQLAlchemy 2.0 migration
documentation. This ultimately is due to locking, which is *heavily*
relied upon in Ironic, and in unit testing with sqlite, there are
no transactions, which means we can get some data inconsistency
in unit testing as well if we're reliant upon the database to
precisely and exactly return what we committed.[1]
* Begins changing the query.one()/query.all() style to use explicit
select statements as part of the new style mandated for migration
to SQLAlchemy 2.0.
* Instead of using field label strings for joined queries, use the
object format, which makes much more sense now, and is part of
the items required for eventual migration to 2.0.
* DB queries involving Traits are now loaded using SelectInLoad
as opposed to Joins. The now deprecated ORM queries were quietly
and silently de-duplicating rows and providing consistent sets
from the resulting joined table responses, however putting much
higher CPU load on the processing of results on the client.
Prior performance testing has informed us this should be a minimal
overhead impact, however these queries should no longer be in
transactions with the Database Servers which should offset the
shift in load pattern. The reason we cannot continue to deduplicate
locally in our code is because we carry Dict data sets which cannot
be hashed for deduplication. Most projects have handled this by
treating them as Text and then converting, but without a massive
rewrite, this seems to be the viable middle ground.
* Adds an explict mapping for traits and tags on the Node object
to point directly to the NodeTrait and NodeTag classes. This
superceeds the prior usage of a backref to make the association.
* Splits SQLAlchemy class model Node into Node and NodeBase, which
allows for high performance queries to skip querying for ``tags``
and ``traits``. Otherwise with the afrormentioned lookups would
always execute as they are now properties as well on the Node
class. This more common of a SQLAlchemy model, but Ironic's model
has been a bit more rigid to date.
* Adds a ``start_consoles`` and ``start_allocations`` option to the
conductor ``init_host`` method. This allows unit tests to be
executed and launched with the service context, while *not* also
creating race conditions which resulted in failed tests.
* The db API ``_paginate_query`` wrapper now contains additional
logic to handle traditional ORM query responses and the newer style
of unified query responses. Due to differences in queries and handling,
which also was part of the driver for the creation of ``NodeBase``,
as SQLAlchemy will only create an object if a base object is referenced.
Also, by default, everything returned is a tuple in 1.4 with the
unified interface.
* Also modified one unit test which counted time.sleep calls, which is
a known pattern which can create failures which are ultimately noise.
Ultimately, I have labelled the remaining places which SQLAlchemy
warnings are raised at for deprecation/removal of functionality,
which needs to be addressed.
[0] https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
[1] https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#transaction-isolation-level-autocommit
Change-Id: Ie0f4b8a814eaef1e852088d12d33ce1eab408e23
Provide the ability to limit resource intensive or potentially
wide scale operations which could be a symptom of a highly
distructive and unplanned operation in progress.
The idea behind this change is to help guard the overall deployment
to prevent an overall resource exhaustion situation, or prevent an
attacker with valid credentials from putting an entire deployment
into a potentially disasterous cleaning situation since ironic only
other wise limits concurrency based upon running tasks by conductor.
Story: 2010007
Task: 45140
Change-Id: I642452cd480e7674ff720b65ca32bce59a4a834a
Instance network boot (not to be confused with ramdisk, iSCSI or
anaconda deploy methods) is insecure, underused and difficult to
maintain. This change removes a lot of related code from Ironic.
The so called "netboot fallback" is still supported for legacy boot when
boot device management is not available or is unreliable.
Change-Id: Ia8510e4acac6dec0a1e4f5cb0e07008548a00c52
We added an additional check, and it turns out the test was passing
when no local webserver was present because it was getting checked,
but things would break when an actual webserver existed which returned
a redirect
Change-Id: If8d29ba3676cf8e11a1010890c90bb67b0d60cb3
Story: 2010181
Task: 45862
Ironic has a lot of logic built up around use of images for filesystems,
however several recent additions, such as the ``ramdisk`` and ``anaconda``
deployment interfaces have started to break this mold.
In working with some operators attempting to utilzie the anaconda
deployment interface outside the context of full OpenStack, we discovered
some issues which needed to be make simpler to help remove the need to
route around data validation checks for things that are not required.
Standalong users also have the ability to point to a URL with anaconda,
where as Operators using OpenStack can only do so with customized kickstart
files. While this is okay, the disparity in configuraiton checking
was also creating additional issues.
In this, we discovered we were not really graceful with redirects,
so we're now a little more graceful with them.
Story: 2009939
Story: 2009940
Task: 44834
Task: 44833
Change-Id: I8b0a50751014c6093faa26094d9f99e173dcdd38
Adds a configuration option and capability to automatically
record the lessee for a deployment based upon the original
auth_token information provided in the request context.
Additional token information is now shared through the context
which is extended in the same fashion as most other projects
saving request token information to their RequestContext,
instead of triggering excess API calls in the background to
Keystone to try and figure out requestor's information.
Change-Id: I42a2ceb9d2e7dfdc575eb37ed773a1bc682cec23
* Do not mention "deploy driver", it's not a thing.
* Be careful with the pattern "Error: %s" or "Reason: %s". It is good
for long introductory sentences, but looks poor for shorter ones and
becomes really problematic when several instances are concatenated.
This change updates deploy_utils, agent code and conductor modules.
Change-Id: Ie1efea02b5f1a174e9ef8c5253ce9754a60b4c56
Using kernel/ramdisk makes no sense with local boot, we need a better
way. We already have an internal image_type instance parameter, let's
make it public.
Glance support will be added in the next patch.
Change-Id: I4ce5f7a2317d952f976194d2022328f4afbb0258
For backward compatibility reasons, they're off by default.
Fix all unit tests that rely on dangling foreign keys.
Change-Id: I54a45fab1aefc03b2252c655e14b375dea3d9f12
We validate the boot interface during adoption because of:
a) potential rebuilding,
b) non-local boot.
Rebuild proved a rarely used feature, and local boot is the default
nowadays, so it makes less sense to unconditionally validate the boot
interface during adoption. We will run the validation anyway the next
time we need to do something with booting.
Similarly, do not record is_whole_disk_image None if it cannot be
reliably determined.
Change-Id: I95252aea808c48ea2d94569449c871f0d483caaa
Fake nodes have power_state None and the power interface returns None as
well. Currently we make an update every loop, even though the values are
the same. This change fixes it.
Change-Id: I5227c058f3f6e1583b54a1cbc7edc6d42e20ae53
Making updates to driver_internal_info can result in hard to read code
due the requirement to assign the whole driver_internal_info back to
the node to trigger the expected update operation. This change
replaces driver_internal_info update operations with a new
methods:
- set_driver_internal_info
- del_driver_internal_info
- timestamp_driver_internal_info
This change defines the functions and moves core conductor logic to
use them. Subsequent changes in this series will move drivers to use
the new functions.
Change-Id: Ib8917c3c674e77cd3aba6a1e73c65162e3ee1141
We have a very common pattern of periodic tasks that use iter_nodes
to fetch some nodes, check them, create a task and conductor some
operation. This change introduces a helper decorator for that and
migrates the drivers to it.
I'm intentionally leaving unit tests intact to demonstrate that the
new decorator works exactly the same way (modulo cosmetic changes)
as the previous hand-written code.
Change-Id: Ifed4a457275d9451cc412dc80f3c09df72f50492
Story: #2009203
Task: #43522
* Adds periodic task to purge node_history entries based upon
provided configuration.
* Adds recording of node history entries for errors in the
core conductor code.
* Also changes the rescue abort behavior to remove the notice
from being recorded as an error, as this is a likely bug in
behavior for any process or service evaluating the node
last_error field.
* Makes use of a semi-free form event_type field to help
provide some additional context into what is going on and
why. For example if deployments are repeatedly failing,
then perhaps it is a configuration issue, as opposed to
a general failure. If a conductor has no resources, then
the failure, in theory would point back to the conductor
itself.
Story: 2002980
Task: 42960
Change-Id: Ibfa8ac4878cacd98a43dd4424f6d53021ad91166
Splitting code specific to node verification from manager.py into
verify.py (as well as test_verify.py for tests). This is done in
preparation for adding support for verify steps.
Story: 2009025
Task: 43137
Change-Id: I22a9bd7ceac3dfd65f20e52cbacff4b9d3998c64
* add fields to Node object
* expose them at endpoint `/v1/nodes/{node_ident}/states`
* update states on powersync / entering managed state.
* tests
* update api endpoint info in api-ref
Story: 2008567
Task: 41709
Change-Id: Iddd1421a6fa37d69da56658a2fefa5bc8cfd15e4
Generic fields, such as deploy_{kernel,ramdisk,iso) and bootloader
should not have vendor prefixes. This patch removes them from
the iLO boot interfaces with deprecation.
Also clean up the code a bit for less repetition.
Change-Id: Ib23203fa01837ee96f8a38ab50fa8252334c8cb8
Story: #2008880
Task: #42430
Currently handling of kernel_append_params is very inconsistent. This
change applies a straightforward process:
1. instance_info[kernel_append_params]
2. driver_info[kernel_append_params]
3. [pxe]kernel_append_params (renamed from pxe_append_params).
Also adds a helper for subsequent fixes in other drivers.
Change-Id: I79bcf4d8ef1f0f55a82e0991dd5bb1685b3f7957
Story: #2008902
Task: #42469
The iSCSI deploy was very easy to start with, but it has since become
apparently that it suffers from scalability and maintenance issues.
It was deprecated in the Victoria cycle and can now be removed.
Hide the guide to upgrade to hardware types since it's very outdated.
I had to remove the iBMC diagram since my SVG-fu is not enough to fix it.
Change-Id: I2cd6bf7b27fe0be2c08104b0cc37654b506b2e62
agent_status is used by anaconda ramdisk to inform the
conductor about state of the deployment. Valid agent
states are 'start', 'end' and 'error'. The agent_status_message
is used to describe the why the agent_status is set to a
particular state. Use of these parameters require API
version 1.72 or greater.
When anaconda finishes deployment the agent_status is
set to 'end'. When anaconda ramdisk is unable to deploy
the OS for some reason the agent_status is set to 'error'.
PXEAnacondaDeploy is implemented to handle the 'anaconda'
deploy interface. PXEAnacondaDeploy ties to together pieces
needed to deploy a node using anaconda ramdisk.
Co-Authored-By: Jay Faulkner <jay@jvf.cc>
Change-Id: Ieb452149730510b001c4712bbb2e0f28acfc3c2e
Dmitry Tantsur