This is the first in a series of commits to add support for codespell. This is continuning the process completed in ironic-python-agent.
Future Commits will add a Tox Target, CI support and potentially a git-blame-ignore-revs file if their are lots of spelling mistakes that could clutter git blame.
Change-Id: Id328ff64c352e85b58181e9d9e35973a8706ab7a
pytz will be removed from RHEL/CentOS 10 because of the built-in
zoneinfo[1].
Because the current usage of pytz can be very easily replaced, this
removes the dependency on pytz.
[1] https://issues.redhat.com/browse/RHEL-219
Change-Id: Ia72c528eadeccf6075894ff58477fecade65ad71
After removing the iSCSI deploy and changing ISO parsing code to use
a corresponding library, Ironic no longer executes any commands as root
and it should stay this way.
Change-Id: I47d2bab9b94345fbcf89a2a80028853050a041ea
When parsing redfish driver info wrap IPv6 address in brackets
before appending default scheme/authority.
Updated common.utils.wrap_ipv6() to ignore ValueError, e.g
simply return the string if ip is not an ipv6 address string.
Related: RHBZ#2239356
Closes-Bug: #2036454
Change-Id: Icefd96d6873474b4cfb7fbf3d8337cd42fd63ca6
Disables internal heartbeat mechanism when ironic has been
configured to utilize a SQLite database backend.
This is done to lessen the possibility of a
"database is locked" error, which can occur when two
distinct threads attempt to write to the database
at the same time with open writers.
The process keepalive heartbeat process was identified as
a major source of these write operations as it was writing
every ten seconds by default, which would also collide with
periodic tasks.
Change-Id: I7b6d7a78ba2910f22673ad8e72e255f321d3fdff
It relied on mocking tenacity.retry, but it's executed on class
initialization. Depending on the ordering, it may do nothing or
it may replace ImageService.call with a mock.
Instead, add a new tenacity helper that loads an option in runtime.
As a nice side effect, [glance]num_retries is now mutable.
Change-Id: I2e02231d294997e824db77c998ef8d352fa69075
Currently when using SNMPv3, iRMC driver does not use SNMPv3
authentication parameters so the SNMPv3 authentication will
always fail. And iRMC cannot recognize FIPS mode, so when FIPS mode
is enabled, iRMC driver could still use non-FIPS-compliant algorithms.
This commit changes iRMC driver to require and use SNMPv3
authentication parameters when 'irmc_snmp_version' is set to v3 and
also makes iRMC driver to force 'irmc_snmp_version' to v3,
'irmc_snmp_auth_proto' to SHA and 'irmc_snmp_priv_proto' to AES
when FIPS mode is enabled, because currently among the algorithms
supported by iRMC, only SHA and AES are FIPS compliant.
Change-Id: Id6f8996e4d103f849325f54fe0619b4acb43453a
Story: 2010085
Task: 45590
In a few places in the codebase, "insufficient" is misspelled as
"insufficent," which includes function names and exception class names.
This can be inconvenient for writing and debugging code, in which case
one would raise an exception/call a function and get an error that is
resolved by intentionally misspelling the function call.
The changes made here are mostly to the names of exceptions and
functions but also include some other instances of this misspelling
in docstrings, policy descriptions, etc. There were also some strings
describing policies in ironic/common/policy.py that were missing
spaces, which were also fixed.
Story: 2010089
Task: 45604
Change-Id: I7b65c449d5d30ca30f537a95a3ffd365492e0274
This change ensures all files written for pxe boot have
permissions determined by the [pxe]file_permission config option.
Change-Id: I1bc24e3871bae3ce070e7abe85fc4c48e844c317
This is useful when some nodes need the "agent" power interface, while
the others can be deployed normally.
Change-Id: Ief7df40c83ef03d0ec5ae92d09ceffd39d3c12a3
* Avoid logging large keys, such as system_logs
* Log request params when sending a request
* Sanitizing requests and responses before logging
Change-Id: If298a98b0807e69a3194e48e42d897e8c8414e7e
Bandit has started to fail on master.
>> Issue: [B701:jinja2_autoescape_false] Using jinja2 templates with autoescape=False is dangerous and can lead to XSS. Ensure autoescape=True or use the select_autoescape function to mitigate XSS vulnerabilities.
Severity: High Confidence: Medium
Location: ironic/common/utils.py:491
More Info: https://bandit.readthedocs.io/en/latest/plugins/b701_jinja2_autoescape_false.html
489 # NOTE(pas-ha) not using default_for_string=False as we set the name
490 # of the template above for strings too.
491 env = jinja2.Environment(
492 loader=loader,
493 autoescape=jinja2.select_autoescape(),
494 undefined=jinja2.StrictUndefined if strict else jinja2.Undefined
495 )
It appears that Arun changed this around a little in
https://review.opendev.org/c/openstack/ironic/+/777448/10/ironic/common/utils.py
however this doesn't seem to pass reliably. As such, I'm returning
the notation of the label to the first line as it was before, which
seems to consistently pass bandit checking.
Change-Id: I7f5b7323b108b303b5b77609d5903128d4adca3c
The kickstart template is supplied by the user and it needs
to be validated to make sure it includes all the expected
variables and nothing else.
We validate the template by rendering it using expected
variables. If any of the expected variables are not present
in the template or unexpected variables are defined in the
template we raise InvalidKickstartTemplate exception
Once we render the template into kickstart file we
pass the file to 'ksvalidator' tool if it is present
on the system to validate the rendered kickstart file
for correctness.
'ksvalidator' tool comes from pykickstart libarary and
it is GPLv2 licensed. GPLv2 license is incompatible with
Openstack. So we do not explicitly include the library in
requirements.txt instead rely on it being pre-existing on
the conductor. If the 'ksvalidator' binary is not present
on the system, kickstart validation will be skipped
Change-Id: I3e040bbdbcefb8764c93355d0ba7179e2110b9c6
Removes API translation layer into port/portgroup internal_info
Removes internal logic to support use of VIFs stored in the extra
field, which effectively means all vif binding must now utilize
vif attachment and detachment.
This is a change that we likely should have pushed forward a couple
years back, but I didn't want to break compatability with very old
of ironic or standalone users which were deploying instances using
very old instructions. That being said, it is necessary to remove
the legacy vif support so new access controls can properly wrap
vif API endpoints.
Depends-On: https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/769204
Change-Id: I588b3a64475656542389ff83786189e2dc16d35c
One of the biggest frustrations larger operators have is when they
trigger a massive number of concurrent deployments. As one would
expect, the memory utilization of the conductor goes up. Except,
even with the default number of worker threads, if we're requested
to convert 80 images at the same time, or to perform the write-out
to the remote node at the same time, we will consume a large amount
of system RAM. Or more specifically, qemu-img will consume a large
amount of memory.
If the amount of memory goes too low, the system can trigger
OOMKiller which will slay processes using ram. Ideally, we do not
want this to happen to our conductor process, much less the work
that is being performed, so we need to add some guard rails to help
keep us from entering into situations where we may compromise the
conductor by taking on too much work.
Adds a guard in the conductor to prevent multiple parallel
deployment operations from running the conductor out of memory.
With the defaults, the conductor will attempt to throttle back
automatically and hold worker threads which will slow down the
amount of work also proceeding through the conductor, as we are
in a memory condition where we should be careful about the work.
The defaults allow this to occur for a total of 15 seconds between
re-check of available RAM, for a total number of six retries.
The minimum default is 1024 (MB), as this is the amount of memory
qemu-img allocates when trying to write images. This quite literally
means no additional qemu-img process can spawn until the default
memory situation has resolved itself.
Change-Id: I69db0169c564c5b22abd0cb1b890f409c13b0ac2
For historical reasons we always base64+gzip configdrives, even
when accessing them via a URL. This change allows binary images
to work for the redfish-virtual-media case.
Change-Id: If19144de800b67275e3f8fb297f0a5c4a54b2981
We used netutils earlier on to have a backportable change
however the longer term goal was to replace the change with
using the python native ipaddress module directly.
For the cases where we can change IP version type compares,
we change them with this change.
Note: other uses of netutils still exist, and we should
eventually see if we can phase them out, however the remaining
uses are around MAC address validations.
Change-Id: I44336423194eed99f026c44b6390030a94ed0522
Some unused HTTP param to arg parsing has not been implemented to
reduce code complexity. This includes the following types:
- DictType
- complex types
Asserts are added to confirm these param types are not used in ironic
currently, and to prevent them being used in future development.
Story: 1651346
Task: 10551
Change-Id: Idfcf99216f10e8928fe4ba6202a7d69bfa916459
this breaks pxe config rendering when e.g. quotes are needed (like for values
with spaces), replacing them with HTML escape codes.
Instead use smart autoescape which by default is only enabled for
htm(l) and xml templates.
Not specifically setting override for strings as we set the template
name for string templates anyway, and matching goes by that name.
https://jinja.palletsprojects.com/en/2.11.x/api/#autoescaping
Change-Id: I27e63557d4bcd81d583c55315029425bec03fd98
Story: 2005791
Task: 39269
While trivial, it's used in several places and will likely be used
more as we work on IPv6 support.
Change-Id: I10933b8dbadbba87ef6bfa0af767cf1184c072d9
On systems in FIPS mode MD5 is not available, just use SHA256.
NOTE: I'm not sure if this changes makes ironic fully operational
in FIPS mode, merely fixing an immediate issue.
Story: 2007306
Task: 38792
Change-Id: I9a9e0286c88ed5a1a05e405d251b46a4708dd556
Since we've dropped support for Python 2.7, it's time to look at
the bright future that Python 3.x will bring and stop forcing
compatibility with older versions.
This patch removes the six library from requirements, not
looking back.
Change-Id: Ib546f16965475c32b2f8caabd560e2c7d382ac5a
This change the required base driver interface additions and inspector
interface changes to support in-band inspection driven by ironic.
Change-Id: Ibf9a80d0f72d5f128bf46ddca4cb9762c9a8191b
Story: #1528920
Task: #10404
Adds bandit configuration template and exclude some of
tests that we don't want to fix for the moment.
Keeping job unvoted so that we can keep an eye on possible
issues while not breaking gate.
Change-Id: Ic577cad3b61421c04023ea887236992acb19f77c
Story: 2005791
Task: 33518
Adds the fields and bumps the objects versions. Excludes the field from
the node API for now.
Also adds the conductor_group config option, and populates the field in
the conductors table.
Also fixes a fundamentally broken test in ironic.tests.unit.db.test_api.
Change-Id: Ice2f90f7739b2927712ed45c969865136a216bd6
Story: 2001795
Task: 22640
Task: 22642
This is a followup change to [1] moving
parse_instance_info_capabilities() from boot_mode_utils.py
to common/utils.py
1. https://review.openstack.org/#/c/569382/
Change-Id: I9ad8896f00b95620bad25dcb15f1f932cd0379fa
Story: 1734131
Task: 10640
This commit will fix authentication in boot from volume (BFV) feature,
which use the volumes from Cinder for Baremetal via Ironic.
Each volume will need pair of account for authentication when perform
sanhook into SAN device via iPXE. And sandboot from drive 0x80 (default)
also need pair accounts same with the iscsi sanhook on drive 0x80 with
multi volumes has supported.
NOTE:
- We could add more than two volumes into iSCSI Boot Firmware Table(iBFT)
- Due to Linux does not support an iBFT that has more than two volumes,
thus BFV only support for add one etra volume. If over two volume in iBFT
then machine will raise "iBFT error: Control header is invalid!".
- Our code-base already for more than two volumes in iBFT, If Linux kernel
bugs[1] is fixing this issue then we can use BFV with more than two volumes.
Tested successfully on Fujitsu Baremetal Server TX2540 M1.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/acpi/actbl1.h#n1567
Co-authored-By: Hoang Trung Hieu <hieuht@vn.fujitsu.com>
Co-authored-By: Dao Cong Tien <tiendc@vn.fujitsu.com>
Change-Id: I98f658cced8491872d39adbd8e0a1a643dd24868
Story: #2001824
Task: #12573
Also a few related errors based on some earlier investigation
may have been pulled in along the lines of E305.
Story: #2001985
Change-Id: Ifb2d3b481202fbd8cbb472e02de0f14f4d0809fd
For API versions >= 1.28, Port & portgroup's .extra['vif_port_id'] was
deprecated in Ocata. Before we can remove support for this, we need to
copy that information to the object's internal_info['tenant_vif_port_id'].
This copy/migration is done at the API layer when the user specifies the
.extra[] value, as well as when the 'ironic db-sync online_data-migrations'
is run.
In order to know whether the ports and port groups have been migrated,
their IronicObject versions are incremented.
This also fixes it so that for API versions < 1.28, the deprecation
warning is not shown, since we still need to support extra['vif_port_id']
in this case.
When a port or portgroup's .extra['vif_port_id'] is removed via a
PATCH API request, that VIF is removed from that object's internal_info.
Change-Id: I69468c935e68dd9d37a474c318c3ceb9cdfc5868
Partial-Bug: 1722850
osl.utils provides method is_valid_port to check port, we can
leverage it to make code more clear. Note that the code here was
incorrect in accepting 1-65535, and this change fixes it to also
include 0.
Change-Id: I60cb36a042fd808edca66b07d7248213debd4dff
SSH drivers are being unsupported for about a year now. All current
stable branches of ironic are officially supporting IPMI-capable HW
simulation via virtualbmc.
All ironic-related gate jobs have already been switched
to not use or enable those drivers.
This patch finally removes SSH-based power and managemtnt driver interfaces
and all classic drivers using those from ironic code and documentation.
Related exceptions and `ssh_connect` function, together with dependency
on `paramiko` package are removed as well.
Change-Id: Ieda7249b9cd78e3be1eff37804996295fc8d3969
Closes-Bug: #1570301
Depends-On: I9b60c9fa24652e9e64e787cd4e5b0152f51e7a28
Removes calls to _LE, _LW, and _LI used for translating
logs. There were no _LC calls in this directory.
Change-Id: If74114cb5b28bac89ea66ee068799d0a3cea38df
Partial-bug: #1674374
the reference file must be opened in binary mode,
otherwise under Py3 it fails to update checksum with exception
`Unicode must be encoded before hashing`.
Change-Id: I3162fe9951750f5a41ef403b9a10ed68c0ba958a
With this patch port groups are activated in Ironic.
When attaching a VIF to a node, it is attached to the first
free port group. If there are no free port groups, the first
available port (pxe_enabled has higher priority) is used
instead.
Related-Bug: #1618754
Related-Bug: #1582188
Co-Authored-By: Vladyslav Drok <vdrok@mirantis.com>
Change-Id: I0dca2c2d98184e370c08c3e05aa3edadead869af
This patch adds the REST APIs for the virtual network interface API in
order to abstract the task of assigning logical network interfaces to
physical network interfaces.
Since Newton Ironic provides an interface for pluggable network
implementations. Different network implementations may want to handle
how logical to physical network interface assignment happens. To do this
the new API calls into new functions on the network implementation
loaded for the specified node.
This is part 3 of 3, and adds the node vif subcontroller to expose the
/nodes/<ident>/vifs REST API endpoint. API version is bumped to 1.28.
Co-Authored-By: Vasyl Saienko (vsaienko@mirantis.com)
Change-Id: I70f1166a15a26f392734e21d6bc30a03da4e5486
Partial-Bug: #1582188
This will not affect those using custom out-of-tree boot scripts,
as such files will be rendered as-is.
This patch also moves template rendering to a common place and sets
create_pxe_config to use it.
Change-Id: Ie5c94fee26054a209d35a25a99f3e9f57bd39d14
Partial-Bug: #1526275
Use hacking 0.12.0
Use the new checks that are available:
[H106] Don’t put vim configuration in source files.
[H203] Use assertIs(Not)None to check for None.
[H904] Delay string interpolations at logging calls.
Fix code so tests pass.
Change-Id: I902e999687b066800e18fafd091571bf718b15f4
Depends-On: I2aa44b62f900d4dfd67701b01eadd0523fbfaf07