Commit Graph

53 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov 8d46ed8842 Allow env.d to contain underscores in physical_skel
At the moment our dynamic_inventory does have assumption that group names
defined in env.d will not contain underscores, except when it's ending
with `_hosts` or `_containers` since inventory script uses split on `_` and taking
the last argument. So in cases when underscore was used elsewhere in the
group name it will result in unexpected behaviour.

Instead of this approach we now use regexp which replaces the last octet separated with underscore with intended suffix to preserve
rest behaviour.

Change-Id: Id9ba56292972b8b52b4786c78684f2d6f289d88a
2024-03-05 09:17:44 +00:00
Dmitriy Rabotyagov d74d038032 Do not use underscores in container names
According to RFC1034 [1], underscores are not legitimate symbols for
hostnames. We have a naming convention that historically uses underscores
in names of containers which leads to a complications in some cases/logic
Moreover, container actual hostnames are always different from their
inventory_hostnames, which might break some logic of external dependencies.

To change this historical issue, all new containers will have different
naming convention, where only dash will be used as a separator.
This will not touch already existing in inventory containers and they will
preserve their names for the time being.

[1] http://www.faqs.org/rfcs/rfc1034.html

Change-Id: Iedebf9935059ecfe8370f2a84ad52516cc93320e
2024-03-05 09:17:37 +00:00
Dmitriy Rabotyagov 7dd23c28a8 Treat dashes/underscores as interchangeable symbols for container names
This patch aims to treat dashes and underscores equally in the existing
inventory.

In case a host already exists in the inventory, but has different separators,
treat underscore and dash symbol as interchangeable.

This is needed for futher coversion of container names to spearated with only dashes,
while not adding new records to the inventory

Change-Id: I63cd14c3353d7a9d7ea4d96155be26697ee4fa40
2024-01-12 12:48:22 +01:00
James Denton 934817b8f9 Stop ignoring hostnames without underscores
The inventory management script was ignoring hostnames without underscores,
which lends its roots to all LXC containers having underscores. In bare
metal deployments, hosts likely won't have underscores, and new containers
might even have hyphens. Group listing now works as expected.

Change-Id: Ied031501bc1da4d1a3c4f9ad101aba355ecd364f
2023-10-11 13:58:11 -05:00
Dmitriy Rabotyagov c8ecc9fa10 Add management_ip option for metal hosts
In cases when SSH and mgmt networks are different, it might be important
to have valid management_address that services are relying on when
listening on interfaces. At the moment for bare metal hosts
management_address will be equal to ansible_host which leads to
unpredictable behaviour under some scenarios. With management_ip we allow
to define another IP address that will be used as container/management
address for bare metal host, while `ip` will still represent
ansible_host.

Related-Bug: #2002645
Change-Id: I3152ae7985319e85b9ea520021f9eea6f5850341
2023-05-22 11:53:55 +02:00
Dmitriy Rabotyagov 86d1bdff55 Rename container_address to management_address
This patch aims to reduce confusion caused by a variable
`container_address` that's applicable for bare metal hosts. With that
it renames  `is_container_address` to `is_management_address`
to be aligned with the purpose of the variable, as `container` part
raised confusion.

Change-Id: I314224f3376cf91e05680b11d225fdaf81ec32ab
2023-05-22 09:57:17 +02:00
Dmitriy Rabotyagov fb34651cc2 Add is_nest property for container_skel
The main purpose of that patch is to allow creation of "virtual"
container_skel, that will not generate any containers in inventory,
but will contain all containers of corresponsive hosts. That might be
useful in usecases like AZs, or when deployer simply wants to create
custom groups and include all bare metal hosts along with all
containers on them to the same group. Such behaviour can be triggered
when empty belongs_to is provided for container_skel along with is_nest
property. Then container_skel item will contain host-containers and it's
children.

Change-Id: Ic5570bfe9f0f54d1ea1e067834c11e6c390a2686
2023-03-27 17:33:18 +02:00
Dmitriy Rabotyagov 90fdc6322f Drop `else` condition in the container_skel_load loop
This cycle does contain `break` statement, thus there is no reason
to have `for/else` statement since `else` is always executed.

Change-Id: I82275f208b6674ac9b528d01b738748965bfe46e
2023-03-27 17:32:04 +02:00
Dmitriy Rabotyagov bb3a58604b Restore dynamic_inventory unit testing
It seems that we have dropped unit testing of our dynamic_inventory
some time ago. This patch aims to setup zuul tests using tox
and restore test functionality to make it passing with current codebase

Change-Id: I8ccee779fd629e8696ec2e12397b148b52cd0c73
2023-01-11 10:14:43 +00:00
Zuul 2753702d53 Merge "Deprecate openstack_hostnames_ips" 2022-08-12 14:13:13 +00:00
Jonathan Rosser c76a4d5dfe Do not create {hostname}-host_containers group as child of other groups
The existing code adds entries like aio1-host_containers as a child group
of many other container type groups in the inventory.

The side effect is that the ansible magic variable group_names for
a particular LXC container lists many group names associated with
all of the containers on its physical host.

Roles such as os_ironic create several containers and use conditional
logic to deploy the correct components into each container using the
contents of group_names. This does not work correctly when group_names
contains all of the possible ironic container groups.

This patch removes the code which adds host container groups as
children of other container type groups.

Removal of the {hostname}-host_containers group from each container
reveals a further bug, where any belongs_to directives in env.d
file container_skel sections were not processed. There is different
functionality in skel_load() and container_skel_load() which is the
cause of this.

This patch adds a call to _parse_belongs_to() into
container_skel_load() so that any groups defined with 'belongs_to'
in container_skel are correctly added as children to the
corresponding parent group.

Change-Id: Ic76b2c211484fb107d8d23f4ef6e6cc9a4ddec4f
2022-08-02 21:06:20 +00:00
Dmitriy Rabotyagov 7ebd4a7914 Deprecate openstack_hostnames_ips
We never used that faile anywhere in the inventory generation process,
except saving mapping of hostnames to container_address.
At the same time we never stored extra IPs there and as of today we
can simplify workflow by simply dropping that file.

Change-Id: Id8c4f6512bc913b2480106dedfa2e457c4776ae7
2022-07-28 11:48:27 +02:00
Dmitriy Rabotyagov 21dd4e6c5d Add ability to define bridge type for containers
This change allows to define `container_bridge_type` among provider_networks to
provide type of bridge being used (ie ovs).

Documentaion on usage will be provided in following patch.

Needed-By: https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/837734
Change-Id: I4f65c13a7dd16a66b2b14ae545516533f5ec69e1
2022-06-08 08:10:36 +00:00
Dmitriy Rabotyagov 273c0e9d95 Rename black/white list variables
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/796629
Change-Id: I079da1803369ba24276788954663b2a7a09f6001
2021-06-18 08:35:55 +00:00
Dmitriy Rabotyagov 13fcd09b0b Add option to remove group from inventory
Added `-d` option to inventory-manage.py that will remove provided
group from the inventory. It won't touch conf.d so you need to remove
entry from there before re-running dynamic_inventory

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/791107
Change-Id: I665ae5ce999c36d8d999ed5f9d659095a900d9eb
Closes-Bug: #1642051
2021-05-14 04:17:43 +00:00
Jonathan Rosser 24b8ea0ba6 Remove support for nspawn containers.
Support for nspawn was previously deprecated, and is now removed.

Change-Id: I37811d66eedf7aa781d4365024b7181ba44081bc
2021-04-20 16:59:13 +03:00
James Denton d1b50915cc Make container_bridge optional for provider networks
When Neutron agents are deployed on baremetal, there is no need
to specify a container_bridge name in the provider network config.
This patch makes it optional in that the script will not fail if
container_bridge is not defined.

Change-Id: I511eff1686b71018338f4fe73fe83dedf41d50fe
2020-12-12 18:39:52 +00:00
Ryan Drew d2657174ce
Fix KeyError raised when max hostname length exceeded
The function osa_toolkit.generate._add_container_hosts contains a bug in
the code used to check if provided hostnames have exceeded their max
allowed length. The logic used to perform this check depends on the
`is_metal` flag within each container's properties. Unfortunately the
`is_metal` flag is accessed within the `properties` dictionary using
bracket notation rather than the safer `dict.get` method, causing a
`KeyError` to be raised when a host's properties dictionary does not
contain the `is_metal` flag.

It is not expected that a `KeyError` would be raised in the function if
hostnames have exceeded their max length. It is instead expected that a
`SystemExit` exception would be raised warning the user of their invalid
hostname(s).

This bug will impacts deployments where hostnames actually do exceed the
max allowed length due to the short circuit logic used in the if-elif
tree.

Closes-Bug: #1886905

Change-Id: Ic1acfea71f27f94e277aa443f0a53ef16b4eb417
2020-07-09 12:01:00 -06:00
Jonathan Rosser 5c53b88e4c Fix inventory_manage when the component for a host is not defined
Previously this would fail with a stacktrace, so now display something
useful when the component is not defined.

Change-Id: I27aa0d0b32fd1db47cc2fc43afbad085ea22d0bf
2020-05-26 17:10:12 +01:00
Andreas Jaeger 2860f1bda5 Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found.

Depends-On: https://review.opendev.org/722854
Change-Id: I190f32b2eea20024c71fc74bac7f5d011768473c
2020-04-29 06:20:40 +00:00
Dmitriy Rabotyagov 2a7edfb036 Do not append container_name for metal containers
dynamic_inventory.py fails in case of hostname+container_name length
is more than 52 symbols. While it's valid appriach for containers,
it shouldn't raise error when container is deployed on metal, since
nothing will be appended to it's current hostname.

Change-Id: Ifeed116a026042ae32985aaa7ff3f167ef3923f4
2020-02-05 14:09:45 +02:00
Dmitriy Rabotyagov 94e477032a Don't require provider_networks with no_containers
When no_containers is set in global_overrides it means that it's going
to be metal deployment. Since metal deployments do not require
provider_networks, we can easily skip check of their existance in
openstack_user_config.yml

Change-Id: Ie12d13f5eb90bc4fbb4bf73e2d5915b1493e819d
2019-12-02 14:38:27 +00:00
Dmitriy Rabotyagov 5ae62b53ce Make modern flake8 happy about our code
In order to fit modern flake8, some minor changes required for scripts

Change-Id: Ic3a02c6c19d230ca6edb6b068a03044225cd9fc4
2019-09-10 14:59:04 +03:00
Dmitriy Rabotyagov 21101aaa4d Allow to globally set no_containers
Previously no_containers could be set only for specified host group.
This patch adds option to define no_containers inside global_overrides
to set fully bare_metal deployment across all host groups.

Change-Id: I375ce28cdb7489af631d4ad41dc1ecaa78bd6d49
2019-08-30 17:09:27 +03:00
Zuul c7a3a75391 Merge "Fix scripts/inventory-manage.py help invocation" 2019-08-13 20:54:23 +00:00
Mohammed Naser e9c9d92987 inventory: don't crash with missing global overrides
Change-Id: If21b919cfbf2dd306ae158ed9ead84da5cb338bc
2019-07-29 15:07:50 +00:00
Francois Deppierraz cdc337d9a1 Fix scripts/inventory-manage.py help invocation
Without this patch, the following exception was raised.

root@os-deploy:~# /opt/openstack-ansible/scripts/inventory-manage.py -h
Traceback (most recent call last):
  File "/opt/openstack-ansible/scripts/inventory-manage.py", line 39, in <module>
    manage.main()
  File "/opt/openstack-ansible/osa_toolkit/manage.py", line 329, in main
    user_args = args()
  File "/opt/openstack-ansible/osa_toolkit/manage.py", line 100, in args
    return vars(parser.parse_args())
  File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1939, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/lib/python2.7/argparse.py", line 1879, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python2.7/argparse.py", line 996, in __call__
    parser.print_help()
  File "/usr/lib/python2.7/argparse.py", line 2340, in print_help
    self._print_message(self.format_help(), file)
  File "/usr/lib/python2.7/argparse.py", line 2314, in format_help
    return formatter.format_help()
  File "/usr/lib/python2.7/argparse.py", line 281, in format_help
    help = self._root_section.format_help()
  File "/usr/lib/python2.7/argparse.py", line 211, in format_help
    func(*args)
  File "/usr/lib/python2.7/argparse.py", line 211, in format_help
    func(*args)
  File "/usr/lib/python2.7/argparse.py", line 517, in _format_action
    help_text = self._expand_help(action)
  File "/usr/lib/python2.7/argparse.py", line 603, in _expand_help
    return self._get_help_string(action) % params
TypeError: unsupported operand type(s) for %: 'tuple' and 'dict'
root@os-deploy:~#

Change-Id: I9c0d6ce574caabcf4b59b5896aba1b1d21096f79
2019-06-07 11:04:47 +02:00
Zuul c8619eed66 Merge "Automatically prune the inventory backup" 2019-04-01 19:52:51 +00:00
Kevin Carter ac28ad1329 Automatically prune the inventory backup
The inventory backup process takes the running inventory json file and
adds it to a tar archive. This process has no limits and will add files
to the tar archive until that is no longer possible and limited by the
underlying operating system. This change automatically prunes the backup
file and retains only the last 15 inventory files. This should provide
the same backup capabilities we've had without trying to saving
archives indefinitely.

> It should be noted that this change is using a subprocess call to
  prune the tar file. This is being done because the "tarfile" library
  does not provide an interface for deleting a file within an archive.

Change-Id: Ida5a9be0d0910c223fe05401bc4f75aef100e456
Closes-Bug: #1750233
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2019-03-21 09:29:26 -05:00
Shannon Mitchell 71a067abf2 Dynamic inventory backup corruption fix
When multiple users and process are accessing the dynamic inventory
the inventory backup and json can get corrupted. This change checks
for inventory modifictions and only saves if needed. The backup
is also moved to right before the actual save.

Change-Id: Ifd348ddd9c21526f5b523963dd1fd247edd6b109
Closes-Bug: #1750233
2019-02-17 18:02:57 -06:00
Kevin Carter 9229315a1a Add option to disable container builds on specific hosts
This change adds an option to disable any and all containers within a given host
The new option "no_containers" is a boolean option and has a default value of
false. When set true no containers will be created in inventory for a given machine.

This will allow deployers to simply disable container builds on specific machines
when nessisary. This is usefule when a deployer simply wants to deploy openstack
in a flattened environment. This has also been useful in cases where
infrastructure hosts may reside within virtualization solutions like OpenStack,
VirtualBox, or VMWare which implement strong port security rules which makes
container networking difficult or impossible.

To implement the change simply add the "no_containers: true|false" option to
a host entry in the openstack_user_config.yml

Change-Id: I8bed3e498a431e2683956e1009d9d9ece9fdb272
Signed-off-by: Kevin Carter <kevin@cloudnull.com>
2019-02-12 20:27:19 +00:00
Zuul 017f736804 Merge "Make container_interface optional for provider network" 2019-01-18 18:05:57 +00:00
James Denton 8d9c0336ec Make container_interface optional for provider network
As Neutron agents move to bare metal by default, the container_interface
attribute for a Neutron provider network (e.g. br-vlan, br-vxlan, etc) is
unnecessary and confusing. This patch aims to change it from a required
attribute to an optional one.

Change-Id: I52692f5a36e7064c3a6ac6ccafc2420549685a66
2019-01-15 16:00:00 +00:00
James Denton 8f088db5d9 Update inventory generator to support container gateway override
This patch updates the inventory generation script to support overriding
the default gateway in a container, and aims to provide support for a feature
documentation claims is there (but isn't).

Change-Id: Id86396f3e016ad83f312988fdc00fec6302d0802
Partial-Bug: #1810319
2019-01-11 13:09:14 +00:00
Kevin Carter 4a07e2612e Minimal(ist) network config for nspawn gating
Nspawn containers can attach directly to the physical host interface
using macvlan which allows us to greatly simplify connectivity in test
instances.

Changes to the user variable files have been made to allow the services
to function on a simplified stack.

Depends-On: I083042a791d9213b9b1872a239dc18dc6c7ae46e
Change-Id: Iaa7cfb051d26bcd080df15ef8949d3cf16208ea9
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-09-14 09:36:12 -06:00
Zuul 77498df75c Merge "Add a warning in openstack_hostnames_ips.yml" 2018-06-24 22:32:05 +00:00
Jesse Pretorius 563a4888c6 Fix osa_toolkit dictutils import
Currently the inventory-manage tooling does
not work correctly because it cannot find the
dictutils/filesystem libraries in its path.

Change-Id: I6055df3da794439fb65ad2244445135dafb1cecf
2018-04-09 20:44:59 +01:00
Zuul db13d26c04 Merge "Stop inventory constantly giving containers new IP" 2018-03-19 10:50:12 +00:00
James Portman b72b3add9f Stop inventory constantly giving containers new IP
See bug for further details, this was affecting pike after applying
the backported patch.

As the bug mentions, some containers were constantly given new IPs,
even with no changes to user_variables yml files or
openstack_user_config.yml file.

Further to the bug, I think that the code in master/patch will
cause containers to receive new IPs when network settings are
changed (bridge, mtu, type) rather than just updating that setting.

Change-Id: I0a757e9f503d3f604b2ba2c3409b4a4507d5edff
Closes-Bug: 1756091
2018-03-16 09:15:10 +00:00
Kevin Carter beb36b8ab9 Remove the "is_ssh_address" option from inventory
The key **is_ssh_address** has been removed from the
`openstack_user_config.yml` and the dynamic inventory. This key was
responsible mapping an address to the container which was used for SSH
connectivity. Because we've created the SSH connectivity plugin, which
allows us the ability to connect to remote containers without SSH, this
option is no longer useful. To keep the `openstack_user_config.yml` clean
deployers can remove the option however moving forward it no longer has any
effect.

Change-Id: I9264b73dcef71eba9ac29b238683a3a8c53e7121
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-03-15 00:58:13 +00:00
Maxime Guyot d080a61d69 Add a warning in openstack_hostnames_ips.yml
Change-Id: I94de38ab0d314e7a029680ec6fc95a3f1bc056e2
2018-02-26 11:50:43 +01:00
Kevin Carter fd9cda8df9
Add nspawn container driver
This change adds an nspawn container driver which will enable deployers
to run clouds with systemd-nspawn instead of LXC. This adds "nspawn" to
as an option to the `container_tech` variable. To support this change,
The inventory generation tools have been updated to allow for a
new group named `nspawn_hosts`. All of the container connectivity and
setup are stored within the integrated repo under the new templates
directory.

The addition of "nspawn" container driver enables the ability for
deployers to change, or mix container technologies within a single
deployment without needing to change our well defined network
topology or storage layout.

Depends-On: I13d05ba8bcfe785257a9cf98dbdb6024ec937816
Change-Id: I41cfec63c423cd56a91c25dabae9aa1031c27e03
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-02-11 19:02:24 -06:00
Jean-Philippe Evrard 8a3dfc01fa container_host can be replaced by physical_host on is_metal
Upgrades broken.
This fix.

Change-Id: I6b881a9657f8860d2cb21e0eb99db6958806e30b
Closes-Bug: #1745382
2018-01-25 14:25:25 +00:00
Jean-Philippe Evrard 81e1f8d173 Include P-Q changes into the upgrade script
We need to include changes in the upgrade script to handle
the changes done during the Queens timeframe:

- Inventory is fixed to prevent a stacktrace if a container
  default was moved from is_metal: False to is_metal: True,
  and now respects existing container properties.
- Ensuring previous inventory location isn't overriding the
  Queens inventory location in memory, by unsetting
  ANSIBLE_INVENTORY.
- Ensuring all containers are using the repo that was build
  during the upgrade.

Change-Id: I0fd3a8803bf345ccad07dc97968e104cd0a7bf70
Partial-Bug: #1741462
Closes-Bug: #1675810
2018-01-19 13:17:42 +00:00
Andy McCrae 850d8f2663 Update existing container_networks
We don't update an existing container_network so once you have settings
set, they will not be changed, even if you change settings.

This patch ensures that if the existing container_network does not match
the current provider_network, it will get updated.

Closes-Bug: 1678165
Change-Id: Ia6d91bd6aabf0f9584a83641d5bd990d3f9e404f
2017-12-11 12:08:37 +00:00
Jesse Pretorius b95eafb0ee Skip host pip installs for ansible bootstrap
The requirements.txt contents do not need to be
installed on to the host. The majority of the
requirements are for ansible, or for release
and management tooling which needs to use the
Ansible runtime venv.

Rather than forcing the installation of pip on
the host, we only install virtualenv via distro
packages (where possible). With virtualenv in
place we can create the runtime venv and install
pip, etc and all requirements into there.

Doing this keeps the system python libraries as
clean as possible, preventing clashes with other
packages (eg: ceph) which try to install other
python libraries which conflict on CentOS.

Change-Id: I0db786645c11649764680697518c97ddf9610cfa
2017-09-12 12:33:20 -06:00
Jimmy McCrory 9cd42929c3 Support cidr_networks in L3 network environments
In some environments, a single container, storage, or tunnel network may
not be applicable to every host. Each configured provider_network would
need to be limited to a particular subset of hosts and the host var keys
within the inventory for container_address, storage_address, and
tunnel_address will need to be maintained since they're specifically
required by various playbooks.

Add two new options for configuring provider_networks, 'reference_group'
and 'address_prefix'.

'reference_group' for providing a group name that any host must
be a member of, in addition to any of the groups listed in
'group_binds', for the network to be applied.

'address_prefix' for overriding the name of the key created for each IP
address allocated by a cidr_network. By default, this key is named
'cidr_network'_address, where 'cidr_network' is the given 'ip_from_q'
option for a provider network.

Closes-Bug: 1650356

Change-Id: Ia7f3119f0affc4fb6be97ca788ca3b46096b82a8
2017-07-31 22:14:25 +00:00
Nolan Brubaker ea6954f72f Use python import machinery for inventory code
Change Idb7780f55e4a1fd77dd76becbf67c1ccbf220db7 restructured the python
inventory generation code so that it would be possible to install it
with pip. This change removes (most) of the import path hacks and
switches to using a pip-installed version of osa_toolkit.

Of note, the path hacks are left in place for the dynamic_inventory.py
file for now, as that file is really more of an endpoint, but is tested.

Also, the bootstrap-ansible.sh script was modified to install the code;
this is unnecessary with the tox environments because the tox directive
'usedevelop=True' does that already. Production environments still need
this, though.

Finally, to maintain usability when called directly, the interpreter for
dynamic_inventory.py was updated from `/usr/bin/env` to
`/opt/ansible-runime/python`. This change ensures that in a full
deployment the user is using the exact same code paths whether Ansible
invokes the script, or it is called directly. This also means that using
the script locally on a development machine, it must be invoked as an
argument to Python, unless the ansible-runtime directory exists.

Change-Id: Iafa573b1b144f98528d5e0aceb3f36e9de2a22a2
2017-07-28 16:39:52 +00:00
Nolan Brubaker e2f56ec634 Fix Python 3 bytes vs strings differences
Writing files was done fairly naively with our Python 2 code - this
change introduces explicit ASCII encoding when writing files.

Change-Id: I93c9fa4ba2fd8af9c9bf9424adaec602dbdca5f0
2017-06-08 18:42:07 +00:00
Nolan Brubaker fa5336ca36 Fix Python 3 dictionary incompatibilities
There were two instances of abusing old Python 2 behaviors around
dictionary keys that Python 3 no longer allows. These are both fixed.

Change-Id: I5865ee3becf52fca590a1c0c897bcd968210ac9d
2017-06-08 18:41:35 +00:00