Commit Graph

537 Commits

Author SHA1 Message Date
Adrian Vladu a373d559e2 winrmlistener: use sha2 instead of insecure sha1
SHA1 is no longer secure and thus needs to be replaced by
a secure algorithm, in this case SHA256.

See:
https://en.wikipedia.org/wiki/SHA-1#Attacks

Fixes: https://github.com/cloudbase/cloudbase-init/issues/123

Change-Id: Ib565b99116fe966421f57b6c1f3bf6d6b9589288
Signed-off-by: Adrian Vladu <avladu@cloudbasesolutions.com>
2024-03-04 11:36:06 +00:00
Adrian Vladu 0fded69cb3 replace unittest.mock._get_target with mock one
Use the original importer for mock. Otherwise, the unit tests that
use importlib with context are not running in an isolated manner,
leading to various transient failures if `stestr` is not run with the
`--isolated` flag.

The unittest.mock._get_target has been changed in Python 3.11, by commit:
ab7fcc8fbd

This change is needed to add support for Python 3.11 and Python 3.12
unit testing.

Change-Id: Ic73f5280116673f771f5460cdfd3702f5fbe16b5
2023-11-09 15:39:45 +02:00
Zuul 36ed936803 Merge "use instance_id as string for plugin sections" 2023-10-16 14:05:01 +00:00
Adrian Vladu 86adb2ce57 use instance_id as string for plugin sections
If `instance_id` is set in the metadata as an integer, cast it as a
string when used to uniquely identify plugin sections.

Fixes-Bug: https://github.com/cloudbase/cloudbase-init/issues/124

Change-Id: I3a9335fb3885c5f3191051aaaeeec94f89a6b89a
2023-10-12 17:09:05 +03:00
Luqman Aden 9b17eaf1d4 Accept drives with vFAT label 'cidata' as a configdrive.
Match vFAT label using a given label instead of hardcoding 'config-2'.

The NoCloud and ConfigDrive metadata services both attempt to find vFAT
formatted drives but the label used was hardcoded to 'config-2' which
only works with ConfigDrive. BaseConfigDriveService which both inherit
from already has a drive_label property set appropriately for both so
let's use that for the vFAT finding logic as well.

Change-Id: I8004a8565338b0615450bb28cecc86901be94766
2023-05-14 06:32:55 +00:00
Adrian Vladu 1a28d39c72 bsdtar: check if tool path exists
Check if the configuration value of bsdtar_path does exist as a path
before trying to execute the binary.

Updated the tutorial reference of bsdtar to FreeBSD instead of Ubuntu.

Change-Id: Ieba5da2f330aa11c40cce6c2ae9de40155f33b07
2023-01-19 14:52:49 +02:00
Adrian Vladu ea5e8da627 crypto: replaced openssl with cryptography module
Use cryptography instead of the flaky openssl libraries loading.
If the libssl DLLs were not present or were present in another order
then the required one, there could be errors when securing the password
before sending it to the metadata service.

Fixes: https://github.com/cloudbase/cloudbase-init/issues/34

Change-Id: I1a2245e199f65f4665071ada9576dcae77a3a432
2022-12-12 14:22:56 +02:00
Zuul 576db310c2 Merge "cloudconfig: fix type in key name no_create_home" 2022-12-07 15:26:14 +00:00
Zuul 9198d0ba38 Merge "Retry user load profile on Windows" 2022-12-07 15:17:29 +00:00
Adrian Vladu cae4135e3b cloudconfig: fix type in key name no_create_home
User cloudconfig plugin had a typo in the key: "no_create_home"
Change-Id: Ia8261dfcc8dc5d817d351735c77cc87db76be8f0
2022-12-07 17:05:57 +02:00
Adrian Vladu da0c0dc7bd crypt: use the py3 builtin openssl
With Python 3.7+, the Python installer has the libcrypto and libssl dlls
in the DLLs Python folder, namely libcrypto-1_1.dll and libssl-1_1.dll,
which can be used directly.

Change-Id: I245c377dc8a9ec9a2e8548806bacd757bfdf27b6
2022-12-07 00:42:59 +02:00
Adrian Vladu 072bee9c5d xml: force return of unicode string
With Python 3.8, the xml.etree.ElementTree.ElementTree function now
preserves the attribute order specified by the user.

This change broke the winrm enable functionality, throwing the erorr:

```
TypeError: Objects of type 'bytes' can not be converted to Unicode.
```

The fix was to set the correct parameter to return an unicode string and
not bytes.

Change-Id: I10f7fa357267f36dfed03ccb708694f6fc1cdd97
2022-11-21 20:13:23 +02:00
Adrian Vladu 888e1b4602 azureservice: added redundance for wireserver
* Added more retries for wireserver discovery
* Added logging to wireserver discovery errors
* Fallback to the well known wireserver IP in case of discovery failure

Github Issue: https://github.com/cloudbase/cloudbase-init/issues/94

Change-Id: I91460f9ad674294a83c9b3c3f05bc04ab61de56f
2022-09-08 12:48:39 +03:00
Adrian Vladu 1a32d738de opendev: bumped gates to python 3.9/3.10
Python 3.6/3.7 are no longer supported for the latest releasese of
the OpenStack packages like oslo or coverage, thus switching to the Zed
OpenStack release gates, which have support for 3.8/3.9/3.10 Python
versions.

For gates to pass, the docs required a defined language to be set ("en"),
while the crypto unit tests required a method signature fix when running
on Python 3.9/3.10.

The version test__check_latest_version* unit tests started failing
in a transient pattern because of the
pbr.version.VersionInfo().release_string() throwing the following error:
"NotImplementedError: cannot instantiate 'WindowsPath' on your system".
This required to have the cloudbaseinit.version.get_version mocked for a
reliable unit test run on Linux.

Change-Id: I4748d1258c072c377825474e1116347b0a085c56
2022-09-07 20:26:34 +03:00
Adrian Vladu f8479e5cbc Continue execution on volume extension failure
When a small amount of disk space remains to be extended,
the 'Virtual Disk Service' can fail with
VDS_E_EXTENT_SIZE_LESS_THAN_MIN error code.

This happens on environments where a volume has already been extended
and there is a small amount of bytes that somehow remain residual after
the initial extension.

Change-Id: I072ed568ef1f2790e95851b45afb8ffcc0acce0e
2021-10-18 17:17:25 +03:00
Zuul 300dabb36e Merge "windows: network: fix DHCP setting" 2021-09-14 10:10:29 +00:00
Adrian Vladu 8a6f478aad windows: network: fix DHCP setting
Fixed Windows osutils DHCP setting, as the former implementation that
used registry keys was no reliable. Using MSFT_NetIPInterface should fix
any reliability issues.

Change-Id: I94cf70b092da706cdfaf57bc78d7b266064b97b0
2021-09-14 12:29:38 +03:00
Adrian Vladu 2f4f315e20 jinja2: remove usage of runtime string decorator
Change-Id: I5b19c7ad798f490031b29b99ecc43d417e1a1a21
2021-07-07 14:36:17 +03:00
yoerg 39e2f6faf4 Support retrieving admin_username from OpenStack metadata
Allows giving an instance an individual cloud username, e.g. for license servers, while still retrieving SSH public keys for that user from OpenStack.

Change-Id: I078380ef8f74b30105fd44378045df8b1260acbc
2021-01-20 12:05:11 +01:00
Adrian Vladu a9ffc62c11 Retry user load profile on Windows
On Windows, the load user profile may fail on laggy systems, if the
Windows subsystems are not ready at that moment.

Retrying the load should fix the issue most of the times.

Change-Id: I28cc564ebeac6d901dcbbef7cebe882a5ccb41b1
2021-01-13 13:04:26 +02:00
Zuul 97295ba569 Merge "Retry netlbfo team creation" 2020-12-04 12:58:36 +00:00
Adrian Vladu ab342fc493 Retry netlbfo team creation
The WMI implementation for NETLBFO is sometimes flaky and does not fail
when a bond cannot be created.

Added 5 retries for the team creation, just in case the WMI call to
create the bond did not fail, but the bond was not created.

In this scenario, the _wait_for_nic method is called and it fails,
as it does not find the team nic and the retry is performed.

Updated the wait for team nic retry interval to 10 seconds,
as the team nic may be retrieved with WMI after a longer time.

Change-Id: I7622a4a30b867f335ed8f3288ca2fea830daf94b
2020-11-03 16:27:56 +02:00
Adrian Vladu 27d6fd01d6 opennebula: fix shell variables parsing
Fix parsing of the empty values written as:

KEY=""
KEY=''

Change-Id: I56ea03da5d78c01af2647444b825bf63d5146137
2020-10-01 09:25:28 +00:00
Adrian Vladu 28f4fbbd45 Fix Python 3 unit tests
Removed the functools partial from being mocked, as it is used
internally in the mocking implementation and generates unit test error.

Updated tox / unit test requirements according to new OpenStack gate
requirements.

Updating the tox requirements added a new set of flakes rules that had
to be skipped for now, which will be addressed in upcoming patches.

Change-Id: I231ca3b15313541e2c43785e80d9db5659aa1cc3
2020-09-30 15:14:05 +03:00
Adrian Vladu 246db50591 nocloud: support for cloud-init network version 1
Added support for NoCloud service with Networking Config Version 1:
https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v1.html

Supported configuration types: Physical, Bond, Vlan and Nameserver.
Unsupported configuration types: Bridge, Route.

NoCloud metadata folder should contain a file network-config.
If no file is found, it falls back to using the contents for network-interfaces
key from the metadata file.

Example:

```yaml
version: 1
config:
   - type: physical
     name: interface0
     mac_address: "52:54:00:12:34:00"
     mtu: 1450
     subnets:
        - type: static
          address: 192.168.1.10
          netmask: 255.255.255.0
          dns_nameservers:
            - 192.168.1.11
   - type: bond
     name: bond0
     bond_interfaces:
       - gbe0
       - gbe1
     mac_address: "52:54:00:12:34:00"
     params:
       bond-mode: active-backup
       bond-lacp-rate: false
     mtu: 1450
     subnets:
        - type: static
          address: 192.168.1.10
          netmask: 255.255.255.0
          dns_nameservers:
            - 192.168.1.11
   - type: vlan
     name: vlan0
     vlan_link: eth1
     vlan_id: 150
     mac_address: "52:54:00:12:34:00"
     mtu: 1450
     subnets:
        - type: static
          address: 192.168.1.10
          netmask: 255.255.255.0
          dns_nameservers:
            - 192.168.1.11
   - type: nameserver
     address:
       - 192.168.23.2
       - 8.8.8.8
     search: acme.local
```

Change-Id: Idc413f6f9f001b2327c33a796e9ed494be28ce26
2020-07-01 19:08:47 +03:00
Zuul 5c3979e1ae Merge "Move ip_netmask_to_cidr to network utils" 2020-06-25 18:27:00 +00:00
Adrian Vladu 1e89827e4d Move ip_netmask_to_cidr to network utils
Move baseopenstackservice._ip_netmask_to_cidr to utils/network so that
it can be reused by other network parsers.

Change-Id: Iacca02cda75fd5d5b80f6200e7d2f26a3381b737
2020-06-24 16:55:19 +03:00
Adrian Vladu cbbcb98680 serialization: catch yaml and json errors
It is necessary to catch yaml and json errors that come from invalid
json / yaml streams like '{}}'.

Having an enumeration after a key value was set is a yaml parser error.

Change-Id: Ia8b298c3f1b36c6dee29326955d1e76ade3104b1
2020-06-24 16:33:10 +03:00
Adrian Vladu 1582ebec2c licensing: properly set KMS host and product key
If a trial license key was set, the KMS default product key and host was
not properly configured, as the execution of the licensing plugin ended
abruptly.

The fix is to set the kms and product key without checking the evaluation
date corresponding for the trial key.

Change-Id: I45e9364661208c454ddf2be0ff925d149fe0a6b0
2020-06-02 11:32:01 +03:00
Zuul d53e765b26 Merge "packet: add phone_home and post_password support" 2020-05-27 11:02:56 +00:00
Adrian Vladu 107bdfaceb packet: add phone_home and post_password support
Add support for the Packet features:

  * phone home - after provisioning is ready, an empty HTTP POST request
  is performed to the phone_home_url
  * post user password - a random password is generated for the default
  user, the password gets encrypted using the encryption public SSH key
  and posted to the phone_home_url

Change-Id: Ib3e405b6b5bb4ca65d047c756513b9d6f87cf6de
2020-05-27 13:12:15 +03:00
Adrian Vladu fd3309c567 Fix empty DNS list
Method osutils.set_static_network_config expects dnsnameservers as list.

Use an empty list if the dnsnameservers is None.

Change-Id: I3b567a941336b61f3a505306f738f14e8b3db04f
2020-05-20 17:43:49 +03:00
Adrian Vladu 4b0d94cd0f Add NoCloudConfigDriveService metadata provider
Add support for NoCloud metadata provider, where the metadata
is provided on a config-drive (vfat or iso9660) with the label
cidata or CIDATA.

The folder structure for NoCloud is:

  * /user-data
  * /meta-data

The user-data and meta-data files respect the EC2 metadata
service format.

Supported features for the NoCloud metadata service:
  * instance id
  * hostname
  * plublic keys
  * static network configuration (Debian format)
  * user data

More information:
cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html

Change-Id: Ib434cf2b2b21bf9faa58e05ba40eb0135385c9ea
Implements: blueprint nocloud-metadata-support
2020-04-03 20:10:09 +03:00
Adrian Vladu fcb68a4dc7 Move parse_data to utils.serialization
Move and rename VMwareGuestInfoService._parse_data to
cloudbaseinit.utils.serialization.parse_json_yaml so that it can be
easiliy reused.

Change-Id: I3b86efd5e2b2062f89ffc7745f6f0481ec665078
2020-04-03 19:40:44 +03:00
Adrian Vladu 3e5b5f37ff Refactor config drive metadata service
Refactor the existing ConfigDrive Metadata service so that another
metadata format implemenation like NoCloud can be easily added.

Now, the drive label and the metadata file can be set in the
constructor, making it easy to add another config drive metadata service
with different label or path.

Change-Id: I8dd8160dfbe9f529bb8f30ab85181f264c18833e
2020-04-03 19:40:12 +03:00
Zuul 9e05ef0e91 Merge "Add Google Cloud Engine metadata service" 2020-02-26 19:22:46 +00:00
Adrian Vladu 29105932c0 Add Google Cloud Engine metadata service
Add cloudbaseinit.metadata.services.gceservice.GCEService that
supports Google Cloud Engine.

Supported features for the metadata service:
  * instance id
  * hostname
  * userdata
  * ssh keys

Change-Id: I9e5e2cbcaa0953fc0c0ae8117e258713ac6443b7
2020-02-26 13:26:05 +02:00
Adrian Vladu 75c6a0e0f8 Fix Py27 cloud-config ordering
On Py27, the yaml module implementation uses a normal dictionary for
implementation, resulting in arbitrary order for the parsed yaml.

If the config option cloud_config_plugins is not set or empty, fallback
to the default order defined in the factory.

Config option `cloud_config_plugins` is now used for filtering the cloud
config plugins to be executed, which was the original intended behaviour.

Change-Id: I59ea56e473ab343a826ff8c38787cb7f11cebd27
2020-02-06 12:49:40 +02:00
Adrian Vladu a77477e16e Add metadata servvice VMwareGuestInfoService
VMwareGuestInfoService is a metadata service which uses VMware's
rpctool to extract guest metadata and userdata configured for machines
running on VMware hypervisors.

The implementation is similar to:
https://github.com/vmware/cloud-init-vmware-guestinfo

Supported features for the metadata service:
  * instance id
  * hostname
  * admin username
  * admin password
  * public SSH keys
  * userdata

Configuration options:
```ini

[vmwarequestinfo]
vmware_rpctool_path=%ProgramFiles%/VMware/VMware Tools/rpctool.exe

```

The VMware RPC tool used to query the instance metadata and userdata
needs to be present at the config option path.

Both json and yaml are supported as metadata formats.
The metadata / userdata can be encoded in base64, gzip or gzip+base64.

Example metadata in yaml format:

```yaml
instance-id: cloud-vm
local-hostname: cloud-vm
admin-username: cloud-username
admin-password: Passw0rd
public-keys-data: |
  ssh-key 1
  ssh-key 2
```

This metadata content needs to be sent as string in the guestinfo
dictionary, thus needs to be converted to base64 (it is recommended to
gzip it too).

To convert to gzip+base64 format:

```bash
cat metadata.yml | gzip.exe -9 | base64.exe -w0
```

Co-Authored-By: Rui Lopes <rgl@ruilopes.com>
Change-Id: I6a8430e87ee03d2e8fdd2685b05e60c5c0ffb5be
2020-01-29 20:35:23 +02:00
Adrian Vladu fd2c15bef3 Fix templating when data is empty
Fixes base_template.load and remove_template_definition which fail when
the userdata is empty / None or when the userdata has only one line.

Change-Id: I2ba0e16489049be390b2c9bc1fdf1e1059f54eb1
2020-01-23 13:43:40 +02:00
Zuul 1ec8cd06d4 Merge "Add support for jinja as userdata template engine" 2020-01-21 14:54:43 +00:00
Zuul ea51dfb053 Merge "Implement cloud-config users and groups plugins" 2020-01-21 14:52:26 +00:00
Adrian Vladu 493e57c195 Add support for jinja as userdata template engine
If the userdata starts with the text '## template:jinja',
Jinja2 templating engine will be used to render that
userdata.

This feature leverages the get_instance_data metadata
service implementation and tries to be as close as possible
with the cloud-init's implementation.

Change-Id: I79cb0066f7d65ae27867ac01b443e9be432b4dc3
2020-01-21 14:36:16 +02:00
Zuul 72a94e0abd Merge "Add metadata interface for instance data" 2020-01-21 12:09:15 +00:00
Adrian Vladu 6ede055475 Implement cloud-config users and groups plugins
Add support for cloud-config users and groups creation.

For Windows, the following format is supported:

groups:
  - windows-group: [user1, user2]
  - cloud-users

users:
  -
    name: brian
    gecos: 'Brian Cohen'
    primary_group: Users
    groups: cloud-users
    passwd: StrongPassw0rd
    inactive: False
    expiredate: 2020-10-01
    ssh_authorized_keys:
      - first key
      - second key

The passwords for Windows users are required to be in
plain text. On *nix systems, the passwords are hashed.

If the password is not present, a random password will
be set.

Fixes: https://github.com/cloudbase/cloudbase-init/issues/26

Change-Id: I035f92849a59a8370df30a6de41f66f5fb2300af
2020-01-21 13:53:09 +02:00
Adrian Vladu 551c4eb318 Add metadata interface for instance data
Some plugins, like the userdata execution plugin, need a standard model
for the instance data, which can be applied by templating engines like
jinja to the userdata.

This way, a user can use this feature to customize her userdata scripts
with values that are specific to that instance runtime env or cloud platform.

The instance data structure is based on the cloud-init specifications:
https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html

The v1 namespace contains a subset of the cloud-init specs for the instance
data.

The ds.meta_data namespace contains all the values the v1 namespace
contains, in order to be compatible with cloud-init, plus a subset
of other instance data.

Change-Id: I5c529498f06fe3c86f7fa3c20fdf3091840c4041
2020-01-16 15:49:39 +02:00
Adrian Vladu c629181c63 Run cloud config plugins in order
The cloud config plugins were configured as a normal Python dictionary,
thus executing them in a random order, defined by the Python
implementation.

We require runcmd plugin to run last so that it is the last
configuration stage. In the future, we need the order as the groups
plugin is required to run before the users plugin.

Change-Id: I754ef7b15d5c7a12d08a12c876cfb52de64a06cc
2020-01-15 18:29:18 +02:00
Adrian Vladu ce74218315 Implement cloud-config runcmd
If the userdata is of type cloud-config, the runcmd entry can contain
multiple entries with commands that will be executed, in the order
of their definition.

The commands can be given as a string or as an array of strings, the
first item being the binary to be executed and the rest being the
parameters of that binary.

The commands will be aggregated and written into one single shell file,
in the order of their definition.
On Windows, the file will be executed by the native Windows
shell cmd.exe.

Example userdata file:

runcmd:
 - 'dir C:\\'
 - ['echo', '1']

Fixes: https://github.com/cloudbase/cloudbase-init/issues/27

Change-Id: Ie307e08f8c4108c7bf9108543cc90b6a7fa2e7ae
2020-01-14 17:05:01 +02:00
Zuul 630345dd38 Merge "Implement cloud-config ntp plugin" 2020-01-14 14:16:22 +00:00
Adrian Vladu b5e3c42f39 Added empty metadata service: EmptyMetadataService
The empty metadata service can be used to run plugins that do not
rely on metadata service information, like setting NTP, MTU,
extending volumes, local scripts execution, licensing, etc.

It can be used also as a fallback metadata service, in case no other
previous metadata service could be loaded.

EmptyMetadataService does not support the following plugins:
  * cloudbaseinit.plugins.windows.createuser.CreateUserPlugin
  * cloudbaseinit.plugins.common.setuserpassword.SetUserPasswordPlugin
  * cloudbaseinit.plugins.common.sshpublickeys.SetUserSSHPublicKeysPlugin
  * cloudbaseinit.plugins.windows.winrmcertificateauth.ConfigWinRMCertificateAuthPlugin

If any of the plugins defined above are executed,
they will fail with exception NotExistingMetadataException. The reason
for the hardcoded failure is that these plugins rely on metadata to execute
correctly. If metadata like username or password is not provided,
these plugins can lock or misconfigure the user, leading to unwanted problems.

Implements: blueprint empty-metadata-provider
Change-Id: I5fb88a07bf72321d0f66b16d151bf6059b448580
2020-01-13 16:39:44 +02:00