openstack
/
ironic-python-agent
Code Issues Proposed changes
ironic-python-agent/doc/source/admin/troubleshooting.rst

7.3 KiB
Raw Blame History

Troubleshooting Ironic-Python-Agent (IPA)

This document contains basic trouble shooting information for IPA.

Gaining Access to IPA on a node

In order to access a running IPA instance a user must be added or enabled on the image. Below we will cover several ways to do this.

Access via ssh

CoreOS

To enable the core user on pre-built or CoreOS images a ssh public key will need to added. To do this you will need to:

  • Add sshkey="ssh-rsa AAAA..." to pxe_append_params setting in ironic.conf file
  • Restart the ironic-conductor with the command service ironic-conductor restart
  • ssh core@<ip-address-of-node>

diskimage-builder (DIB)

SSH access can be added to DIB built IPA images with the dynamic-login1 or the devuser element2

The dynamic-login element allows the operator to inject a SSH key when the image boots. Kernel command line parameters are used to do this.

dynamic-login element example:

  • Add sshkey="ssh-rsa BBA1..." to pxe_append_params setting in the ironic.conf file
  • Restart the ironic-conductor with the command service ironic-conductor restart

devuser element example:

export DIB_DEV_USER_USERNAME=username
export DIB_DEV_USER_PWDLESS_SUDO=yes
export DIB_DEV_USER_AUTHORIZED_KEYS=$HOME/.ssh/id_rsa.pub
disk-image-create -o /path/to/custom-ipa debian ironic-agent devuser

tinyipa

If you want to enable SSH access to the image, set ENABLE_SSH variable in your shell to true before building the tinyipa image:

export ENABLE_SSH=true

By default it will use default public RSA (or, if not available, DSA) key of the user running the build (~/.ssh/id_{rsa,dsa}.pub).

To provide other public SSH key, export full path to it in your shell before building tinyipa as follows:

export SSH_PUBLIC_KEY=/path/to/other/ssh/public/key

The user to use for access is default Tiny Core Linux user tc. This user has no password and has password-less sudo permissions. Installed SSH server is configured to disable Password authentication.

Access via console

If you need to use console access, passwords must be enabled there are a couple ways to enable this depending on how the IPA image was created:

CoreOS

CoreOS has support for auto login on the console3. This can be enabled by:

  • Adding coreos.autologin to pxe_append_params setting in the ironic.conf file. See4 for more information on using autologin.

If you do not wish to enable auto login users can be added to CoreOS by editing the cloud-config.yml file and adding the following5:

users:
  - name: username
    passwd: $6$5s2u6/jR$un0AvWnqilcgaNB3Mkxd5... <example password hash>
    groups:
      - sudo

If using a pre-built image the cloud-config.yml must first be extracted:

mkdir tmp_folder
cd tmp_folder
zcat ../coreos_production_pxe_image-oem-stable-mitaka.cpio | cpio --extract --make-directories

To create a password hash the mkpasswd command can be used:

mkpasswd --method=SHA-512 --rounds=4096

After adding the user block with your favorite editor recompress the image:

find . | cpio --create --format='newc' |gzip -c -9 > ../coreos_production_pxe_image-oem-stable-mitaka.cpio.NEW.gz

An alternative to editing the embedded cloud-config.yml6 file is to pass a new one on the kernel command line by:

  • adding cloud-config-url=http://example.com/cloud-config.yml to pxe_append_params setting in the ironic.conf file

diskimage-builder (DIB)

Users wishing to use password access can be add the dynamic-login7 or the devuser element8

The dynamic-login element allows the operator to change the root password dynamically when the image boots. Kernel command line parameters are used to do this.

dynamic-login element example:

Generate a ENCRYPTED_PASSWORD with the openssl passwd -1 command
Add rootpwd="$ENCRYPTED_PASSWORD" value on the pxe_append_params setting in /etc/ironic/ironic.conf
Restart the ironic-conductor with the command service ironic-conductor restart

Users can also be added to DIB built IPA images with the devuser element9

Example:

export DIB_DEV_USER_USERNAME=username
export DIB_DEV_USER_PWDLESS_SUDO=yes
export DIB_DEV_USER_PASSWORD=PASSWORD
disk-image-create -o /path/to/custom-ipa debian ironic-agent devuser

tinyipa

The image built with scripts provided in imagebuild/tinyipa folder of Ironic Python Agent repository by default auto-logins the default Tiny Core Linux user tc to the console. This user has no password and has password-less sudo permissions.

Set IPA to debug logging

Debug logging can be enabled a several different ways. The easiest way is to add ipa-debug=1 to the kernel command line. To do this:

  • Append ipa-debug=1 to the pxe_append_params setting in the ironic.conf file
  • Restart the ironic-conductor with the command service ironic-conductor restart

Another method is to edit the cloud-config.yml file. IPA's instructions on building a custom image can be found at10.

This essentially boils down to the following steps:

  1. git clone https://git.openstack.org/openstack/ironic-python-agent
  2. cd ironic-python-agent
  3. pip install -r ./requirements.txt
  4. If not installed, please install the docker container engine.11
  5. cd imagebuild/coreos
  6. Edit oem/cloud-config.yml and add --debug to the end of the ExecStart setting for the ironic-python-agent.service unit.
  7. Execute make to complete the build process.

If the system is running and uses systemd then editing the services file will be required.

  • systemctl edit ironic-python-agent.service
  • Append --debug to end of the ExecStart command
  • Restart IPA. See the Manually restart IPA section below.

Where can I find the IPA logs

Retrieving the IPA logs will differ depending on which base image was used.

  • Operating system that do not use systemd (ie Ubuntu 14.04)

    • logs will be found in the /var/log/ folder.

  • Operating system that do use systemd (ie Fedora 22, CoreOS)

    • logs may be viewed with sudo journalctl -u ironic-python-agent

    Note

    sudo is not required with the CoreOS images.

Manually restart IPA

In some cases it is helpful to enable debug mode on a running node. If the system does not use systemd then IPA can be restarted directly:

sudo /usr/local/bin/ironic-python-agent [--debug]

If the system uses systemd then systemctl can be used to restart the service:

sudo systemctl restart ironic-python-agent.service

References

  1. `Dynamic-login DIB element`: https://github.com/openstack/diskimage-builder/tree/master/elements/dynamic-login↩︎

  2. `DevUser DIB element`: https://github.com/openstack/diskimage-builder/tree/master/elements/devuser↩︎

  3. `Booting CoreOS via PXE`: https://coreos.com/os/docs/latest/booting-with-pxe.html↩︎

  4. `Booting CoreOS via PXE`: https://coreos.com/os/docs/latest/booting-with-pxe.html↩︎

  5. `Add User to CoreOS`: https://coreos.com/os/docs/latest/adding-users.html↩︎

  6. `Booting CoreOS via PXE`: https://coreos.com/os/docs/latest/booting-with-pxe.html↩︎

  7. `Dynamic-login DIB element`: https://github.com/openstack/diskimage-builder/tree/master/elements/dynamic-login↩︎

  8. `DevUser DIB element`: https://github.com/openstack/diskimage-builder/tree/master/elements/devuser↩︎

  9. `DevUser DIB element`: https://github.com/openstack/diskimage-builder/tree/master/elements/devuser↩︎

  10. `IPA image build reference`: https://github.com/openstack/ironic-python-agent/tree/master/imagebuild/coreos/README.rst↩︎

  11. `Install docker engine`: https://docs.docker.com/engine/installation/↩︎