diff --git a/diskimage_builder/elements/base/extra-data.d/50-store-build-settings b/diskimage_builder/elements/base/extra-data.d/50-store-build-settings deleted file mode 100755 index 0ba873716..000000000 --- a/diskimage_builder/elements/base/extra-data.d/50-store-build-settings +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash -# Store the build-time environment and command line arguments - -if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then - set -x -fi -set -eu -set -o pipefail - -[ -n "$TMP_HOOKS_PATH" ] || die "Temp hook path not set" - -echo "$DIB_ENV" > $TMP_HOOKS_PATH/dib_environment -echo "$DIB_ARGS" > $TMP_HOOKS_PATH/dib_arguments diff --git a/diskimage_builder/elements/base/install.d/50-store-build-settings b/diskimage_builder/elements/base/install.d/50-store-build-settings deleted file mode 100755 index 8a25b53f4..000000000 --- a/diskimage_builder/elements/base/install.d/50-store-build-settings +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash -# Store build-time environment and command line arguments - -if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then - set -x -fi -set -eu -set -o pipefail - -if [ -e "/tmp/in_target.d/dib_environment" ]; then - cp /tmp/in_target.d/dib_environment /etc/ -fi - -if [ -e "/tmp/in_target.d/dib_arguments" ]; then - cp /tmp/in_target.d/dib_arguments /etc/ -fi diff --git a/diskimage_builder/elements/cloud-init/README.rst b/diskimage_builder/elements/cloud-init/README.rst index 707823e48..2f51545af 100644 --- a/diskimage_builder/elements/cloud-init/README.rst +++ b/diskimage_builder/elements/cloud-init/README.rst @@ -1,6 +1,6 @@ -======== +========== cloud-init -======== +========== Install's and enables cloud-init for systems that don't come with it pre-installed diff --git a/diskimage_builder/elements/debian-minimal/README.rst b/diskimage_builder/elements/debian-minimal/README.rst index f1931159f..ac2591afa 100644 --- a/diskimage_builder/elements/debian-minimal/README.rst +++ b/diskimage_builder/elements/debian-minimal/README.rst @@ -2,48 +2,51 @@ debian-minimal ============== -Create a minimal image based on Debian. We default to unstable but `DIB_RELEASE` -can be set to any series of Debian. +Create a minimal image based on Debian. We default to unstable but +``DIB_RELEASE`` can be set to any series of Debian. There are two ways to configure apt-sources: 1. Using the standard way of defining the default, backports, updates and security repositories is the default. In this case you can overwrite the two environment variables to adapt the behavior: - `DIB_DISTRIBUTION_MIRROR`: the mirror to use - default: http://ftp.us.debian.org/debian - `DIB_DEBIAN_COMPONENTS`: (default) `main` - a comma separated list of components. For Debian this can be - e.g. `main,contrib,non-free`. - Note it is not recommended to use http://httpredir.debian.org/ for - `DIB_DISTRIBUTION_MIRROR` due to how unreliable it is. Be sure to - select a mirror from the official mirror list: + * ``DIB_DISTRIBUTION_MIRROR``: the mirror to use (default: + ``__) - https://www.debian.org/mirror/list + * ``DIB_DEBIAN_COMPONENTS``: (default: ``main``) a comma + separated list of components. For Debian this can be + e.g. ``main,contrib,non-free``. - By default only `main` component is used. If - `DIB_DEBIAN_COMPONENTS` (comma separated) from the `debootstrap` - element has been set, that list of components will be used instead. + Note it is not recommended to use + ``__ for ``DIB_DISTRIBUTION_MIRROR`` + due to how unreliable it is. Be sure to select a mirror from the + official mirror list at ``__ - Backports, updates and security are included unless `DIB_RELEASE` - is `unstable`. + By default only the ``main`` component is used. If + ``DIB_DEBIAN_COMPONENTS`` (comma separated) from the + ``debootstrap`` element has been set, that list of components will + be used instead. + + Backports, updates and security are included unless ``DIB_RELEASE`` + is ``unstable``. + +2. Complete configuration given in the variable ``DIB_APT_SOURCES_CONF``. -2. Complete configuration given in the variable - `DIB_APT_SOURCES_CONF`. Each line contains exactly one entry for the sources.list.d - directory. - The first word must be the logical name (which is used as file name - with `.list` automatically appended), followed by a colon `:`, - followed by the complete repository specification. - Example: - DIB_APT_SOURCES_CONF=\ - "default:deb http://10.0.0.10/ stretch main contrib - mysecurity:deb http://10.0.0.10/ stretch-security main contrib" + directory. The first word must be the logical name (which is used + as file name with ``.list`` automatically appended), followed by a + colon ``:``, followed by the complete repository specification. + + .. code-block:: bash + + DIB_APT_SOURCES_CONF=\ + "default:deb http://10.0.0.10/ stretch main contrib + mysecurity:deb http://10.0.0.10/ stretch-security main contrib" If necessary, a custom apt keyring and debootstrap script can be -supplied to the `debootstrap` command via `DIB_APT_KEYRING` and -`DIB_DEBIAN_DEBOOTSTRAP_SCRIPT` respectively. Both options require the +supplied to the ``debootstrap`` command via ``DIB_APT_KEYRING`` and +``DIB_DEBIAN_DEBOOTSTRAP_SCRIPT`` respectively. Both options require the use of absolute rather than relative paths. Use of this element will also require the tool 'debootstrap' to be @@ -51,26 +54,26 @@ available on your system. It should be available on Ubuntu, Debian, and Fedora. It is also recommended that the 'debian-keyring' package be installed. -The `DIB_OFFLINE` or more specific `DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE` +The ``DIB_OFFLINE`` or more specific ``DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE`` variables can be set to prefer the use of a pre-cached root filesystem tarball. -The `DIB_DEBOOTSTRAP_EXTRA_ARGS` environment variable may be used to +The ``DIB_DEBOOTSTRAP_EXTRA_ARGS`` environment variable may be used to pass extra arguments to the debootstrap command used to create the -base filesystem image. If --keyring is is used in `DIB_DEBOOTSTRAP_EXTRA_ARGS`, -it will override `DIB_APT_KEYRING` if that is used as well. +base filesystem image. If --keyring is is used in ``DIB_DEBOOTSTRAP_EXTRA_ARGS``, +it will override ``DIB_APT_KEYRING`` if that is used as well. -For further information about `DIB_DEBIAN_DEBOOTSTRAP_SCRIPT` , -`DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE` and `DIB_DEBOOTSTRAP_EXTRA_ARGS` +For further information about ``DIB_DEBIAN_DEBOOTSTRAP_SCRIPT`` , +``DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE`` and ``DIB_DEBOOTSTRAP_EXTRA_ARGS`` please consult "README.rst" of the debootstrap element. ------------------- Note on ARM systems ------------------- -Because there is not a one-to-one mapping of `ARCH` to a kernel package, if +Because there is not a one-to-one mapping of ``ARCH`` to a kernel package, if you are building an image for ARM on debian, you need to specify which kernel -you want in the environment variable `DIB_ARM_KERNEL`. For instance, if you want -the `linux-image-mx5` package installed, set `DIB_ARM_KERNEL` to `mx5`. +you want in the environment variable ``DIB_ARM_KERNEL``. For instance, if you want +the ``linux-image-mx5`` package installed, set ``DIB_ARM_KERNEL`` to ``mx5``. .. element_deps:: diff --git a/diskimage_builder/elements/fedora/package-installs.yaml b/diskimage_builder/elements/fedora/package-installs.yaml index 2356d1663..1b298c892 100644 --- a/diskimage_builder/elements/fedora/package-installs.yaml +++ b/diskimage_builder/elements/fedora/package-installs.yaml @@ -17,7 +17,7 @@ openssl: # dependency of the python-paste package needed for the heat element, # this seems to be conflicting and causing the image building process to # fail. The problem is hapenning on a Fedora 18 system. -python-pyopenssl: +pyOpenSSL: # Workaround for: # https://bugzilla.redhat.com/show_bug.cgi?id=1066983 diff --git a/diskimage_builder/elements/manifests/README.rst b/diskimage_builder/elements/manifests/README.rst index d707c1b06..14065f144 100644 --- a/diskimage_builder/elements/manifests/README.rst +++ b/diskimage_builder/elements/manifests/README.rst @@ -1,13 +1,25 @@ ========= manifests ========= -Copy any manifests generated into the build area post-image creation -This element should be a dependency of any element that writes a manifest -into the `DIB_MANIFEST_IMAGE_DIR`, which defaults to `/etc/dib-manifests`. -This is created in extra-data.d rather than pre-install.d to allow the -source-repositories element to make use of it +An framework for saving manifest information generated during the +build for later inspection. Manifests are kept in the final image and +also copied to the build area post-image creation. -The manifests are copied to `DIB_MANIFEST_SAVE_DIR`, which defaults to -`${IMAGE_NAME}.d/`, resulting in the manifests being available as -`${IMAGE_NAME}.d/dib-manifests` by default +Elements that wish to save any form of manifest should depend on this +element and can save their data to into the ``DIB_MANIFEST_IMAGE_DIR`` ( +which defaults to ``/etc/dib-manifests``). Note this is created in +``extra-data.d`` rather than ``pre-install.d`` to allow the +``source-repositories`` element to make use of it + +The manifests are copied to ``DIB_MANIFEST_SAVE_DIR``, which defaults +to ``${IMAGE_NAME}.d/``, resulting in the manifests being available as +``${IMAGE_NAME}.d/dib-manifests`` by default after the build. + +Extra status +------------ + +This element will also add the files ``dib_environment`` and +``dib_arguments`` to the manifest recording the ``diskimage-builder`` +specific environment (``DIB_*`` variables) and command-line arguments +respectively. diff --git a/diskimage_builder/elements/manifests/cleanup.d/01-copy-manifests-dir b/diskimage_builder/elements/manifests/cleanup.d/01-copy-manifests-dir index 6248e829f..27d861210 100755 --- a/diskimage_builder/elements/manifests/cleanup.d/01-copy-manifests-dir +++ b/diskimage_builder/elements/manifests/cleanup.d/01-copy-manifests-dir @@ -1,6 +1,7 @@ #!/bin/bash # # Copyright 2014 Hewlett-Packard Development Company, L.P. +# Copyright 2017 Andreas Florath (andreas@florath.net) # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain @@ -21,15 +22,15 @@ fi set -eu set -o pipefail -if [ -d $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR} ]; then - # Move the dib_environment and dib_arguments files into the manifests dir - if [ -e $TMP_MOUNT_PATH/etc/dib_arguments ]; then - sudo mv $TMP_MOUNT_PATH/etc/dib_arguments $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR} - fi - if [ -e $TMP_MOUNT_PATH/etc/dib_environment ]; then - sudo mv $TMP_MOUNT_PATH/etc/dib_environment $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR} - fi - mkdir -p ${DIB_MANIFEST_SAVE_DIR} - cp --no-preserve=ownership -rv $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR} \ - ${DIB_MANIFEST_SAVE_DIR} -fi +MANIFEST_IMAGE_PATH=${TMP_MOUNT_PATH}/${DIB_MANIFEST_IMAGE_DIR} + +# Double check: directory must be created in extra-data.d/20-manifest-dir +[ -d ${MANIFEST_IMAGE_PATH} ] || { + echo "Error: MANIFEST_IMAGE_PATH [${MANIFEST_IMAGE_PATH}] does not exist"; + exit 1; } + +echo "$DIB_ENV" | sudo dd of=${MANIFEST_IMAGE_PATH}/dib_environment # dib-lint: safe_sudo +echo "$DIB_ARGS" | sudo dd of=${MANIFEST_IMAGE_PATH}/dib_arguments # dib-lint: safe_sudo + +mkdir -p ${DIB_MANIFEST_SAVE_DIR} +cp --no-preserve=ownership -rv ${MANIFEST_IMAGE_PATH} ${DIB_MANIFEST_SAVE_DIR} diff --git a/diskimage_builder/elements/package-installs/bin/package-installs-v2 b/diskimage_builder/elements/package-installs/bin/package-installs-v2 index 305e2a666..e475bc247 100755 --- a/diskimage_builder/elements/package-installs/bin/package-installs-v2 +++ b/diskimage_builder/elements/package-installs/bin/package-installs-v2 @@ -14,6 +14,8 @@ # License for the specific language governing permissions and limitations # under the License. +from __future__ import print_function + import argparse import json import subprocess @@ -23,18 +25,19 @@ import sys # run a command, return output # if follow is set, output will be echoed to stdout def process_output(cmdline, follow=False): - proc = subprocess.Popen(cmdline, stdout=subprocess.PIPE) + proc = subprocess.Popen(cmdline, stdout=subprocess.PIPE, + universal_newlines=True) if follow: print("Running command: %s" % cmdline) out = "" with proc.stdout: - for line in iter(proc.stdout.readline, b''): - out += line.decode('utf-8') - print("> %s" % line.strip()) + for line in iter(proc.stdout.readline, ''): + out += line + print("> %s" % line, end="") proc.wait() - print("> -- done") + print("returncode: %d" % proc.returncode) else: - out = proc.communicate()[0].decode('utf-8') + out = proc.communicate()[0] if proc.returncode: e = subprocess.CalledProcessError(proc.returncode, cmdline) diff --git a/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip b/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip index 4efd1a213..704fa338b 100755 --- a/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip +++ b/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip @@ -19,7 +19,7 @@ if [[ $DISTRO_NAME =~ (centos|fedora) ]]; then ${YUM:-yum} install -y python-virtualenv python-pip python-setuptools # install pip; this overwrites packaged pip - python /tmp/get-pip.py + /usr/local/bin/dib-python /tmp/get-pip.py # pip and setuptools are closely related; we want to ensure the # latest for sanity. Because distro packages don't include enough @@ -50,6 +50,6 @@ if [[ $DISTRO_NAME =~ (centos|fedora) ]]; then fi echo "exclude=python-virtualenv,python-pip,python-setuptools" >> ${conf} else - python /tmp/get-pip.py + /usr/local/bin/dib-python /tmp/get-pip.py pip install virtualenv fi diff --git a/elements/pip-and-virtualenv/test-elements/source-install-fedora/element-deps b/diskimage_builder/elements/pip-and-virtualenv/test-elements/source-install-fedora/element-deps similarity index 100% rename from elements/pip-and-virtualenv/test-elements/source-install-fedora/element-deps rename to diskimage_builder/elements/pip-and-virtualenv/test-elements/source-install-fedora/element-deps diff --git a/elements/pip-and-virtualenv/test-elements/source-install-fedora/environment.d/10-source-installtype-pip-and-virtualenv b/diskimage_builder/elements/pip-and-virtualenv/test-elements/source-install-fedora/environment.d/10-source-installtype-pip-and-virtualenv similarity index 100% rename from elements/pip-and-virtualenv/test-elements/source-install-fedora/environment.d/10-source-installtype-pip-and-virtualenv rename to diskimage_builder/elements/pip-and-virtualenv/test-elements/source-install-fedora/environment.d/10-source-installtype-pip-and-virtualenv diff --git a/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/element-deps b/diskimage_builder/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/element-deps similarity index 100% rename from elements/pip-and-virtualenv/test-elements/source-install-ubuntu/element-deps rename to diskimage_builder/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/element-deps diff --git a/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/environment.d/10-source-installtype-pip-and-virtualenv b/diskimage_builder/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/environment.d/10-source-installtype-pip-and-virtualenv similarity index 100% rename from elements/pip-and-virtualenv/test-elements/source-install-ubuntu/environment.d/10-source-installtype-pip-and-virtualenv rename to diskimage_builder/elements/pip-and-virtualenv/test-elements/source-install-ubuntu/environment.d/10-source-installtype-pip-and-virtualenv diff --git a/diskimage_builder/elements/python-brickclient/README.rst b/diskimage_builder/elements/python-brickclient/README.rst index 24612815d..aa1850860 100644 --- a/diskimage_builder/elements/python-brickclient/README.rst +++ b/diskimage_builder/elements/python-brickclient/README.rst @@ -1,34 +1,48 @@ python-brickclient ================== -* This element is aimed for providing cinder local attach/detach functionality. +* This element is aimed for providing cinder local attach/detach + functionality. + * Currently the feature has a dependency on a known bug - "https://launchpad.net/bugs/1623549", which has been resolved and will be part - of the upstream with the next release of python-brick-cinderclient-ext. - Note: Current version of python-brick-cinderclient-ext i.e. 0.2.0 requires and update - to be made in Line32 fo below script. - /usr/share/python-brickclient/venv/lib/python2.7/site-packages/brick_cinderclient_ext/__init__.py - update "brick-python-cinderclient-ext" to "python-brick-cinderclient-ext". + ``__, which has been resolved + and will be part of the upstream with the next release of + ``python-brick-cinderclient-ext``. Note: Current version of + ``python-brick-cinderclient-ext`` i.e. 0.2.0 requires and update to + be made in Line32 for + ``/usr/share/python-brickclient/venv/lib/python2.7/site-packages/brick_cinderclient_ext/__init__.py``: + update ``brick-python-cinderclient-ext`` to + ``python-brick-cinderclient-ext``. -* Usage: - Pass the below shell script to parameter 'user-data' and set 'config-drive=true' - at the time of provisioning the node via nova-boot to make cinder local - attach/detach commands talk to your cloud controller. - [Example of Config Drive Script] - #!/bin/bash - FILE="/etc/bash.bashrc" - [ ! -f "$FILE" ] && touch "$FILE" - echo 'export OS_AUTH_URL="http://:5000/v2.0"' >> "$FILE" - echo 'export OS_PASSWORD="password"' >> "$FILE" - echo 'export OS_USERNAME="demo"' >> "$FILE" - echo 'export OS_TENANT_NAME="demo"' >> "$FILE" - echo 'export OS_PROJECT_NAME="demo"' >> "$FILE" - exec bash - To attach: /usr/share/python-brickclient/venv/bin/cinder local-attach - To detach: /usr/share/python-brickclient/venv/bin/cinder local-detach +Usage +----- -* Alternatively, the same action can be completed manually at the node which does - not require setting up of config drive such as: - /usr/share/python-brickclient/venv/bin/cinder --os-username demo --os-password \ - password --os-tenant-name demo --os-project-name demo \ - --os-auth-url=http://:5000/v2.0 local-attach +Pass the below shell script to parameter ``user-data`` and set +``config-drive=true`` at the time of provisioning the node via +nova-boot to make cinder local attach/detach commands talk to your +cloud controller. + +.. code-block:: bash + + #!/bin/bash + FILE="/etc/bash.bashrc" + [ ! -f "$FILE" ] && touch "$FILE" + echo 'export OS_AUTH_URL="http://:5000/v2.0"' >> "$FILE" + echo 'export OS_PASSWORD="password"' >> "$FILE" + echo 'export OS_USERNAME="demo"' >> "$FILE" + echo 'export OS_TENANT_NAME="demo"' >> "$FILE" + echo 'export OS_PROJECT_NAME="demo"' >> "$FILE" + exec bash + + To attach: ``/usr/share/python-brickclient/venv/bin/cinder local-attach `` + To detach: ``/usr/share/python-brickclient/venv/bin/cinder local-detach `` + +Alternatively, the same action can be completed manually at the node +which does not require setting up of config drive such as: + +.. code-block:: bash + + /usr/share/python-brickclient/venv/bin/cinder \ + --os-username demo --os-password password \ + --os-tenant-name demo --os-project-name demo \ + --os-auth-url=http://:5000/v2.0 local-attach diff --git a/diskimage_builder/elements/python-brickclient/package-installs.yaml b/diskimage_builder/elements/python-brickclient/package-installs.yaml index 7e371d23e..754307f71 100644 --- a/diskimage_builder/elements/python-brickclient/package-installs.yaml +++ b/diskimage_builder/elements/python-brickclient/package-installs.yaml @@ -1,3 +1,8 @@ libssl-dev: libffi-dev: python-dev: + installtype: source + dib_python_version: 2 +python3-dev: + installtype: source + dib_python_version: 3 diff --git a/diskimage_builder/elements/python-brickclient/pkg-map b/diskimage_builder/elements/python-brickclient/pkg-map index 9f9058e19..a153c92b6 100644 --- a/diskimage_builder/elements/python-brickclient/pkg-map +++ b/diskimage_builder/elements/python-brickclient/pkg-map @@ -1,13 +1,15 @@ { "family": { "redhat": { - "python-dev": "python2-devel", + "python2-dev": "python2-devel", + "python3-dev": "python3-devel", "libssl-dev": "openssl-devel", "libffi-dev": "libffi-devel" } }, "default": { - "python-dev": "python-dev", + "python2-dev": "python2-dev", + "python3-dev": "python3-dev", "libssl-dev": "libssl-dev", "libffi-dev": "libffi-dev" } diff --git a/diskimage_builder/elements/redhat-common/bin/map-packages b/diskimage_builder/elements/redhat-common/bin/map-packages index 0f0da702f..37fa514db 100755 --- a/diskimage_builder/elements/redhat-common/bin/map-packages +++ b/diskimage_builder/elements/redhat-common/bin/map-packages @@ -22,6 +22,13 @@ import sys # Manually maintained for brevity; consider making this compiled from # distromatch or other rich data sources. # Debian name on the left, Fedora/RHEL on the right. + +# +# !!! DO NOT ADD ANY ENTRIES TO THIS FILE !!! +# +# This global list has been deprecated by the pkg-map element. New +# package mappings should go in pkg-map files inside each element. +# package_map = { 'apache2': 'httpd', 'arping': 'iputils', @@ -81,13 +88,30 @@ package_map = { 'openstack-neutron-dhcp-agent': 'openstack-neutron', } -print("WARNING: map-packages is deprecated. Please use the pkg-map element.", - file=sys.stderr) - +deprecated = [] for arg in sys.argv[1:]: if arg not in package_map and arg.endswith('-dev'): # convert -dev into devel - print('%s%s' % (arg, 'el')) + converted = '%s%s' % (arg, 'el') + deprecated.append((arg, converted)) + print(converted) else: - print(package_map.get(arg, arg)) + converted = package_map.get(arg, arg) + if converted != arg: + deprecated.append((arg, converted)) + print(converted) + +if deprecated: + print("WARNING: The following packages were re-mapped by " + "redhat-common map-packages\n" + "They should be converted to pkg-map:", file=sys.stderr) + for arg, converted in deprecated: + print(" %s -> %s" % (arg, converted), file=sys.stderr) + sys.exit(0) + + +# Tell emacs to use python-mode +# Local variables: +# mode: python +# End: diff --git a/diskimage_builder/elements/rhel/bin/map-packages b/diskimage_builder/elements/rhel/bin/map-packages index 814d2e47a..c14207e16 100755 --- a/diskimage_builder/elements/rhel/bin/map-packages +++ b/diskimage_builder/elements/rhel/bin/map-packages @@ -18,6 +18,14 @@ import sys # Manually maintained for brevity; consider making this compiled from # distromatch or other rich data sources. # Debian name on the left, RHEL on the right. + +# +# !!! DO NOT ADD ANY ENTRIES TO THIS FILE !!! +# +# This global list has been deprecated by the pkg-map element. New +# package mappings should go in pkg-map files inside each element. +# + package_map = { 'augeas-tools': 'augeas', 'build-essential': 'make automake gcc gcc-c++ kernel-devel', @@ -40,9 +48,18 @@ package_map = { 'vlan': 'vconfig', } -print("WARNING: map-packages is deprecated. Please use the pkg-map element.", - file=sys.stderr) - +deprecated = [] for arg in sys.argv[1:]: - print(package_map.get(arg, arg)) + mapped = package_map.get(arg, arg) + if mapped != arg: + deprecated.append((arg, mapped)) + print(mapped) + +if deprecated: + print("WARNING: The following packages were re-mapped by " + "rhel map-packages.\n" + "They should be converted to pkg-map:\n", file=sys.stderr) + for arg, converted in deprecated: + print(" %s -> %s" % (arg, converted), file=sys.stderr) + sys.exit(0) diff --git a/diskimage_builder/elements/rhel7/README.rst b/diskimage_builder/elements/rhel7/README.rst index 11536af01..f387b006e 100644 --- a/diskimage_builder/elements/rhel7/README.rst +++ b/diskimage_builder/elements/rhel7/README.rst @@ -13,7 +13,10 @@ https://access.redhat.com/downloads/content/69/ver=/rhel---7/7.1/x86_64/product- Then before running the image build, define DIB_LOCAL_IMAGE (replace the file name with the one downloaded, if it differs from the example):: - export DIB_LOCAL_IMAGE=rhel-guest-image-7.1-20150224.0.x86_64.qcow2 + +.. code-block:: bash + + export DIB_LOCAL_IMAGE=rhel-guest-image-7.1-20150224.0.x86_64.qcow2 The downloaded file will then be used as the basis for any subsequent image builds. diff --git a/diskimage_builder/elements/ubuntu-common/README.rst b/diskimage_builder/elements/ubuntu-common/README.rst new file mode 100644 index 000000000..c50160bd6 --- /dev/null +++ b/diskimage_builder/elements/ubuntu-common/README.rst @@ -0,0 +1,5 @@ +ubuntu-common +============= + +This element holds configuration and scripts that are common for all +Ubuntu images. diff --git a/diskimage_builder/elements/base/install.d/80-disable-rfc3041 b/diskimage_builder/elements/ubuntu-common/install.d/80-disable-rfc3041 similarity index 64% rename from diskimage_builder/elements/base/install.d/80-disable-rfc3041 rename to diskimage_builder/elements/ubuntu-common/install.d/80-disable-rfc3041 index b57ccbfbe..af671c234 100755 --- a/diskimage_builder/elements/base/install.d/80-disable-rfc3041 +++ b/diskimage_builder/elements/ubuntu-common/install.d/80-disable-rfc3041 @@ -21,7 +21,17 @@ if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then fi set -e -# This will disable the disable Privacy extensions for IPv6 (RFC3041) +# This will disable the privacy extensions for IPv6 (RFC4941) +# (RFC4941 obsoletes RFC3041) +# +# Note that depending on the implementation of how the sysctl +# settings are applied there is a race condition: there might be +# a small time-frame where the original Ubuntu settings from +# /etc/sysctl.d/10-ipv6-privacy.conf are already applied but the +# new settings from /etc/sysctl.d/99-cloudimg-ipv6.conf +# not yet. +# If during this time all started network interfaces will +# use the privacy extension. cat > /etc/sysctl.d/99-cloudimg-ipv6.conf <=F22 YUM=${YUM:-yum} +# save global xtrace state +_xtrace=$(set +o | grep xtrace) + SCRIPTNAME=$(basename $0) function show_options () { echo "Usage: $SCRIPTNAME [package ...]" @@ -103,7 +106,7 @@ if [ -n "$WHITELIST" ]; then exit 0 fi - echo "Running install-packages ${ACTION}. Package list: $PKGS" + echo "Running install-packages ${ACTION}." if [ "$ACTION" == "download" ]; then mkdir -p $DOWNLOAD_PATH @@ -130,6 +133,7 @@ if [ -n "$WHITELIST" ]; then # point. # # [1] https://bugzilla.redhat.com/show_bug.cgi?id=965567 + set -o xtrace ${YUM} -v -y $ACTION $EXTRA_ARGS $PKGS if [ "$ACTION" == "install" ]; then @@ -138,6 +142,7 @@ if [ -n "$WHITELIST" ]; then dnf mark install $PKGS fi fi + $_xtrace # probably not the right place for this; but python-pip package on # fedora/rh calls pip "pip-python" while the rest of the work diff --git a/diskimage_builder/elements/zypper-minimal/README.rst b/diskimage_builder/elements/zypper-minimal/README.rst index d9c8fd323..eb1c2153e 100644 --- a/diskimage_builder/elements/zypper-minimal/README.rst +++ b/diskimage_builder/elements/zypper-minimal/README.rst @@ -3,7 +3,7 @@ zypper-minimal ============== Base element for creating minimal SUSE-based images -This element is incomplete by itself so you probaby want to use it along +This element is incomplete by itself so you probably want to use it along with the opensuse-minimal one. It requires 'zypper' to be installed on the host. diff --git a/diskimage_builder/lib/common-functions b/diskimage_builder/lib/common-functions index 6cdfd7e67..b1f770588 100644 --- a/diskimage_builder/lib/common-functions +++ b/diskimage_builder/lib/common-functions @@ -77,6 +77,7 @@ function copy_hooks_not_overwrite () { test -d $TMP_HOOKS_PATH/$_DIR || mkdir $TMP_HOOKS_PATH/$_DIR for _HOOK in $(ls $1); do if [ ! -f $TMP_HOOKS_PATH/$_DIR/$_HOOK ]; then + echo "Copying hooks $1/$_HOOK" cp -t $TMP_HOOKS_PATH/$_DIR -a $1/$_HOOK else echo "There is a duplicated hook in your elements: $_ELEMENT/$_DIR/$_HOOK" @@ -220,6 +221,7 @@ function run_d() { check_element check_break before-$1 ${break_cmd:-bash} if [ -d ${TMP_HOOKS_PATH}/$1.d ] ; then + echo "Running hooks from ${TMP_HOOKS_PATH}/$1.d" if [ -n "$2" ]; then dib-run-parts ${TMP_HOOKS_PATH}/$1.d | tee $2 if [[ ${PIPESTATUS[0]} != 0 ]]; then diff --git a/diskimage_builder/lib/disk-image-create b/diskimage_builder/lib/disk-image-create index ea0f68354..5f9fcb7ab 100644 --- a/diskimage_builder/lib/disk-image-create +++ b/diskimage_builder/lib/disk-image-create @@ -52,7 +52,7 @@ function show_options () { echo " File types should be comma separated. VHD outputting requires the vhd-util" echo " executable be in your PATH. ACI outputting requires the ACI_MANIFEST " echo " environment variable be a path to a manifest file." - echo " -x -- turn on tracing (use -x -x for very detailed tracing)" + echo " -x -- turn on tracing (use -x -x for very detailed tracing)." echo " -u -- uncompressed; do not compress the image - larger but faster" echo " -c -- clear environment before starting work" echo " --checksum -- generate MD5 and SHA256 checksum files for the created image" @@ -107,6 +107,7 @@ function show_version() { python -c "from diskimage_builder import version; print(version.version_info.version_string())" } +DIB_DEBUG_TRACE=${DIB_DEBUG_TRACE:-0} INSTALL_PACKAGES="" IMAGE_TYPES=("qcow2") COMPRESS_IMAGE="true" @@ -128,7 +129,7 @@ while true ; do -t) IFS="," read -a IMAGE_TYPES <<< "$2"; export IMAGE_TYPES ; shift 2 ;; -h|--help) show_options; exit 0;; --version) show_version; exit 0;; - -x) shift; export DIB_DEBUG_TRACE=$(( $DIB_DEBUG_TRACE + 1 )); set -x;; + -x) shift; DIB_DEBUG_TRACE=$(( $DIB_DEBUG_TRACE + 1 ));; -u) shift; export COMPRESS_IMAGE="";; -c) shift ; export CLEAR_ENV=1;; -n) shift; export SKIP_BASE="1";; @@ -151,6 +152,8 @@ while true ; do esac done +export DIB_DEBUG_TRACE + export DIB_IMAGE_CACHE=${DIB_IMAGE_CACHE:-~/.cache/image-create} mkdir -p $DIB_IMAGE_CACHE @@ -170,7 +173,6 @@ function _ps4 { export -f _ps4 export PS4='+ $(_ps4): ' - source $_LIB/img-defaults source $_LIB/common-functions source $_LIB/img-functions @@ -189,6 +191,11 @@ if [ -z "$*" ]; then fi arg_to_elements "$@" +# start tracing after most boilerplate +if [ ${DIB_DEBUG_TRACE} -gt 0 ]; then + set -x +fi + if [ "${#IMAGE_TYPES[@]}" = "1" ]; then export IMAGE_NAME=${IMAGE_NAME%%\.${IMAGE_TYPES[0]}} fi @@ -290,8 +297,8 @@ fi unmount_image mv $TMP_BUILD_DIR/mnt $TMP_BUILD_DIR/built -# save xtrace state, as we want to turn it off to avoid spamming the -# logs with du output below. +# save xtrace state, as we always want to turn it off to avoid +# spamming the logs with du output below. xtrace=$(set +o | grep xtrace) # temp file for holding du output diff --git a/doc/source/developer/developing_elements.rst b/doc/source/developer/developing_elements.rst index 8c4b89fb3..80fb5c9a8 100644 --- a/doc/source/developer/developing_elements.rst +++ b/doc/source/developer/developing_elements.rst @@ -350,10 +350,6 @@ exported in bash). Debugging elements ------------------ -The build-time environment and command line arguments are captured by the -:doc:`../elements/base/README` element and written to ``/etc/dib_environment`` -and ``/etc/dib_arguments`` inside the image. - Export ``break`` to drop to a shell during the image build. Break points can be set either before or after any of the hook points by exporting "break=[before|after]-hook-name". Multiple break points can be specified as a @@ -366,6 +362,12 @@ comma-delimited string. Some examples: * ``break=after-error`` will break after an error during an in target hookpoint. +The :doc:`../elements/manifests/README` element will make a range of +manifest information generated by other elements available for +inspection inside and outside the built image. Environment and +command line arguments are captured as described in the documentation +and can be useful for debugging. + Images are built such that the Linux kernel is instructed not to switch into graphical consoles (i.e. it will not activate KMS). This maximises compatibility with remote console interception hardware, such as HP's iLO. diff --git a/doc/source/index.rst b/doc/source/index.rst index 7d277349c..c683ab654 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -8,7 +8,7 @@ It includes support for building images based on many major distributions and can produce cloud-images in all common formats (``qcow2``, ``vhd``, ``raw``, etc), bare metal file-system images and ram-disk images. These images are composed from the many included -`elements`; ``diskimage-builder`` acts as a framework to easily add +``elements``; ``diskimage-builder`` acts as a framework to easily add your own elements for even further customization. ``diskimage-builder`` is used extensively by the `TripleO project @@ -35,15 +35,15 @@ Issues Issues are tracked on launchpad at: - * `https://bugs.launchpad.net/diskimage-builder/+bugs` - + * ``__ Communication ------------- Communication among the diskimage-builder developers happens on IRC in -#openstack-dib on freenode and on the openstack-dev mailing list (openstack-dev@lists.openstack.org). +``#openstack-dib`` on freenode and on the ``openstack-dev`` mailing list +(``openstack-dev@lists.openstack.org``). Table of Contents @@ -55,3 +55,4 @@ Table of Contents user_guide/index developer/index elements + specs/README diff --git a/doc/source/specs/README.rst b/doc/source/specs/README.rst index 2840b9b6a..8f9f5e33c 100644 --- a/doc/source/specs/README.rst +++ b/doc/source/specs/README.rst @@ -1,10 +1,9 @@ -======= -README -======= - +================================ diskimage-builder Specifications ================================ +Overview +======== This directory is used to hold approved design specifications for changes to the diskimage-builder project. Reviews of the specs are done in gerrit, using a @@ -46,7 +45,7 @@ given release should only refer to the ``implemented`` directory. Example specifications ---------------------- -You can find an example spec in ``specs/template.rst``. +You can find an example spec in :doc:`v1/approved/v1-template` Backlog specifications ---------------------- @@ -80,3 +79,11 @@ change any of our public APIs are sometimes not required to provide a specification. The decision of whether something is trivial or not is a judgement made by the author or by consensus of the project cores, generally trying to err on the side of spec creation. + +Approved Specifications +======================= + +.. toctree:: + :glob: + + v1/approved/* diff --git a/doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst b/doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst index ec17e9eb7..81b96ffe2 100644 --- a/doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst +++ b/doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst @@ -20,7 +20,9 @@ The implementation for this proposed changed already exists, was discussed and is currently waiting for reviews [1]. To have a complete overview over the block device setup, this document is provided. + The dependencies are not implemented as they should be, because + * the spec process is currently in the phase of discussion and not finalized [2], * the implementation was finished and reviewed before the spec process @@ -137,14 +139,15 @@ size Example: -:: - ["partitioning", - {"rootdisk": { - "label": "mbr", - "partitions": - [{"name": "part-01", - "flags": ["boot"], - "size": "100%"}]}}] +.. code-block:: yaml + + ["partitioning", + {"rootdisk": { + "label": "mbr", + "partitions": + [{"name": "part-01", + "flags": ["boot"], + "size": "100%"}]}}] Security impact --------------- diff --git a/doc/source/specs/v1/approved/block-device-overview.rst b/doc/source/specs/v1/approved/block-device-overview.rst index 105d413b5..3f10cc8ee 100644 --- a/doc/source/specs/v1/approved/block-device-overview.rst +++ b/doc/source/specs/v1/approved/block-device-overview.rst @@ -123,6 +123,7 @@ level or module needs it's own spec. A first step is to reimplement the existing functionality, this contains: + #. Level 0: Local Loop module Use loop device on local image file (This is already implemented: [1]) @@ -133,6 +134,7 @@ contains: #. Level 3: Mounting As a second step the following functionality can be added: + * Level 1: LVM module * Level 2: Create File System (swap)