The `diskimage-builder` command provides a yaml file based interface
to `disk-image-create` and `ramdisk-image-create`. Every argument to
these scripts has a YAML equivalent. The command has the following
features:
- Environment values can be provided from the calling environment as
well as YAML
- All arguments are validated with jsonschema in the most appropriate
YAML type
- Schema is self-documenting and printed when running with --help
- Multiple YAML files can be specified and each file can have multiple
images defined
- Entries with duplicate image names will be merged into a single
image build, with attributes overwritten, elements appended, and
environment values updated/overwritten. A missing image name implies
the same image name as the previous entry.
- --dry-run and --stop-on-failure flags
A simple YAML defintion would resemble:
- imagename: centos-minimal
checksum: true
install-type: package
elements: [centos, vm]
- imagename: ironic-python-agent
elements:
- ironic-python-agent-ramdisk
- extra-hardware
The TripleO project has managed image build options with YAML files
and it has proved useful having git history and a diff friendly
format, specifically for the following situations:
- Managing differences between distros (centos, rhel)
- Managing changes in major distro releases (centos-8, centos-9-stream)
- Managing the python2 to python3 transition, within and across major
distro releases
Now that the TripleO toolchain is being retired this tool is being
proposed to be used for the image builds of TripleO's successor, as
well as the rest of the community.
Subsequent commits will add documentation and switch some tests to
using `diskimage-builder`.
Change-Id: I95cba3530d1b1c6c52cf547338762e33738f7225
With the removal of Python 2.x we can remove the unittest2 compat
wrappers and switch to assertCountEqual instead of assertItemsEqual
We have been able to use them since then, because
testtools required unittest2, which still included it. With testtools
removing Python 2.7 support [3][4], we will lose support for
assertItemsEqual, so we should switch to use assertCountEqual.
[1] - https://bugs.python.org/issue17866
[2] - https://hg.python.org/cpython/rev/d9921cb6e3cd
[3] - testing-cabal/testtools#286
[4] - testing-cabal/testtools#277
Change-Id: I870286a2557e41099597c22dc9747743e1077615
This causes problems for other projects incorporating dib; we don't
have a specific need for a cap.
Fix a few issues, mostly spacing or regex matches. No functional
changes.
W503 and W504 relate to leaving artithmetic operators at the start or
end of lines, and are mutually exclusive and, due to "ignore"
overriding the defaults both get enabled. It seems everyone gets this
wrong (https://gitlab.com/pycqa/flake8/issues/466). Don't take a
position on this and ignore both.
Use double # around comments including YAML snippets using "# type: "
which now gets detected as PEP484/mypy type hints.
Change-Id: I8b7ce6dee02dcce31c82427a2441c931d136ef57
This replaces the deprecated (in python 3.2) unittest.TestCase method
assertRaisesRegexp() with assertRaisesRegex().
Change-Id: I0ac482741ad4adc1127dd9e9f64c1c8101f370e5
Signed-off-by: Chuck Short <chucks@redhat.com>
The current check only validates that an element that specifies
"element-provides" doesn't conflict with a "real" element. We also
want to check this against the provides of other elements.
A real example is with a "block-device" element. There is no actual
"block-device" element; we can have multiple elements provide it
(block-device-[gpt,mbr,efi], say) but we only want one of them at a
time.
Update the unit test for this.
Change-Id: I59d4aa5f6f09e2892b213e154befa10d85e95ca3
This is code motion with some small changes to make follow-on's
easier.
test_blockdevice_mbr.py is moved alongside the other tests. It is
modified slightly to use the standard base class and remove a lot of
repeated test setup; a fixture is used for the tempdir (so it doesn't
have to be torn-down, and is removed properly on error) and the partx
args are moved into the setUp() so each test doesn't have to create
it. No functional change. renamed test_mbr.py for shortness.
test_blockdevice_utils.py is merged with existing test_utils.py. No
change to the tests.
test_blockdevice.py is removed. It isn't doing anything currently; to
work it will need to take an approach based more on mocking of calls
that require elevated permissions. It's in history if we need it.
Change-Id: I87b1ea94afaaa0b44e6a57b9d073f95a63a04cf0
Currently we keep a global list of mount-points defined in the
configuration and automatically setup dependencies between mount nodes
based on their global "mount order" (i.e. higher directories mount
first).
The current method for achieving this is roughly to add the mount
points to a dictionary indexed my mount-point, then at "get_edge()"
call build the sorted list ... unless it has already been built
because this gets called for every node.
It seems much simpler to simply keep a sorted list of the
MountPointNode objects as we add them. We don't need to implement a
sorting algorithm then, we can just use sort() and implement __lt__
for the nodes.
I believe the existing mount-order unit testing is sufficient; I'm
struggling to find a valid configuration where the mount-order is
*not* correctly specified in the configuration graph.
Change-Id: Idc05cdf42d95e230b9906773aa2b4a3b0f075598
As described in pep282 [1], the variable part of a log message
should be passed in via parameter. In this case the parameters
are evaluated only when they need to be.
This patch fixes (unifies) this for DIB.
A check using pylint was added that this kind of passing parameters to
the logging subsystem is enforced in future. As a blueprint a similar
(stripped-down) approach from cinder [2] was used.
[1] https://www.python.org/dev/peps/pep-0282/
[2] https://github.com/openstack/cinder/blob/master/tox.ini
Change-Id: I2d7bcc863e4e9583d82d204438b3c781ac99824e
Signed-off-by: Andreas Florath <andreas@florath.net>
This is an initial creation of pylint with a basic indent checker.
Small issues corrected. Job added to gate with
Ib554a284e92583cc1d6a5c2219b3922852ca4c73
Change-Id: I7e24d8348db3aef79e1395d12692199a1f80161a
Co-Authored-By: Andreas Florath <andreas@florath.net>
This completes the transitions started in
Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
The new file plugin.py is the place to start with this change. The
abstract base classes PluginBase and NodeBase are heavily documented.
NodeBase essentially replaces Digraph.Node
The changes in level?/*.py make no functional changes, but are just
refactoring to implement the plugin and node classes consistently.
Additionally we have added asserts during parsing & generation to
ensure plugins are implemented PluginBase, and get_nodes() is always
returning NodeBase objects for the graph.
Change-Id: Ie648e9224749491260dea65d7e8b8151a6824b9c
Because in some cases (e.g. partitioning) the order is needed,
add weights to the digraph to get an (somewhat) stable
topological sort.
Change-Id: I5ef1acc6338ac93c593faa0eafe26cbed42ed887
Signed-off-by: Andreas Florath <andreas@florath.net>
The await function is essentially a non-standard check_output call.
Let's use standard calls to increase maintainability.
Change-Id: I2c25e1cd7122791fcaa86b46bd801e661471bc9e
As this method can be introduced without any dependency,
provide it on an independent change to simplify reviews.
This is a partial refactor based on
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: Idaf3d2b3b3e23d0b9d6bc071d67b961a829ae422
Co-Authored-By: Andreas Florath <andreas@florath.net>
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.
This patch implements partitioning handling.
Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
expand_dependencies() was a public interface so we should try and
preserve backwards compat. However, since the interface is really
broken, add a new exported function "get_elements" that instack can
switch to. This returns the canonical list of elements without
duplicates, and gives the path to each element too.
This highlighted that the unit tests were really a bit wrong. They're
testing inner functions when we have an "API" in the get_elements()
function. Convert all unit-tests to use this function instead. Since
this is a library call, convert the sys.exit() calls to raised
exceptions.
Refactor the variable output into a separate function so we can do a
sanity check on it.
The added flake8 ignores are for the "over-indented for ... indent"
which happens a lot with these new longer lines. Most other projects
ignore them.
This is an alternative proposal to
I15609389c18adf3017220fc94552514d195b323a
Change-Id: If97bcd21e45de1b5ed91023fdc441a4617051a6b
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
This is a re-factor of element_dependencies to achieve two things --
centralising override policy and storing path names.
Firstly we want to make the override policy for elements completely
explicit. Currently, elements that wish to copy parts of other
elements walk ELEMENTS_PATH themselves and look for elements in
IMAGE_ELEMENT. How they handle duplicate elements can differ, leading
to inconsistent behaviour.
We introduce logic in element-info to find elements in each of the
directories in ELEMENT_PATHS in *reverse* order -- that is to say,
earlier entries in the paths will overwrite later ones.
For example
ELEMENT_PATHS=foo:bar:baz
will mean that "foo/element" will override "baz/element", since "foo"
is first. This should be sane to anyone familiar with $PATH.
Documentation is clarified around this point and a test-case is added.
The second thing is that we want to keep the complete path of the
elements we have chosen. We want the aforementioned elements that
walk the element list to use these canonical paths to pickup files;
this way they don't need to make local decisions about element
overrides, but can simply iterate a list and copy/merge files if they
exist.
A follow-on change (I7092e1845942f249175933d67ab121188f3511fd) will
expose this data in a separate variable that can be parsed by elements
(a further follow-on I0a64b45e9f2cfa28e84b2859d76b065a6c4590f0
modifies the elements to use this information). Thus this does not
change the status-quo -- elements that are walking ELEMENTS_PATH
themselves and can/will continue doing that.
Change-Id: I2a29861c67de2d25c595cb35d850e92807d26ac6
I got quite confused what this was trying to tell me at first. It's
saying that you requested an element but another element already
provides that element, so we don't know which one to choose.
To help clarify the situation to the user, keep track of what is
providing elements so we can describe where the conflict came from.
Change-Id: Ie7471ac900a8cbee5684c928badd1b8ce6d3e3cf
Use standard logging module for output. Add some basic testing of
error messages to the unit-tests. Use the logging_config module to
setup the logging for interactive use.
Change-Id: Ia23722a7bd00aba336118edb155356a3b3ef6926
This adds some fixtures to ensure the log output is what we expect,
along with ensuring we test the operation of the debug environment
variables.
Change-Id: I01c36299539b15a633b9307da5a348a5ae4e2563
This adds a common usable logging configuration for DIB, that
can be used by different python tools or elements.
This change is a factored out patch from the block device
refactoring.
Change-Id: I34d44610f74030936e7c0f602340b183baec3f1b
Signed-off-by: Andreas Florath <andreas@florath.net>
A sanity check that cache-url does the right thing when presented
with the appropriate curl return codes. This change brings in the
test code for scripts from tripleo-image-elements, which needs to
be factored out into a place both projects can use it.
I'm stubbing out curl so we don't have a dep on curl in our unit
tests, and because I've seen some strange behavior out of curl in
the gate that caused random failures in this test.
Change-Id: I31e6b06b45415feec7285511d07e65eb78d0d045
Add a new test_no_dup_filenames module to help ensure we don't have
duplicate names across elements.
Change-Id: I2af4a614f71fe58eba04e3fc50d3c0a70dd91f71
The phrase is no longer needed as of August 23, 2000 with Nicaragua's
joining of the Berne Convention.
Additionally, in at least one instance,
elements/cache-url/bin/cache-url, its existence in the file between
Copyright lines is just weird and feels misleading, even though it is
not.
Remove all of the lines, because sanity.
Change-Id: I24fd76c2b4f66b8036010b5079db39ead729abee
To ensure that we have the list of provided elements from the
command line before looking up dependencies, use a deque and
popleft() to take the elements on a first-in/first-out basis.
Change-Id: I0a2c21fa081763fd3cc8358be25dafeee1ed4718
Closes-Bug: #1303911
An element can define a list of elements provided by itself. This allows
using an alternative element and keeping element dependencies at the
same time. This will be needed for example for mariadb element which will be
an alternative to mysql element - mysql dependency can be kept everywhere
but user can choose mariadb explicitly as a replacement by putting "mysql"
into "element-provides" file.
Change-Id: I7f193612f279dd71d9f1f1edfc60009838a00c5a
This commit refactors the python code in diskimage_builder slightly. This is
in preparation for a larger review that adds more functionality to the python
code, namely the ability to apply elements to the current system as opposed to
a chroot. Further, the refactorings can stand on their own for better clarity.
They include:
- renaming elements.py to element_dependencies.py. Adds clarity about the
purpose of this module.
- updating other code for this rename.
- move tests into a tests submodule.
Change-Id: I5519cc52398e442b24e33802bae42070d64b0c1d
Takashi Kajinami