stackforge
/
novaimagebuilder
Code Issues Proposed changes

Add support for creating volume snapshots 

Add initial support for using ISO install media
Add example of Fedora 18 ks file that uses ISO install media
Add RHEL5 and RHEL6 example files
Add significant documentation to README
Add license
Split into CLI file and utility functions
This commit is contained in:
Ian McLeod 2013-04-02 23:21:48 -05:00
parent effd0c2049
commit 0436ddbe80
13 changed files with 801 additions and 165 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

161
COPYING Normal file
View File

@ -0,0 +1,161 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and
distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright
owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities
that control, are controlled by, or are under common control with that entity.
For the purposes of this definition, "control" means (i) the power, direct or
indirect, to cause the direction or management of such entity, whether by
contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising
permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including
but not limited to software source code, documentation source, and
configuration files.
"Object" form shall mean any form resulting from mechanical transformation or
translation of a Source form, including but not limited to compiled object
code, generated documentation, and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form,
made available under the License, as indicated by a copyright notice that is
included in or attached to the work (an example is provided in the Appendix
below).
"Derivative Works" shall mean any work, whether in Source or Object form, that
is based on (or derived from) the Work and for which the editorial revisions,
annotations, elaborations, or other modifications represent, as a whole, an
original work of authorship. For the purposes of this License, Derivative Works
shall not include works that remain separable from, or merely link (or bind by
name) to the interfaces of, the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original
version of the Work and any modifications or additions to that Work or
Derivative Works thereof, that is intentionally submitted to Licensor for
inclusion in the Work by the copyright owner or by an individual or Legal
Entity authorized to submit on behalf of the copyright owner. For the purposes
of this definition, "submitted" means any form of electronic, verbal, or
written communication sent to the Licensor or its representatives, including
but not limited to communication on electronic mailing lists, source code
control systems, and issue tracking systems that are managed by, or on behalf
of, the Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise designated in
writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf
of whom a Contribution has been received by Licensor and subsequently
incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of this
License, each Contributor hereby grants to You a perpetual, worldwide,
non-exclusive, no-charge, royalty-free, irrevocable copyright license to
reproduce, prepare Derivative Works of, publicly display, publicly perform,
sublicense, and distribute the Work and such Derivative Works in Source or
Object form.
3. Grant of Patent License. Subject to the terms and conditions of this
License, each Contributor hereby grants to You a perpetual, worldwide,
non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this
section) patent license to make, have made, use, offer to sell, sell, import,
and otherwise transfer the Work, where such license applies only to those
patent claims licensable by such Contributor that are necessarily infringed by
their Contribution(s) alone or by combination of their Contribution(s) with the
Work to which such Contribution(s) was submitted. If You institute patent
litigation against any entity (including a cross-claim or counterclaim in a
lawsuit) alleging that the Work or a Contribution incorporated within the Work
constitutes direct or contributory patent infringement, then any patent
licenses granted to You under this License for that Work shall terminate as of
the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the Work or
Derivative Works thereof in any medium, with or without modifications, and in
Source or Object form, provided that You meet the following conditions:
You must give any other recipients of the Work or Derivative Works a copy of
this License; and
You must cause any modified files to carry prominent notices stating that You
changed the files; and
You must retain, in the Source form of any Derivative Works that You
distribute, all copyright, patent, trademark, and attribution notices from the
Source form of the Work, excluding those notices that do not pertain to any
part of the Derivative Works; and
If the Work includes a "NOTICE" text file as part of its distribution, then any
Derivative Works that You distribute must include a readable copy of the
attribution notices contained within such NOTICE file, excluding those notices
that do not pertain to any part of the Derivative Works, in at least one of the
following places: within a NOTICE text file distributed as part of the
Derivative Works; within the Source form or documentation, if provided along
with the Derivative Works; or, within a display generated by the Derivative
Works, if and wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and do not modify the
License. You may add Your own attribution notices within Derivative Works that
You distribute, alongside or as an addendum to the NOTICE text from the Work,
provided that such additional attribution notices cannot be construed as
modifying the License. You may add Your own copyright statement to Your
modifications and may provide additional or different license terms and
conditions for use, reproduction, or distribution of Your modifications, or for
any such Derivative Works as a whole, provided Your use, reproduction, and
distribution of the Work otherwise complies with the conditions stated in this
License.
5. Submission of Contributions. Unless You explicitly state otherwise, any
Contribution intentionally submitted for inclusion in the Work by You to the
Licensor shall be under the terms and conditions of this License, without any
additional terms or conditions. Notwithstanding the above, nothing herein shall
supersede or modify the terms of any separate license agreement you may have
executed with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade names,
trademarks, service marks, or product names of the Licensor, except as required
for reasonable and customary use in describing the origin of the Work and
reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in
writing, Licensor provides the Work (and each Contributor provides its
Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied, including, without limitation, any warranties
or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any risks
associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory, whether in
tort (including negligence), contract, or otherwise, unless required by
applicable law (such as deliberate and grossly negligent acts) or agreed to in
writing, shall any Contributor be liable to You for damages, including any
direct, indirect, special, incidental, or consequential damages of any
character arising as a result of this License or out of the use or inability to
use the Work (including but not limited to damages for loss of goodwill, work
stoppage, computer failure or malfunction, or any and all other commercial
damages or losses), even if such Contributor has been advised of the
possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing the Work or
Derivative Works thereof, You may choose to offer, and charge a fee for,
acceptance of support, warranty, indemnity, or other liability obligations
and/or rights consistent with this License. However, in accepting such
obligations, You may act only on Your own behalf and on Your sole
responsibility, not on behalf of any other Contributor, and only if You agree
to indemnify, defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason of your
accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS

139
README.md
View File

@ -1,46 +1,149 @@
image-building-poc
==================
Building OS images in NOVA
==========================
This is a very early demonstration of doing native OS installs inside of Nova.
This is an early demonstration of a new image building approach for OpenStack.
To try it out
It is a command line tool that builds working OpenStack images by
running Anaconda or other native installers within Nova. In its simplest form
it requires only a kickstart or preseed file as input. All of the heavy lifting
is done inside of OpenStack instances.
1) Install the requirements listed below
2) Replace all <FIXME> lines in glance_install.sh with the details of your OpenStack setup
3) Uncomment and edit the <FIXME> lines in the *.ks files to set your own root password and VNC password.
Early discussion of this approach can be found here:
Then run:
https://wiki.openstack.org/wiki/NovaImageBuilding
./glance_install.sh fedora-18.ks
It has been developed and tested on RHEL6 and the Folsom OpenStack release installed
using packstack. However, it should work with newer host OSes and newer OpenStack releases.
To try it out install the requirements listed below then run commands like this:
(substituting the details of your own OpenStack environment where indicated)
#### Create a Fedora 18 JEOS image in glance using a network install
./create_image.py --username admin --tenant admin --password password --auth-url http://10.10.10.10:5000/v2.0 \
--glance-url http://10.10.10.10:9292/ --root-password myrootpw install_scripts/fedora-18-jeos.ks
#### Create an Ubuntu 12.04 image in glance using a network install
./create_image.py --username admin --tenant admin --password password --auth-url http://10.10.10.10:5000/v2.0 \
--glance-url http://10.10.10.10:9292/ --root-password myrootpw \
install_scripts/ubuntu-12.04-jeos.preseed
#### Create a Fedora 18 JEOS image as a volume snapshot using a network install
./create_image.py --username admin --tenant admin --password password --auth-url http://10.10.10.10:5000/v2.0 \
--glance-url http://10.10.10.10:9292/ --root-password myrootpw --create-volume \
install_scripts/fedora-18-jeos.ks
#### Create a Fedora 18 JEOS image as a volume snapshot using an install DVD pulled from a Fedora mirror
./create_image.py --username admin --tenant admin --password password --auth-url http://10.10.10.10:5000/v2.0 \
--create-volume --install-media-url \
http://mirror.pnl.gov/fedora/linux/releases/18/Fedora/x86_64/iso/Fedora-18-x86_64-DVD.iso \
--install-tree-url \
http://mirror.pnl.gov/fedora/linux/releases/18/Fedora/x86_64/os/ \
--glance-url http://10.10.10.10:9292/ --root-password myrootpw install_scripts/fedora-18-jeos-DVD.ks
#### Create a Fedora 18 JEOS image as a volume snapshot by re-using the DVD volume snapshot created above
./create_image.py --username admin --tenant admin --password password --auth-url http://10.10.10.10:5000/v2.0 \
--create-volume --install-media-snapshot <SNAPSHOT_ID_REPORTED_ABOVE> \
--install-tree-url \
http://mirror.pnl.gov/fedora/linux/releases/18/Fedora/x86_64/os/ \
--glance-url http://10.10.10.10:9292/ --root-password myrootpw install_scripts/fedora-18-jeos-DVD.ks
### What does this do?
The script generates a small syslinux-based bootable image that is used
to start unattended Anaconda or Ubuntu installations. It contains only
the initrd and vmlinuz from the install source and a syslinux.cfg file.
The installer then writes over this minimal image.
The kickstart/preseed files are passed to the installers via OpenStack
user-data and the appropriate kernel command line parameters in the
syslinux configuration file.
The script uploads this bootstrapping image to glance, launches it, and
waits for it to shut down. If shutdown occurs within the timeout period
we assume that the installer has finished and take a snapshot of the current
instance state, which is the completed install.
You can monitor progress via Anaconda's VNC support, which is enabled
in the example kickstarts under the "install_scripts" directory. The
script reports the instance IP and gives the exact invocation of
vncviewer that is needed to connect to the install.
You can do something similar with an Ubuntu install using an SSH console.
However, this feature stops the installation and waits for user input so
it is commented out in the example preseed files. See instructions in
the comments for how to enable this.
If all goes well, this will install Fedora 18 entirely within a Nova container using
the well known network install sources found in the kickstart file.
You can edit the details of the kickstart file as you wish. The only requirement
at the moment is that the install must be "url" based.
It has only been run against the packstack Folsom distribution running on RHEL6.
### Volume based images
Packstack details:
https://wiki.openstack.org/wiki/Packstack
By default the script will build a Glance backed image. If passed the
--create-volume option it will instead build a volume backed "snapshot"
image.
It should work on newer OpenStack releases and more recent OSes including Fedora 17 and 18
It requires the following OpenStack packages (tested version listed):
### ISO install media
It also contains initial support for presenting installer ISO images as
a source for installation packages. This support has only been tested for
Fedora 18 for the moment. It is somewhat limited because OpenStack currently
only allows these images to be mapped into the instance as "normal"
block devices, rather than CDROMs. Not all installers can deal with this.
(Note: When using the install media volume feature you must still pass
a "--install-tree-url" option as demonstrated in the examples above. This
is necessary to allow the script to retrieve the install kernel and ramdisk
without having to pull down a copy of the entire ISO.)
### Requirements
This script has been tested with the following OpenStack client packages:
python-glanceclient-0.5.1-1.el6.noarch
python-novaclient-2.10.0-2.el6.noarch
python-keystoneclient-0.1.3.27-1.el6.noarch
These are automatically installed when using Packstack
Newer and older versions may work.
It also requires:
python-libguestfs
syslinux
qemu-img
And benefits from having "vncviewer" from the tigervnc package.
If you want to view ongoing installs over VNC you will need:
tigervnc
### TODO
Better documentation
Better error detection and reporting
Support for more operating systems.
Support for sourcing install scripts through libosinfo
Support for enhanced block device mapping when it becomes available
Support for direct booting of kernel/ramdisk/cmdline combinations when/if it is added to Nova
Improved detection of install success or failure
Support for caching of self-install images

278
create_image.py Executable file
View File

@ -0,0 +1,278 @@
#!/usr/bin/python
#
# Copyright 2013 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import os.path
import sys
import shutil
import argparse
from tempfile import mkdtemp, NamedTemporaryFile, TemporaryFile
from image_utils import *
parser = argparse.ArgumentParser(description='Launch and snapshot a kickstart install using syslinux and Glance')
ospar=parser.add_argument_group('OpenStack Enviornment')
ospar.add_argument('--auth-url', dest='auth_url', required=True,
help='URL for keystone authorization')
ospar.add_argument('--username', dest='username', required=True,
help='username for keystone authorization')
ospar.add_argument('--tenant', dest='tenant', required=True,
help='tenant for keystone authorization')
ospar.add_argument('--password', dest='password', required=True,
help='password for keystone authorization')
ospar.add_argument('--glance-url', dest='glance_url', required=True,
help='URL for glance service')
install_media_desc="""When one of these arguments is given the install environment will contain a second
block device. The image presented on this device can come from a URL, a file or
a pre-existing volume snapshot. You may only use one of these options at a time
and you can only use them in conjunction with the 'create-volume' option."""
install_media = parser.add_argument_group('Install Media', install_media_desc)
install_media.add_argument('--install-media-url', dest='install_media_url',
help='Add an install media device using content at this URL')
install_media.add_argument('--install-media-file', dest='install_media_file',
help='Add an install media device using this file as a media image')
install_media.add_argument('--install-media-snapshot', dest='install_media_snapshot',
help='Add an install media device by creating a volume from this snapshot id')
instpar = parser.add_argument_group('Installation Parameters')
instpar.add_argument('--root-password', dest='admin_password', required=True,
help='root password for the resulting image - also used for optional remote access during install')
instpar.add_argument('--create-volume', dest='create_volume', action='store_true', default=False,
help='Create a volume snapshot instead of the default Glance snapshot (optional)')
instpar.add_argument('--install-volume-size', dest='install_volume_size', default=10,
help='Size of the install destination volume in GB (default: 10)')
instpar.add_argument('--install-tree-url', dest='install_tree_url',
help='URL for preferred network install tree (optional)')
instpar.add_argument('--distro', dest='distro',
help='distro - must be "rpm" or "ubuntu" (optional)')
instpar.add_argument('--image-name', dest='image_name',
help='name to assign newly created image (optional)')
instpar.add_argument('--leave-mess', dest='leave_mess', action='store_true', default=False,
help='Do not clean up local or remote artifacts when finished or when an error is encountered')
parser.add_argument('ks_file',
help='kickstart/install-script file to use for install')
args = parser.parse_args()
# This is a string
working_kickstart = do_pw_sub(args.ks_file, args.admin_password)
distro = detect_distro(working_kickstart)
if args.distro:
# Allow the command line distro to override our guess above
distro = args.distro
(install_tree_url, console_password, console_command, poweroff) = install_extract_bits(working_kickstart, distro)
if args.install_tree_url:
# Allow the specified tree to override anything extracted above
install_tree_url = args.install_tree_url
if args.image_name:
image_name = args.image_name
else:
image_name = "Image from ks file: %s - Date: %s" % (os.path.basename(args.ks_file), strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime()))
# Let's be nice and report as many error conditions as we can before exiting
error = False
if (args.install_media_url or args.install_media_file or args.install_media_snapshot) and not args.create_volume:
print "ERROR: You can only use install media when creating a volume snapshot image using the --create-volume option."
error = True
if (args.install_media_url and args.install_media_file) or (args.install_media_file and args.install_media_snapshot) or \
(args.install_media_url and args.install_media_snapshot):
print "ERROR: You may only specify a single install media source"
error = True
if not install_tree_url:
print "ERROR: no install tree URL specified and could not extract one from the kickstart/install-script"
error = True
if not distro:
print "ERROR: no distro specified and could not guess based on the kickstart/install-script"
error = True
if not poweroff:
if distro == "rpm":
print "ERROR: supplied kickstart file must contain a 'poweroff' line"
elif distro == "ubuntu":
print "ERROR: supplied preseed must contain a 'd-i debian-installer/exit/poweroff boolean true' line"
error = True
if error:
sys.exit(1)
# We start creating artifacts here - cleanup in finally
modified_image = None # filename
tmp_content_dir = None # directory
install_image = None # Nova image object
install_media_volume=None # cinder volume object
install_media_snapshot_id=None # UUID string
installed_instance = None # Nova instance object
finished = False # silly marker
retcode = 0
try:
# Artifact of borrowing factory code - pass this as a dict
creds = { 'username': args.username, 'tenant': args.tenant, 'password': args.password, 'auth_url': args.auth_url }
# Generate "blank" syslinux bootable mini-image
# This is the only step that strictly requires root access due to the need
# for a loopback mount to install the bootloader
generate_blank_syslinux()
# Take a copy of it
if args.create_volume:
disk_format = 'raw'
modified_image = "./syslinux_modified_%s.raw" % os.getpid()
try:
subprocess_check_output(["qemu-img","convert","-O","raw","./syslinux.qcow2",modified_image])
except:
print "Exception while converting image to raw"
raise
else:
disk_format = 'qcow2'
modified_image = "./syslinux_modified_%s.qcow2" % os.getpid()
shutil.copy("./syslinux.qcow2",modified_image)
# Generate the content to put into the image
tmp_content_dir = mkdtemp()
print "Collecting boot content for auto-install image"
generate_boot_content(install_tree_url, tmp_content_dir, distro, args.create_volume)
# Copy in the kernel, initrd and conf files into the blank boot stub using libguestfs
print "Copying boot content into a bootable syslinux image"
copy_content_to_image(tmp_content_dir, modified_image)
# Upload the resulting image to glance
print "Uploading image to glance"
install_image = glance_upload(image_filename = modified_image, image_url = None, creds = creds, glance_url = args.glance_url,
name = "INSTALL for: %s" % (image_name), disk_format=disk_format)
print "Uploaded successfully as glance image (%s)" % (install_image.id)
install_volume=None
# TODO: Make volume size configurable
if args.create_volume:
print "Converting Glance install image to a Cinder volume"
install_volume = volume_from_image(install_image.id, creds, args.glance_url, volume_size = args.install_volume_size)
if args.install_media_url or args.install_media_file:
if args.install_media_url:
print "Generating Glance image from URL: %s" % (args.install_media_url)
install_media_image = glance_upload(image_filename = None, image_url = args.install_media_url,
creds = creds, glance_url = args.glance_url, name = "FromURL: %s" % (args.install_media_url),
disk_format='raw')
else:
print "Generating Glance image from file: %s" % (args.install_media_file)
install_media_image = glance_upload(image_filename = args.install_media_file, image_url = None,
creds = creds, glance_url = args.glance_url, name = os.path.basename(args.install_media_file),
disk_format='raw')
print "Generating volume from image (%s)" % (install_media_image.id)
install_media_volume = volume_from_image(install_media_image.id, creds, args.glance_url)
print "Generating snapshot of volume (%s) to allow install media reuse" % (install_media_volume.id)
install_media_snapshot = snapshot_from_volume(install_media_volume.id, creds)
install_media_snapshot_id = install_media_snapshot.id
print "#### Future installs can reference this snapshot with the following argument:"
print " --install-media-snapshot %s" % install_media_snapshot_id
elif args.install_media_snapshot:
print "Generating working volume from snapshot (%s)" % (args.install_media_snapshot)
install_media_snapshot_id = args.install_media_snapshot
install_media_volume = volume_from_snapshot(args.install_media_snapshot, creds)
# Launch the image with the provided ks.cfg as the user data
# Optionally - spawn a vncviewer to watch the install graphically
# Poll on image status until it is SHUTDOWN or timeout
print "Launching install image"
installed_instance = launch_and_wait(install_image, install_volume, install_media_volume, working_kickstart,
os.path.basename(args.ks_file), creds, console_password, console_command)
# Take a snapshot of the now safely shutdown image
# For volume snapshots we must terminate the instance first then snapshot
# For glance/image snapshots we must _not_ terminate the instance until the snapshot is complete
print "Taking snapshot of completed install"
if args.create_volume:
print "Terminating instance (%s) in preparation for taking a snapshot of the root volume" % (installed_instance.id)
terminate_instance(installed_instance.id, creds)
installed_instance = None
finished_image_snapshot = snapshot_from_volume(install_volume.id, creds)
print "Volume-based image available from snapshot ID: %s" % (finished_image_snapshot.id)
print "Finished snapshot name is: %s" % (finished_image_snapshot.display_name)
finished = True
else:
finished_image_id = installed_instance.create_image(image_name)
print "Waiting for glance image snapshot to complete"
wait_for_glance_snapshot(finished_image_id, creds, args.glance_url)
print "Terminating instance (%s) now that snapshot is complete" % (installed_instance.id)
terminate_instance(installed_instance.id, creds)
installed_instance = None
print "Finished image snapshot ID is: %s" % (finished_image_id)
print "Finished image name is: %s" % (image_name)
finished = True
except Exception as e:
print "Uncaught exception encountered during install"
print str(e)
retcode = 1
finally:
if args.leave_mess:
print "Leaving a mess - this includes local files, local dirs, remote images, remote volumes and remote snapshots"
sys.exit(retcode)
print "Cleaning up"
try:
if tmp_content_dir:
print "Removing boot content dir"
shutil.rmtree(tmp_content_dir)
if modified_image:
print "Removing install image %s" % (modified_image)
#TODO:Note that thie is actually cacheable on a per-os-version basis
os.remove(modified_image)
if installed_instance:
# Note that under normal operation this is terminated when completing the snapshot process
print "Terminating install instance (%s)" % (installed_instance.id)
terminate_instance(installed_instance.id, creds)
if install_image:
print "Deleting Glance image (%s) used to launch install" % (install_image.id)
install_image.delete()
if install_media_volume:
print "Removing working volume containing install media"
print "Snapshot (%s) remains available for future use" % (install_media_snapshot_id)
install_media_volume.delete()
except:
print "WARNING: Exception while attempting to clean up - we may have left a mess"
retcode = 1
# For usability - reprint the most important bits from above as the last output
if finished:
print "FINISHED!"
print
print "Image Details:"
if args.create_volume:
print "Volume snapshot name: %s" % (finished_image_snapshot.display_name)
print "ID: %s" % (finished_image_snapshot.id)
else:
print "Glance image name: %s" % (image_name)
print "ID: %s" % (finished_image_id)
sys.exit(retcode)

6
glance_install.sh
View File

@ -1,6 +0,0 @@
#!/bin/sh
./create_glance_image.py --username <FIXME> --tenant <FIXME> --password <FIXME> --auth-url <FIXME> \
--glance-url <FIXME> --admin-password <FIXME> $1

269
create_glance_image.py → image_utils.py
View File

@ -26,6 +26,7 @@ import re
from string import Template
from tempfile import mkdtemp, NamedTemporaryFile, TemporaryFile
from glanceclient import client as glance_client
from cinderclient import client as cinder_client
from keystoneclient.v2_0 import client as keystone_client
from novaclient.v1_1 import client as nova_client
from time import sleep, gmtime, strftime
@ -128,7 +129,7 @@ def http_download_file(url, filename):
### Borrowed from Image Factory OpenStack plugin
def glance_upload(image_filename, creds = {'auth_url': None, 'password': None, 'strategy': 'noauth', 'tenant': None, 'username': None},
def glance_upload(image_filename = None, image_url = None, creds = {'auth_url': None, 'password': None, 'strategy': 'noauth', 'tenant': None, 'username': None},
glance_url = None, token = None, name = 'Factory Test Image', disk_format = 'raw'):
k = keystone_client.Client(username=creds['username'], password=creds['password'], tenant_name=creds['tenant'], auth_url=creds['auth_url'])
@ -136,41 +137,103 @@ def glance_upload(image_filename, creds = {'auth_url': None, 'password': None, '
if (k.authenticate()):
#Connect to glance to upload the image
glance = glance_client.Client("1", endpoint=glance_url, token=k.auth_token)
image_data = open(image_filename, "r")
image_meta = {'container_format': 'bare',
'disk_format': disk_format,
'is_public': True,
'min_disk': 0,
'min_ram': 0,
'name': name,
'data': image_data,
'properties': {'distro': 'rhel'}}
try:
image = glance.images.create(name=name)
print "Uploading to Glance"
image.update(**image_meta)
return image.id
if image_filename:
image_data = open(image_filename, "r")
image_meta['data'] = image_data
print "Uploading to Glance"
image.update(**image_meta)
elif image_url:
image_meta['copy_from'] = image_url
image.update(**image_meta)
print "Waiting for Glance to finish creating image from URL: %s" % (image_url)
while (image.status != 'active'):
if image.status == 'killed':
raise Exception("Glance error while waiting for image to generate from URL")
print '.',
sys.stdout.flush()
sleep(10)
image=glance.images.get(image.id)
return image
except Exception, e:
raise
else:
raise Exception("Unable to authenticate into glance")
def volume_from_image(image_id, creds, glance_url, volume_size = None):
k = keystone_client.Client(username=creds['username'], password=creds['password'], tenant_name=creds['tenant'], auth_url=creds['auth_url'])
if not k.authenticate():
raise Exception("Could not authenticate into keystone")
glance = glance_client.Client("1", endpoint=glance_url, token=k.auth_token)
cinder = cinder_client.Client('1', creds['username'], creds['password'], creds['tenant'], creds['auth_url'])
try:
image = glance.images.get(image_id)
except:
raise Exception("Could not find Glance image with id" % (image_id))
# Unclear if this is strictly needed
# If size is not explicitly set then set it based on the image size
# TODO: Check if we even have to set a size when pulling from an image
if not volume_size:
# Gigabytes rounded up
volume_size = int(image.size/(1024*1024*1024)+1)
print "Starting asyncronous copying to Cinder"
volume = cinder.volumes.create(volume_size, display_name=image.name, imageRef=image.id)
while (volume.status != 'available'):
print "Waiting for volume to be ready ... current status (%s)" % (volume.status)
sleep(5)
volume = cinder.volumes.get(volume.id)
if (volume.status == 'error'):
raise Exception('Error converting image to volume')
return volume
def snapshot_from_volume(volume_id, creds):
cinder = cinder_client.Client('1', creds['username'], creds['password'], creds['tenant'], creds['auth_url'])
volume = volume=cinder.volumes.get(volume_id)
snapshot = cinder.volume_snapshots.create(volume.id,False,volume.display_name,volume.display_description)
while (snapshot.status != 'available'):
print "Waiting for snapshot to be ready ... current status (%s)" % (snapshot.status)
sleep(5)
snapshot = cinder.volume_snapshots.get(snapshot.id)
if snapshot.status == 'error':
raise Exception('Error while taking volume snapshot')
return snapshot
def volume_from_snapshot(snapshot_id, creds):
cinder = cinder_client.Client('1', creds['username'], creds['password'], creds['tenant'], creds['auth_url'])
snapshot = cinder.volume_snapshots.get(snapshot_id)
volume = cinder.volumes.create(size=None, snapshot_id=snapshot_id, display_name=snapshot.display_name,
display_description=snapshot.display_description)
while (volume.status != 'available'):
print "Waiting for volume to be ready ... current status (%s)" % (volume.status)
sleep(5)
volume = cinder.volumes.get(volume.id)
if volume.status == 'error':
raise Exception('Error while taking volume snapshot')
return volume
def ks_extract_bits(ksfile):
# I briefly looked at pykickstart but it more or less requires you know the version of the
# format you wish to use
# The approach below actually works as far back as RHEL5 and as recently as F18
f = open(ksfile)
lines = f.readlines()
f.close()
install_url = None
console_password = None
console_command = None
poweroff = False
distro = None
for line in lines:
for line in ksfile.splitlines():
# Install URL lines look like this
# url --url=http://download.devel.redhat.com/released/RHEL-5-Server/U9/x86_64/os/
m = re.match("url.*--url=(\S+)", line)
@ -213,16 +276,12 @@ def install_extract_bits(install_file, distro):
def preseed_extract_bits(preseedfile):
f = open(preseedfile)
lines = f.readlines()
f.close()
install_url = None
console_password = None
console_command = None
poweroff = False
for line in lines:
for line in preseedfile.splitlines():
# Network console lines look like this:
# d-i network-console/password password r00tme
@ -251,11 +310,7 @@ def preseed_extract_bits(preseedfile):
def detect_distro(install_script):
f = open(install_script)
lines = f.readlines()
f.close()
for line in lines:
for line in install_script.splitlines():
if re.match("d-i\s+debian-installer", line):
return "ubuntu"
elif re.match("%packages", line):
@ -338,7 +393,7 @@ def generate_blank_syslinux():
#os.remove(raw_image_name)
def generate_boot_content(url, dest_dir, distro="rpm"):
def generate_boot_content(url, dest_dir, distro, create_volume):
"""
Insert kernel, ramdisk and syslinux.cfg file in dest_dir
source from url
@ -348,11 +403,15 @@ def generate_boot_content(url, dest_dir, distro="rpm"):
if distro == "rpm":
kernel_url = url + "images/pxeboot/vmlinuz"
initrd_url = url + "images/pxeboot/initrd.img"
cmdline = "ks=http://169.254.169.254/latest/user-data"
if create_volume:
# NOTE: RHEL5 and other older Anaconda versions do not support specifying the CDROM device - use with caution
cmdline = "ks=http://169.254.169.254/latest/user-data repo=cdrom:/dev/vdb"
else:
cmdline = "ks=http://169.254.169.254/latest/user-data"
elif distro == "ubuntu":
kernel_url = url + "main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux"
initrd_url = url + "main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz"
cmdline = "append preseed/url=http://169.254.169.254/latest/user-data vga=788 debian-installer/locale=en_US console-setup/layoutcode=us netcfg/choose_interface=auto keyboard-configuration/layoutcode=us priority=critical --"
cmdline = "append preseed/url=http://169.254.169.254/latest/user-data debian-installer/locale=en_US console-setup/layoutcode=us netcfg/choose_interface=auto keyboard-configuration/layoutcode=us priority=critical --"
kernel_dest = os.path.join(dest_dir,"vmlinuz")
http_download_file(kernel_url, kernel_dest)
@ -434,18 +493,18 @@ def wait_for_noping(instance, nova, console_password, console_command):
if not started:
raise Exception("Instance at IP (%s) failed to start after 3 minutes." % (instance_ip) )
print "Instance responding to pings - waiting up to 20 minutes for it to stop"
print "Instance responding to pings - waiting up to 40 minutes for it to stop"
# TODO: Automate this using subprocess
if console_password:
print "Install script contains a remove console directive with a password"
print "You should be able to view progress with the following command:"
print "$",
print console_command % (instance_ip)
print "When prompted for a password enter: %s" % (console_password)
print "password: %s" % (console_password)
print
print "Note that it may take a few mintues for the server to become available"
# Now wait for up to 20 minutes for it to stop ping replies for at least 30 seconds
misses=0
for i in range(120):
for i in range(240):
print '.',
sys.stdout.flush()
if do_one(instance_ip, 10):
@ -460,18 +519,25 @@ def wait_for_noping(instance, nova, console_password, console_command):
print ''
if misses != 4:
print "Instance still pinging after 20 seconds - Assuming install failure"
print "Instance still pinging after 40 minutes - Assuming install failure"
return
print "Instance has stopped responding to ping for at least 30 seconds - assuming install is complete"
return instance
def launch_and_wait(image, image_volume, install_media_volume, working_ks, instance_name, creds, console_password, console_command):
if install_media_volume and image_volume:
block_device_mapping = {'vda': image_volume.id + ":::0", 'vdb': install_media_volume.id + ":::0"}
elif image_volume:
block_device_mapping = {'vda': image_volume.id + ":::0" }
else:
block_device_mapping = None
def launch_and_wait(image_id, working_ks, instance_name, creds, console_password, console_command):
nova = nova_client.Client(creds['username'], creds['password'], creds['tenant'],
auth_url=creds['auth_url'], insecure=True)
instance = nova.servers.create(instance_name, image_id, 2, userdata=working_ks, meta={})
instance = nova.servers.create(instance_name, image.id, 2, userdata=working_ks, meta={},
block_device_mapping = block_device_mapping)
print "Started instance id (%s)" % (instance.id)
#noping for Folsom - shutoff for newer
@ -483,6 +549,39 @@ def launch_and_wait(image_id, working_ks, instance_name, creds, console_password
return result
def terminate_instance(instance_id, creds):
nova = nova_client.Client(creds['username'], creds['password'], creds['tenant'],
auth_url=creds['auth_url'], insecure=True)
instance = nova.servers.get(instance_id)
instance.delete()
print "Waiting for instance id (%s) to be terminated/delete" % (instance_id)
while True:
print "Current instance status: %s" % (instance.status)
sleep(2)
try:
instance = nova.servers.get(instance_id)
except Exception as e:
print "Got exception (%s) assuming deletion complete" % (e)
break
def wait_for_glance_snapshot(image_id, creds, glance_url):
k = keystone_client.Client(username=creds['username'], password=creds['password'], tenant_name=creds['tenant'], auth_url=creds['auth_url'])
if not k.authenticate():
raise Exception("Unable to authenticate into Keystone")
glance = glance_client.Client("1", endpoint=glance_url, token=k.auth_token)
image = glance.images.get(image_id)
print "Waiting for glance image id (%s) to become active" % (image_id)
while True:
print "Current image status: %s" % (image.status)
sleep(2)
image = glance.images.get(image.id)
if image.status == "error":
raise Exception("Image entered error status while waiting for completion")
elif image.status == 'active':
break
def do_pw_sub(ks_file, admin_password):
f = open(ks_file, "r")
working_ks = ""
@ -490,111 +589,3 @@ def do_pw_sub(ks_file, admin_password):
working_ks += Template(line).safe_substitute({ 'adminpw': admin_password })
f.close()
return working_ks
parser = argparse.ArgumentParser(description='Launch and snapshot a kickstart install using syslinux and Glance')
parser.add_argument('--auth-url', dest='auth_url', required=True,
help='URL for keystone authorization')
parser.add_argument('--username', dest='username', required=True,
help='username for keystone authorization')
parser.add_argument('--tenant', dest='tenant', required=True,
help='tenant for keystone authorization')
parser.add_argument('--password', dest='password', required=True,
help='password for keystone authorization')
parser.add_argument('--glance-url', dest='glance_url', required=True,
help='URL for glance service')
parser.add_argument('--admin-password', dest='admin_password', required=True,
help='administrator password - also used for optional remote access during install')
parser.add_argument('--install-tree-url', dest='install_tree_url',
help='URL for preferred network install tree (optional)')
parser.add_argument('--distro', dest='distro',
help='distro - must be "rpm" or "ubuntu (optional)"')
parser.add_argument('--image-name', dest='image_name',
help='name to assign newly created image (optional)')
parser.add_argument('ks_file',
help='kickstart/install-script file to use for install')
args = parser.parse_args()
# This is a string
working_kickstart = do_pw_sub(args.ks_file, args.admin_password)
distro = detect_distro(args.ks_file)
if args.distro:
# Allow the command line distro to override our guess above
distro = args.distro
(install_tree_url, console_password, console_command, poweroff) = install_extract_bits(args.ks_file, distro)
if args.install_tree_url:
# Allow the specified tree to override anything extracted above
install_tree_url = args.install_tree_url
if args.image_name:
image_name = args.image_name
else:
image_name = "Image from ks file: %s - Date: %s" % (os.path.basename(args.ks_file), strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime()))
# Let's be nice and report as many error conditions as we can before exiting
error = False
if not install_tree_url:
print "ERROR: no install tree URL specified and could not extract one from the kickstart/install-script"
error = True
if not distro:
print "ERROR: no distro specified and could not guess based on the kickstart/install-script"
error = True
if not poweroff:
if distro == "rpm":
print "ERROR: supplied kickstart file must contain a 'poweroff' line"
elif distro == "ubuntu":
print "ERROR: supplied preseed must contain a 'd-i debian-installer/exit/poweroff boolean true' line"
error = True
if error:
sys.exit(1)
# Artifact of borrowing factory code - pass this as a dict
creds = { 'username': args.username, 'tenant': args.tenant, 'password': args.password, 'auth_url': args.auth_url }
# Generate "blank" syslinux bootable mini-image
# This is the only step that strictly requires root access due to the need
# for a loopback mount to install the bootloader
generate_blank_syslinux()
# Take a copy of it
modified_image = "./syslinux_modified_%s.qcow2" % os.getpid()
shutil.copy("./syslinux.qcow2",modified_image)
# Generate the content to put into the image
tmp_content_dir = mkdtemp()
print "Collecting boot content for auto-install image"
generate_boot_content(install_tree_url, tmp_content_dir, distro)
# Copy in the kernel, initrd and conf files into the blank boot stub using libguestfs
print "Copying boot content into a bootable syslinux image"
copy_content_to_image(tmp_content_dir, modified_image)
# Upload the resulting image to glance
print "Uploading image to glance"
image_id = glance_upload(modified_image, creds = creds, glance_url = args.glance_url,
name = "INSTALL for: %s" % (image_name), disk_format='qcow2')
print "Uploaded successfully as glance image (%s)" % (image_id)
# Launch the image with the provided ks.cfg as the user data
# Optionally - spawn a vncviewer to watch the install graphically
# Poll on image status until it is SHUTDOWN or timeout
print "Launching install image"
installed_instance = launch_and_wait(image_id, working_kickstart, os.path.basename(args.ks_file), creds, console_password, console_command)
# Take a snapshot of the now safely shutdown image
print "Taking snapshot of completed install"
finished_image_id = installed_instance.create_image(image_name)
print "Finished image snapshot ID is: %s" % (finished_image_id)
print "Finished image name is: %s" % (image_name)
#print "Cleaning up"
#print "Removing temp content dir"
#shutil.rmtree(tmp_content_dir)
#print "Removing install image"
##TODO:Note that thie is actually cacheable on a per-os-version basis
#os.remove(modified_image)

3
fedora-17-jeos.ks → install_scripts/fedora-17-jeos.ks
View File

@ -18,7 +18,7 @@ bootloader --location=mbr
zerombr
clearpart --all --drives=vda
part biosboot --fstype=biosboot --size=1
part biosboot --fstype=biosboot --size=1 --ondisk=vda
part /boot --fstype ext4 --size=200 --ondisk=vda
part pv.2 --size=1 --grow --ondisk=vda
volgroup VolGroup00 --pesize=32768 pv.2
@ -30,5 +30,6 @@ bootloader --location=mbr --timeout=5 --append="rhgb quiet"
%packages
@base
cloud-init
%end

37
install_scripts/fedora-18-jeos-DVD.ks Normal file
View File

@ -0,0 +1,37 @@
cdrom
#url --url=http://mirrors.kernel.org/fedora/releases/18/Fedora/x86_64/os/
# Without the Everything repo, we cannot install cloud-init
#repo --name="fedora-everything" --baseurl=http://mirrors.kernel.org/fedora/releases/18/Everything/x86_64/os/
repo --name="fedora-everything" --mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-18&arch=x86_64
install
graphical
vnc --password=${adminpw}
text
keyboard us
lang en_US.UTF-8
skipx
network --device eth0 --bootproto dhcp
rootpw ${adminpw}
firewall --disabled
authconfig --enableshadow --enablemd5
selinux --enforcing
timezone --utc America/New_York
bootloader --location=mbr
zerombr
clearpart --all --drives=vda
part biosboot --fstype=biosboot --size=1 --ondisk=vda
part /boot --fstype ext4 --size=200 --ondisk=vda
part pv.2 --size=1 --grow --ondisk=vda
volgroup VolGroup00 --pesize=32768 pv.2
logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=768 --grow --maxsize=1536
logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow
poweroff
bootloader --location=mbr --timeout=5 --append="rhgb quiet"
%packages
@core
cloud-init
%end

3
fedora-18-jeos.ks → install_scripts/fedora-18-jeos.ks
View File

@ -19,7 +19,7 @@ bootloader --location=mbr
zerombr
clearpart --all --drives=vda
part biosboot --fstype=biosboot --size=1
part biosboot --fstype=biosboot --size=1 --ondisk=vda
part /boot --fstype ext4 --size=200 --ondisk=vda
part pv.2 --size=1 --grow --ondisk=vda
volgroup VolGroup00 --pesize=32768 pv.2
@ -31,5 +31,6 @@ bootloader --location=mbr --timeout=5 --append="rhgb quiet"
%packages
@core
cloud-init
%end

35
install_scripts/rhel-5-jeos.ks Normal file
View File

@ -0,0 +1,35 @@
install
url --url=<replace with RHEL5 install tree URL>
#text
graphical
vnc --password=${adminpw}
key --skip
keyboard us
lang en_US.UTF-8
skipx
network --device eth0 --bootproto dhcp
rootpw ${adminpw}
firewall --disabled
authconfig --enableshadow --enablemd5
selinux --enforcing
timezone --utc America/New_York
bootloader --location=mbr --append="console=tty0 console=ttyS0,115200"
zerombr yes
clearpart --all
part /boot --fstype ext3 --size=200
part pv.2 --size=1 --grow
volgroup VolGroup00 --pesize=32768 pv.2
logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=768 --grow --maxsize=1536
logvol / --fstype ext3 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow
#reboot
poweroff
# Needed for cloud-init
repo --name="EPEL-5" --baseurl="http://mirrors.kernel.org/fedora-epel/5/x86_64/"
%packages
@base
cloud-init
%post

32
install_scripts/rhel-6-jeos.ks Normal file
View File

@ -0,0 +1,32 @@
install
url --url=<replace with rhel6 install tree URL>
# Needed for cloud-init
repo --name="EPEL-6" --baseurl="http://mirrors.kernel.org/fedora-epel/6/x86_64/"
graphical
vnc --password=${adminpw}
key --skip
keyboard us
lang en_US.UTF-8
skipx
network --device eth0 --bootproto dhcp
rootpw ${adminpw}
firewall --disabled
authconfig --enableshadow --enablemd5
selinux --enforcing
timezone --utc America/New_York
bootloader --location=mbr --append="console=tty0 console=ttyS0,115200"
zerombr yes
clearpart --all
part /boot --fstype ext4 --size=200
part pv.2 --size=1 --grow
volgroup VolGroup00 --pesize=32768 pv.2
#logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=768 --grow --maxsize=1536
logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow
poweroff
%packages
@base
cloud-init
%post

1
ubuntu-10.04-jeos.preseed → install_scripts/ubuntu-10.04-jeos.preseed
View File

@ -21,6 +21,7 @@ d-i netcfg/wireless_wep string
d-i clock-setup/utc boolean true
d-i time/zone string US/Eastern
d-i partman-auto/disk string /dev/vda
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe select home
d-i partman/confirm_write_new_label boolean true

1
ubuntu-12.04-jeos.preseed → install_scripts/ubuntu-12.04-jeos.preseed
View File

@ -20,6 +20,7 @@ d-i netcfg/wireless_wep string
d-i clock-setup/utc boolean true
d-i time/zone string US/Eastern
d-i partman-auto/disk string /dev/vda
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe select home
d-i partman/confirm_write_new_label boolean true

1
ubuntu-12.10-jeos.preseed → install_scripts/ubuntu-12.10-jeos.preseed
View File

@ -20,6 +20,7 @@ d-i netcfg/wireless_wep string
d-i clock-setup/utc boolean true
d-i time/zone string US/Eastern
d-i partman-auto/disk string /dev/vda
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe select home
d-i partman/confirm_write_new_label boolean true