Add initial docs about datasources.

Start moving the current README for
datasources to a RST format and include
those files in the rtd site.
This commit is contained in:
harlowja 2013-02-04 23:10:36 -08:00
parent a1efdc8a34
commit 147776bd98
11 changed files with 385 additions and 242 deletions

View File

@ -22,6 +22,7 @@ Summary
topics/format
topics/dir_layout
topics/examples
topics/datasources
topics/modules
topics/moreinfo
topics/hacking

View File

@ -0,0 +1,99 @@
.. _datasources:
=========
Datasources
=========
----------
What is a datasource?
----------
Datasources are sources of configuration data for cloud-init that typically come
from the user (aka userdata) or come from the stack that created the configuration
drive (aka metadata). Typical userdata would include files, yaml, and shell scripts
while typical metadata would include server name, instance id, display name and other
cloud specific details. Since there are multiple ways to provide this data (each cloud
solution seems to prefer its own way) internally a datasource abstract class was
created to allow for a single way to access the different cloud systems methods
to provide this data through the typical usage of subclasses.
The current interface that a datasource object must provide is the following:
.. sourcecode:: python
def get_userdata(self, apply_filter=False)
@property
def launch_index(self)
@property
def is_disconnected(self)
def get_userdata_raw(self)
# the data sources' config_obj is a cloud-config formated
# object that came to it from ways other than cloud-config
# because cloud-config content would be handled elsewhere
def get_config_obj(self)
def get_public_ssh_keys(self)
def device_name_to_device(self, name)
def get_locale(self)
@property
def availability_zone(self)
def get_instance_id(self)
def get_hostname(self, fqdn=False)
def get_package_mirror_info(self)
---------------------------
EC2
---------------------------
TBD
---------------------------
Config Drive
---------------------------
.. include:: ../../sources/configdrive/README.rst
---------------------------
Alt cloud
---------------------------
.. include:: ../../sources/altcloud/README.rst
---------------------------
No cloud
---------------------------
.. include:: ../../sources/nocloud/README.rst
---------------------------
MAAS
---------------------------
TBD
---------------------------
CloudStack
---------------------------
TBD
---------------------------
OVF
---------------------------
See: https://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/sources/ovf/
---------------------------
Fallback/None
---------------------------
TBD

View File

@ -25,9 +25,9 @@ Cloudinits's directory structure is somewhat different from a regular applicatio
- user-data.txt
- user-data.txt.i
- scripts/
- per-boot/
- per-instance/
- per-once/
- per-boot/
- per-instance/
- per-once/
- seed/
- sem/

View File

@ -1 +1 @@
.. include:: ../../../HACKING
.. include:: ../../../HACKING.rst

View File

@ -1,65 +0,0 @@
Data souce AltCloud will be used to pick up user data on
RHEVm and vSphere.
RHEVm:
======
For REHVm v3.0 the userdata is injected into the VM using floppy
injection via the RHEVm dashboard "Custom Properties". The format
of the Custom Properties entry must be:
"floppyinject=user-data.txt:<base64 encoded data>"
e.g.: To pass a simple bash script
% cat simple_script.bash
#!/bin/bash
echo "Hello Joe!" >> /tmp/JJV_Joe_out.txt
% base64 < simple_script.bash
IyEvYmluL2Jhc2gKZWNobyAiSGVsbG8gSm9lISIgPj4gL3RtcC9KSlZfSm9lX291dC50eHQK
To pass this example script to cloud-init running in a RHEVm v3.0 VM
set the "Custom Properties" when creating the RHEMv v3.0 VM to:
floppyinject=user-data.txt:IyEvYmluL2Jhc2gKZWNobyAiSGVsbG8gSm9lISIgPj4gL3RtcC9KSlZfSm9lX291dC50eHQK
NOTE: The prefix with file name must be: "floppyinject=user-data.txt:"
It is also possible to launch a RHEVm v3.0 VM and pass optional user
data to it using the Delta Cloud.
For more inforation on Delta Cloud see: http://deltacloud.apache.org
vSphere:
========
For VMWare's vSphere the userdata is injected into the VM an ISO
via the cdrom. This can be done using the vSphere dashboard
by connecting an ISO image to the CD/DVD drive.
To pass this example script to cloud-init running in a vSphere VM
set the CD/DVD drive when creating the vSphere VM to point to an
ISO on the data store.
The ISO must contain the user data:
For example, to pass the same simple_script.bash to vSphere:
Create the ISO:
===============
% mkdir my-iso
NOTE: The file name on the ISO must be: "user-data.txt"
% cp simple_scirpt.bash my-iso/user-data.txt
% genisoimage -o user-data.iso -r my-iso
Verify the ISO:
===============
% sudo mkdir /media/vsphere_iso
% sudo mount -o loop JoeV_CI_02.iso /media/vsphere_iso
% cat /media/vsphere_iso/user-data.txt
% sudo umount /media/vsphere_iso
Then, launch the vSphere VM the ISO user-data.iso attached as a CDrom.
It is also possible to launch a vSphere VM and pass optional user
data to it using the Delta Cloud.
For more inforation on Delta Cloud see: http://deltacloud.apache.org

View File

@ -0,0 +1,87 @@
The datasource altcloud will be used to pick up user data on `RHEVm`_ and `vSphere`_.
RHEVm
~~~~~~
For `RHEVm`_ v3.0 the userdata is injected into the VM using floppy
injection via the `RHEVm`_ dashboard "Custom Properties".
The format of the Custom Properties entry must be:
::
floppyinject=user-data.txt:<base64 encoded data>
For example to pass a simple bash script:
::
% cat simple_script.bash
#!/bin/bash
echo "Hello Joe!" >> /tmp/JJV_Joe_out.txt
% base64 < simple_script.bash
IyEvYmluL2Jhc2gKZWNobyAiSGVsbG8gSm9lISIgPj4gL3RtcC9KSlZfSm9lX291dC50eHQK
To pass this example script to cloud-init running in a `RHEVm`_ v3.0 VM
set the "Custom Properties" when creating the RHEMv v3.0 VM to:
::
floppyinject=user-data.txt:IyEvYmluL2Jhc2gKZWNobyAiSGVsbG8gSm9lISIgPj4gL3RtcC9KSlZfSm9lX291dC50eHQK
**NOTE:** The prefix with file name must be: ``floppyinject=user-data.txt:``
It is also possible to launch a `RHEVm`_ v3.0 VM and pass optional user
data to it using the Delta Cloud.
For more information on Delta Cloud see: http://deltacloud.apache.org
vSphere
~~~~~~~~
For VMWare's `vSphere`_ the userdata is injected into the VM as an ISO
via the cdrom. This can be done using the `vSphere`_ dashboard
by connecting an ISO image to the CD/DVD drive.
To pass this example script to cloud-init running in a `vSphere`_ VM
set the CD/DVD drive when creating the vSphere VM to point to an
ISO on the data store.
**Note:** The ISO must contain the user data.
For example, to pass the same ``simple_script.bash`` to vSphere:
Create the ISO
-----------------
::
% mkdir my-iso
NOTE: The file name on the ISO must be: ``user-data.txt``
::
% cp simple_scirpt.bash my-iso/user-data.txt
% genisoimage -o user-data.iso -r my-iso
Verify the ISO
-----------------
::
% sudo mkdir /media/vsphere_iso
% sudo mount -o loop JoeV_CI_02.iso /media/vsphere_iso
% cat /media/vsphere_iso/user-data.txt
% sudo umount /media/vsphere_iso
Then, launch the `vSphere`_ VM the ISO user-data.iso attached as a CDROM.
It is also possible to launch a `vSphere`_ VM and pass optional user
data to it using the Delta Cloud.
For more information on Delta Cloud see: http://deltacloud.apache.org
.. _RHEVm: https://www.redhat.com/virtualization/rhev/desktop/rhevm/
.. _vSphere: https://www.vmware.com/products/datacenter-virtualization/vsphere/overview.html

View File

@ -1,118 +0,0 @@
The 'ConfigDrive' DataSource supports the OpenStack configdrive disk.
See doc/source/api_ext/ext_config_drive.rst in the nova source code for
more information on config drive.
The following criteria are required to be identified by
DataSourceConfigDrive as a config drive:
* must be formated with vfat filesystem
* must be a un-partitioned block device (/dev/vdb, not /dev/vdb1)
* must contain one of the following files:
* etc/network/interfaces
* root/.ssh/authorized_keys
* meta.js
By default, cloud-init does not consider this source to be a full-fledged
datasource. Instead, the default behavior is to assume it is really only
present to provide networking information. Cloud-init will copy off the
network information, apply it to the system, and then continue on. The
"full" datasource would then be found in the EC2 metadata service.
== Content of config-drive ==
* etc/network/interfaces
This file is laid down by nova in order to pass static networking
information to the guest. Cloud-init will copy it off of the config-drive
and into /etc/network/interfaces as soon as it can, and then attempt to
bring up all network interfaces.
* root/.ssh/authorized_keys
This file is laid down by nova, and contains the keys that were
provided to it on instance creation (nova-boot --key ....)
Cloud-init will copy those keys and put them into the configured user
('ubuntu') .ssh/authorized_keys.
* meta.js
meta.js is populated on the config-drive in response to the user passing
"meta flags" (nova boot --meta key=value ...). It is expected to be json
formated.
== Configuration ==
Cloud-init's behavior can be modified by keys found in the meta.js file in
the following ways:
* dsmode:
values: local, net, pass
default: pass
This is what indicates if configdrive is a final data source or not.
By default it is 'pass', meaning this datasource should not be read.
Set it to 'local' or 'net' to stop cloud-init from continuing on to
search for other data sources after network config.
The difference between 'local' and 'net' is that local will not require
networking to be up before user-data actions (or boothooks) are run.
* instance-id:
default: iid-dsconfigdrive
This is utilized as the metadata's instance-id. It should generally
be unique, as it is what is used to determine "is this a new instance".
* public-keys:
default: None
if present, these keys will be used as the public keys for the
instance. This value overrides the content in authorized_keys.
Note: it is likely preferable to provide keys via user-data
* user-data:
default: None
This provides cloud-init user-data. See other documentation for what
all can be present here.
== Example ==
Here is an example using the nova client (python-novaclien)
Assuming the following variables set up:
* img_id : set to the nova image id (uuid from image-list)
* flav_id : set to numeric flavor_id (nova flavor-list)
* keyname : set to name of key for this instance (nova keypair-list)
$ cat my-user-data
#!/bin/sh
echo ==== USER_DATA FROM EC2 MD ==== | tee /ud.log
$ ud_value=$(sed 's,EC2 MD,META KEY,')
## Now, 'ud_value' has same content of my-user-data file, but
## with the string "USER_DATA FROM META KEY"
## launch an instance with dsmode=pass
## This will really not use the configdrive for anything as the mode
## for the datasource is 'pass', meaning it will still expect some
## other data source (DataSourceEc2).
$ nova boot --image=$img_id --config-drive=1 --flavor=$flav_id \
--key_name=$keyname \
--user_data=my-user-data \
"--meta=instance-id=iid-001 \
"--meta=user-data=${ud_keyval}" \
"--meta=dsmode=pass" cfgdrive-dsmode-pass
$ euca-get-console-output i-0000001 | grep USER_DATA
echo ==== USER_DATA FROM EC2 MD ==== | tee /ud.log
## Now, launch an instance with dsmode=local
## This time, the only metadata and userdata available to cloud-init
## are on the config-drive
$ nova boot --image=$img_id --config-drive=1 --flavor=$flav_id \
--key_name=$keyname \
--user_data=my-user-data \
"--meta=instance-id=iid-001 \
"--meta=user-data=${ud_keyval}" \
"--meta=dsmode=local" cfgdrive-dsmode-local
$ euca-get-console-output i-0000002 | grep USER_DATA
echo ==== USER_DATA FROM META KEY ==== | tee /ud.log
--
[1] https://github.com/openstack/nova/blob/master/doc/source/api_ext/ext_config_drive.rst for more if

View File

@ -0,0 +1,123 @@
The configuration drive datasource supports the `OpenStack`_ configuration drive disk.
See `the config drive extension`_ and `introduction`_ in the public
documentation for more information.
By default, cloud-init does *always* consider this source to be a full-fledged
datasource. Instead, the typical behavior is to assume it is really only
present to provide networking information. Cloud-init will copy off the
network information, apply it to the system, and then continue on. The
"full" datasource could then be found in the EC2 metadata service. If this is
not the case then the files contained on the located drive must provide equivalents
to what the EC2 metadata service would provide (which is typical of the version
2 support listed below)
Version 1
~~~~~~~~~
The following criteria are required to as a config drive:
1. Must be formatted with `vfat`_ filesystem
2. Must be a un-partitioned block device (/dev/vdb, not /dev/vdb1)
3. Must contain *one* of the following files
::
/etc/network/interfaces
/root/.ssh/authorized_keys
/meta.js
``/etc/network/interfaces``
This file is laid down by nova in order to pass static networking
information to the guest. Cloud-init will copy it off of the config-drive
and into /etc/network/interfaces (or convert it to RH format) as soon as it can,
and then attempt to bring up all network interfaces.
``/root/.ssh/authorized_keys``
This file is laid down by nova, and contains the ssk keys that were
provided to nova on instance creation (nova-boot --key ....)
``/meta.js``
meta.js is populated on the config-drive in response to the user passing
"meta flags" (nova boot --meta key=value ...). It is expected to be json
formatted.
Version 2
~~~~~~~~~~~
The following criteria are required to as a config drive:
1. Must be formatted with `vfat`_ or `iso9660`_ filesystem
or have a *filesystem* label of **config-2**
2. Must be a un-partitioned block device (/dev/vdb, not /dev/vdb1)
3. The files that will typically be present in the config drive are:
::
openstack/
- 2012-08-10/ or latest/
- meta_data.json
- user_data (not mandatory)
- content/
- 0000 (referenced content files)
- 0001
- ....
ec2
- latest/
- meta-data.json (not mandatory)
Keys and values
~~~~~~~~~~~
Cloud-init's behavior can be modified by keys found in the meta.js (version 1 only) file in the following ways.
::
dsmode:
values: local, net, pass
default: pass
This is what indicates if configdrive is a final data source or not.
By default it is 'pass', meaning this datasource should not be read.
Set it to 'local' or 'net' to stop cloud-init from continuing on to
search for other data sources after network config.
The difference between 'local' and 'net' is that local will not require
networking to be up before user-data actions (or boothooks) are run.
::
instance-id:
default: iid-dsconfigdrive
This is utilized as the metadata's instance-id. It should generally
be unique, as it is what is used to determine "is this a new instance".
::
public-keys:
default: None
If present, these keys will be used as the public keys for the
instance. This value overrides the content in authorized_keys.
Note: it is likely preferable to provide keys via user-data
::
user-data:
default: None
This provides cloud-init user-data. See :ref:`examples <yaml_examples>` for
what all can be present here.
.. _OpenStack: http://www.openstack.org/
.. _introduction: http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html
.. _python-novaclient: https://github.com/openstack/python-novaclient
.. _iso9660: https://en.wikipedia.org/wiki/ISO_9660
.. _vfat: https://en.wikipedia.org/wiki/File_Allocation_Table
.. _the config drive extension: http://docs.openstack.org/developer/nova/api_ext/ext_config_drive.html

View File

@ -1,55 +0,0 @@
The data source 'NoCloud' and 'NoCloudNet' allow the user to provide user-data
and meta-data to the instance without running a network service (or even without
having a network at all)
You can provide meta-data and user-data to a local vm boot via files on a vfat
or iso9660 filesystem. These user-data and meta-data files are expected to be
in the format described in doc/example/seed/README . Basically, user-data is
simply user-data and meta-data is a yaml formated file representing what you'd
find in the EC2 metadata service.
Given a disk 12.04 cloud image in 'disk.img', you can create a sufficient disk
by following the example below.
## create user-data and meta-data files that will be used
## to modify image on first boot
$ { echo instance-id: iid-local01; echo local-hostname: cloudimg; } > meta-data
$ printf "#cloud-config\npassword: passw0rd\nchpasswd: { expire: False }\nssh_pwauth: True\n" > user-data
## create a disk to attach with some user-data and meta-data
$ genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data
## alternatively, create a vfat filesystem with same files
## $ truncate --size 2M seed.img
## $ mkfs.vfat -n cidata seed.img
## $ mcopy -oi seed.img user-data meta-data ::
## create a new qcow image to boot, backed by your original image
$ qemu-img create -f qcow2 -b disk.img boot-disk.img
## boot the image and login as 'ubuntu' with password 'passw0rd'
## note, passw0rd was set as password through the user-data above,
## there is no password set on these images.
$ kvm -m 256 \
-net nic -net user,hostfwd=tcp::2222-:22 \
-drive file=boot-disk.img,if=virtio \
-drive file=seed.iso,if=virtio
Note, that the instance-id provided ('iid-local01' above) is what is used to
determine if this is "first boot". So if you are making updates to user-data
you will also have to change that, or start the disk fresh.
Also, you can inject an /etc/network/interfaces file by providing the content
for that file in the 'network-interfaces' field of metadata. Example metadata:
instance-id: iid-abcdefg
network-interfaces: |
iface eth0 inet static
address 192.168.1.10
network 192.168.1.0
netmask 255.255.255.0
broadcast 192.168.1.255
gateway 192.168.1.254
hostname: myhost

View File

@ -0,0 +1,71 @@
The data source ``NoCloud`` and ``NoCloudNet`` allow the user to provide user-data
and meta-data to the instance without running a network service (or even without
having a network at all).
You can provide meta-data and user-data to a local vm boot via files on a `vfat`_
or `iso9660`_ filesystem.
These user-data and meta-data files are expected to be
in the following format.
::
/user-data
/meta-data
Basically, user-data is simply user-data and meta-data is a yaml formatted file
representing what you'd find in the EC2 metadata service.
Given a disk ubuntu 12.04 cloud image in 'disk.img', you can create a sufficient disk
by following the example below.
::
## create user-data and meta-data files that will be used
## to modify image on first boot
$ { echo instance-id: iid-local01; echo local-hostname: cloudimg; } > meta-data
$ printf "#cloud-config\npassword: passw0rd\nchpasswd: { expire: False }\nssh_pwauth: True\n" > user-data
## create a disk to attach with some user-data and meta-data
$ genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data
## alternatively, create a vfat filesystem with same files
## $ truncate --size 2M seed.img
## $ mkfs.vfat -n cidata seed.img
## $ mcopy -oi seed.img user-data meta-data ::
## create a new qcow image to boot, backed by your original image
$ qemu-img create -f qcow2 -b disk.img boot-disk.img
## boot the image and login as 'ubuntu' with password 'passw0rd'
## note, passw0rd was set as password through the user-data above,
## there is no password set on these images.
$ kvm -m 256 \
-net nic -net user,hostfwd=tcp::2222-:22 \
-drive file=boot-disk.img,if=virtio \
-drive file=seed.iso,if=virtio
**Note:** that the instance-id provided (``iid-local01`` above) is what is used to
determine if this is "first boot". So if you are making updates to user-data
you will also have to change that, or start the disk fresh.
Also, you can inject an ``/etc/network/interfaces`` file by providing the content
for that file in the ``network-interfaces`` field of metadata.
Example metadata:
::
instance-id: iid-abcdefg
network-interfaces: |
iface eth0 inet static
address 192.168.1.10
network 192.168.1.0
netmask 255.255.255.0
broadcast 192.168.1.255
gateway 192.168.1.254
hostname: myhost
.. _iso9660: https://en.wikipedia.org/wiki/ISO_9660
.. _vfat: https://en.wikipedia.org/wiki/File_Allocation_Table