Remove old school TOCI files

Removes a bunch of the now deprecated toci_* files which now
exists in a different form under tripleo-incubator/scripts/devtest_*.

These scripts served us well over the past few months but the
time has come to focus this project solely on CI.

Well done old toci_*. Thanks for helping us raise the bar.

Change-Id: I4b9ec09e2c841c8b9374d21ea9f442cbe2ef53d3
This commit is contained in:
Dan Prince 2014-01-22 22:43:10 -05:00
parent 129d90f514
commit b625c9295d
14 changed files with 9 additions and 893 deletions

94
FAQ.md
View File

@ -1,94 +0,0 @@
Before Running any commands below ensure you have sourced the toci_env file
$ cd /opt/toci
$ . toci_env
**Q. How can I ssh to the seed vm**
```
$ ssh root@$($TOCI_WORKING_DIR/incubator/scripts/get-vm-ip seed)
```
**Q. How can I ssh to the undercloud controller**
```
$ ssh heat-admin@192.0.2.2
```
**Q. How do I list the nodes making up the overcloud**
```
$ . undercloudrc
$ nova list
```
**Q. How can I create more bm_poseur nodes and add them to my undercloud**
```
$ . undercloudrc
# <cpus> <memory> <disk> <architecture> <quantity>
$ create-nodes 1 1024 10 i386 5
$ export MACS=$($TOCI_WORKING_DIR/bm_poseur/bm_poseur get-macs)
$ setup-baremetal 1 768 10 i386 all
```
**Q. How can I create a new image, using the elements fedora,
selinux-permissive and stackuser**
```
$ ./diskimage-builder/bin/disk-image-create -a i386 -o myimage fedora selinux-permissive stackuser
```
**Q. I have a heat template, how can I start a stack on the undercloud using
it**
```
$ . undercloudrc
$ heat stack-create -f /path/to/myheattemplate.yaml mystack
```
**Q. I want to rerun toci without having to build a new set of images each
time.**
Yes, toci can apply a patch to disk-imagebuilder that will cause it place
build images in a /opt/toci/image_cache, all other runs of toci will use
these files instead of building a new one. To use the patch move it into
the patches directory where you clone toci to.
```
$ cp patches_dev/diskimage-builder-0001-Save-images-in-a-toci-cache-file-or-use-if-present.patch patches
```
Do not forget to remove images from /opt/toci/image_cache if you change
anything that would require a new image build
**Q. There is a new version of diskimage-builder I want to update to and use it
to rebuild images**
```
$ rm -rf /opt/toci/diskimage-builder
$ ./toci.sh
```
**Q. I would like to redeploy tripleo with a patch that has been submitted to
nova**
```
$ cd /opt/toci/nova
# You can get the git reference from the patchset in gerrit
$ git fetch https://review.openstack.org/openstack/nova refs/changes/36/45536/1
$ git reset --hard FETCH_HEAD
$ cd /path/to/toci
# Remove /opt/toci/image_cache/* if you are using the image caching described above
# Run toci
$ ./toci.sh
```
**Q. My toci run failed which logfile should I be looking in?**
Toci gathers various log file into a single directory (ouput at the start
of the run), the content of this directory typically looks like this.
```
./git.out - output from various git commands as they clone the repositories
./error-applying-patches.log - if this exists a patch from the patches directory failed to apply
./setup.out - output from toci_setup.sh
./test.out - output from toci_test.sh
./cleanup.out - output from tcoi_cleanup.sh
./192.168.122.218.tgz - tarball of /var/logs and /etc on seed VM
./192.0.2.2.tgz - tarball of /var/logs and /etc on undercloud
./192.0.2.5.tgz - tarball of /var/logs and /etc on overcloud notcompute
```
Some of these files may not exist if toci failed to complete

110
README.md
View File

@ -1,109 +1,17 @@
toci
====
Description
-----------
TripleO CI test framework.
By default toci builds images for seed, undercloud and overcloud hosts, it
then uses bare metal poseur nodes to set up a virtualized TripleO environment.
Options also exist so you can specify hosts to setup TripleO on a real
baremetal environment.
Configuration
-------------
If using toci to setup TripleO on a virtualized environment we recommend you
setup a proxy server for http traffic
edit ~/.toci and add a value for
```bash
export http_proxy=http://1.2.3.4:3128
```
See toci-defaults for a list of additional environment variables that can be
defined in ~/.toci in order to control things like
* Changing the architecture to amd64
* Deploying TripleO images on real baremetal hosts
* Increasing the resources allocated to bm_poseur nodes
* notifying an irc channel and uploading toci results to a server (used if
running toci as a CI framwork)
Using Toci to setup a dev environment
-------------------------------------
I usually do this as root, in theory it will also work as a non privileged
user.
$ git clone https://github.com/openstack-infra/tripleo-ci.git toci
$ cd toci
$ vi ~/.toci # Will work without a proxy but can be a lot slower
export http_proxy=http://1.2.3.4:8080
To deploy toci run the command command
$ ./toci.sh
Toci will start with a line outputting the working and log directories e.g.
Starting run Wed 3 Jul 11:46:39 IST 2013 ( /opt/toci /tmp/toci_logs_nGnrhLN )
Once it ran successfully (ERROR wasn't echo'd to the terminal) you should have
* seed vm
* undercloud vm
* overcloud controller vm
* overcloud compute vm
NOTE: toci will now have cloned the dependency git repositories to /opt/toci,
If you rerun toci it will NOT re-clone these again, if you would like it to
reclone the most recent version of any of these repositories you can simply
delete it before running toci.
If you would like to test a specific change locally in TripleO you can simply
edit the repository locally and commit this change to its master branch and
rerun toci. See the FAQ if you would like to do this without rebuilding all of
the images (e.g. For speed reasons, you would only like to rebuild the
overcloud images and reuse the previously built seed and undercloud image)
See FAQ.md for more information on how to use the TripleO deployment
Using Toci as a CI framework for TripleO
----------------------------------------
If running toci as part of a automated CI job several environment variables
can be defined to help make toci more suitable e.g.
*So toci cleans up after itself*
```bash
export TOCI_UPLOAD=1
export TOCI_REMOVE=1
```
*scp logs to a server when finished*
```bash
TOCI_RESULTS_DST=user@1.2.3.4:/var/www/html/toci
```
*Notify a freenode irc channel upon error*
```bash
TOCI_IRC=channeltonotify
```
*Only build and deploy the seed and undercloud*
```bash
export TOCI_DO_OVERCLOUD=0
```
Use toci to deploy on real baremetal
-----------------------------------
```bash
export TOCI_PM_DRIVER="nova.virt.baremetal.ipmi.IPMI"
#Space delimited, aligned in order
export TOCI_MACS="84:2b:22:11:11:11 84:2b:22:11:11:12 84:2b:22:11:11:13"
export TOCI_PM_IPS="10.16.111.111 10.16.111.112 10.16.111.113"
export TOCI_PM_USERS="root root root"
export TOCI_PM_PASSWORDS="passwd passwd passwd"
```
Tools to help run CI jobs for TripleO. Includes things like:
* Shell scripts to help execute jobs on CI slave nodes (Jenkins slaves)
* A test environment broker framework which uses a client-server
model to execute jobs on a remote bare metal machine in an isolated
test environment (using VMs).
* Image elements to help build images for the test environment
broker nodes.
* Heat templates to help deploy and maintain test environment nodes
using an undercloud.

View File

@ -1,2 +0,0 @@
Put patches in here that need to be applied to the tripleo repositories
for more info see toci_git.sh

View File

@ -1,24 +0,0 @@
From 76fb773073a02d9d6c39d0b0150947573c6de4e9 Mon Sep 17 00:00:00 2001
From: Derek Higgins <derekh@redhat.com>
Date: Fri, 19 Jul 2013 10:21:27 +0100
Subject: [PATCH] Allow heat-admin sudo without tty
---
elements/rpm-distro/pre-install.d/00-allow-root-sudo | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/elements/rpm-distro/pre-install.d/00-allow-root-sudo b/elements/rpm-distro/pre-install.d/00-allow-root-sudo
index 4736ac8..ff59cfb 100755
--- a/elements/rpm-distro/pre-install.d/00-allow-root-sudo
+++ b/elements/rpm-distro/pre-install.d/00-allow-root-sudo
@@ -5,3 +5,8 @@ set -e
echo "Defaults:root !requiretty" >> /etc/sudoers.d/root-notty
chmod 0440 /etc/sudoers.d/root-notty
visudo -c
+
+# For toci we need heat-admin to sudo without tty
+echo "Defaults:heat-admin !requiretty" >> /etc/sudoers.d/heat-admin-notty
+chmod 0440 /etc/sudoers.d/heat-admin-notty
+visudo -c
--
1.8.1.4

View File

@ -1,44 +0,0 @@
From 448fbaa131521e6060f9d87c83098c8cb90696ea Mon Sep 17 00:00:00 2001
From: Derek Higgins <derekh@redhat.com>
Date: Fri, 19 Jul 2013 13:55:41 +0100
Subject: [PATCH] Save images in a toci cache file or use if present
FOR DEV ONLY
If you want to
o skip building images in place of a cached version,
o mv first-boot scripts instead of rm
place this file in patches.
---
bin/disk-image-create | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/bin/disk-image-create b/bin/disk-image-create
index de14433..fba1e1e 100755
--- a/bin/disk-image-create
+++ b/bin/disk-image-create
@@ -105,6 +105,13 @@ arg_to_elements "$@"
IMAGE_NAME=${IMAGE_NAME%%\.${IMAGE_TYPE}}
+CACHEFILE=$TOCI_WORKING_DIR/image_cache/$(basename $IMAGE_NAME.$IMAGE_TYPE)
+if [ -e $CACHEFILE ] ; then
+ echo Using $CACHEFILE
+ cp $CACHEFILE $IMAGE_NAME.$IMAGE_TYPE
+ exit 0
+fi
+
mk_build_dir
create_base
run_d extra-data
@@ -151,6 +158,7 @@ unmount_image
if [ "$IS_RAMDISK" == "0" ]; then
compress_and_save_image $IMAGE_NAME.$IMAGE_TYPE
+ cp $IMAGE_NAME.$IMAGE_TYPE $TOCI_WORKING_DIR/image_cache/$(basename $IMAGE_NAME.$IMAGE_TYPE)
else
remove_image
fi
--
1.8.1.4

View File

@ -1,71 +0,0 @@
#!/usr/bin/bash
# Time the script was started
export STARTTIME=$(date)
# All toci working files should go here
export TOCI_WORKING_DIR=${TOCI_WORKING_DIR:-/opt/toci}
export TRIPLEO_ROOT=$TOCI_WORKING_DIR # some scripts are expecting this
# Should toci get tripleo repositories
export TOCI_GIT_CHECKOUT=${TOCI_GIT_CHECKOUT:-1}
# Files that should be cached between runs should go in here
# e.g. downloaded images, git repo's etc...
export TOCI_CACHE_DIR=${TOCI_CACHE_DIR:-/var/tmp/toci_cache}
# Any files to be uploaded to results server go here
export TOCI_LOG_DIR=${TOCI_LOG_DIR:-$(mktemp -d --tmpdir toci_logs_XXXXXXX)}
# Set TOCI_ARCH to x86_64 to build 64 bit arch
export TOCI_ARCH=${TOCI_ARCH:-'i386'}
export TOCI_DIB_ARCH='i386'
if [ "$TOCI_ARCH" == 'x86_64' ]; then
export TOCI_DIB_ARCH='amd64'
fi
# This gets passed into diskimage builder
export NODE_DIST=${NODE_DIST:-'pip-cache fedora selinux-permissive'}
# custom diskimage builder elements for the seed
export TOCI_SEED_EXTRA_ELEMENTS=${TOCI_SEED_EXTRA_ELEMENTS:-''}
# custom diskimage builder elements for the undercloud
export TOCI_UNDERCLOUD_EXTRA_ELEMENTS=${TOCI_UNDERCLOUD_EXTRA_ELEMENTS:-''}
# custom diskimage builder elements for the overcloud
export TOCI_OVERCLOUD_EXTRA_ELEMENTS=${TOCI_OVERCLOUD_EXTRA_ELEMENTS:-''}
# Set to 0 if you don't want an overcloud to be started
export TOCI_DO_OVERCLOUD=${TOCI_DO_OVERCLOUD:-1}
# Set to 1 if you want a single machine overcloud (uses less resources)
export TOCI_OVERCLOUD_ALL_IN_ONE=${TOCI_OVERCLOUD_ALL_IN_ONE:-0}
# This is needed for help build the index.html if upload results to server
export RESULT_CACHE=${RESULT_CACHE:-$TOCI_CACHE_DIR/results_cache.html}
# Add incubator scripts to path
export PATH=$PATH:$TOCI_WORKING_DIR/tripleo-incubator/scripts
# Should toci remove VM's when finished
export TOCI_CLEANUP=${TOCI_CLEANUP:-0}
# Should toci remove WORKING and LOG dirs when finished
export TOCI_REMOVE=${TOCI_REMOVE:-0}
# Should toci upload results to a server
export TOCI_UPLOAD=${TOCI_UPLOAD:-0}
# irc channel (on freenode) to notify on error
export TOCI_IRC
# bm_poser host resources, use by various scripts
export TOCI_NODE_CPU=${TOCI_NODE_CPU:-1} # Number of cpu's
export TOCI_NODE_MEM=${TOCI_NODE_MEM:-2048} # RAM MB
export TOCI_NODE_DISK=${TOCI_NODE_DISK:-30} # Disk size
export TOCI_ADMIN_TOKEN='admin'
export UNDERCLOUD_ADMIN_PASSWORD='stack'
export OVERCLOUD_ADMIN_PASSWORD='stack'
export OVERCLOUD_DEMO_PASSWORD='stack'

95
toci.sh
View File

@ -1,95 +0,0 @@
#!/usr/bin/env bash
export TOCI_SOURCE_DIR=$(realpath $(dirname $0))
# setup toci env variables
[ -e ~/.toci ] && source ~/.toci # your local toci env setup
source $TOCI_SOURCE_DIR/toci-defaults # defaults for env variables toci expects
[ -e $TOCI_SOURCE_DIR/tocirc ] && source $TOCI_SOURCE_DIR/tocirc # env variables you may want to setup for this run
. $TOCI_SOURCE_DIR/toci_functions.sh
check_dependencies
mkdir -p $TOCI_WORKING_DIR $TOCI_LOG_DIR $TOCI_CACHE_DIR
echo "Starting run $STARTTIME ( $TOCI_WORKING_DIR $TOCI_LOG_DIR )"
# On Exit write relevant toci env to a rc file
trap get_tocienv EXIT
STATUS=0
mark_time Starting git
./toci_git.sh > $TOCI_LOG_DIR/git.out 2>&1 || STATUS=1
# set d-i-b env variables to fetch git repositories from local caches
for repo in $TOCI_WORKING_DIR/*/.git ; do
repo_dir=$(dirname $repo)
repo_name=$(basename $repo_dir)
if [[ "^(tripleo-incubator|diskimage-builder|tripleo-image-elements|tripleo-heat-templates)$" =~ "$repo_name" ]] ; then
continue
fi
export DIB_REPOLOCATION_$repo_name=$repo_dir
done
mark_time Starting pre-cleanup
./toci_cleanup.sh > $TOCI_LOG_DIR/cleanup.out 2>&1
if [ $STATUS == 0 ] ; then
mark_time Starting setup
./toci_setup.sh > $TOCI_LOG_DIR/setup.out 2>&1 || STATUS=1
fi
if [ $STATUS == 0 ] ; then
mark_time Starting test
./toci_test.sh > $TOCI_LOG_DIR/test.out 2>&1 || STATUS=1
fi
if [ $TOCI_CLEANUP == 1 ] ; then
mark_time Starting cleanup
./toci_cleanup.sh >> $TOCI_LOG_DIR/cleanup.out 2>&1 || STATUS=1
fi
mark_time Finished
if [ $TOCI_UPLOAD == 1 ] ; then
cd $(dirname $TOCI_LOG_DIR)
[[ "$TOCI_RESULTS_DST" =~ (.*)@(.*):(.*) ]] || ( echo "Couldn't parse '$TOCI_RESULTS_DST'" && exit 1 )
TOCI_RESULTS_USER=${BASH_REMATCH[1]}
TOCI_RESULTS_HOST=${BASH_REMATCH[2]}
TOCI_RESULTS_PATH=${BASH_REMATCH[3]}
tar -czf - $(basename $TOCI_LOG_DIR) | ssh $TOCI_RESULTS_USER@$TOCI_RESULTS_HOST tar -C $TOCI_RESULTS_PATH -xzf -
touch $RESULT_CACHE
mv $RESULT_CACHE result_cache.html.bck
echo "<html><head/><body>" > index.html
if [ $STATUS == 0 ] ; then
echo "<a href=\"$(basename $TOCI_LOG_DIR)\"\>$STARTTIME : OK</a\><br/\>" > $RESULT_CACHE
else
echo "<a style=\"COLOR: #FF0000\" href=\"$(basename $TOCI_LOG_DIR)\"\>$STARTTIME : ERR</a\><br/\>" > $RESULT_CACHE
fi
# keep only the last 100 runs
head -n 100 result_cache.html.bck >> $RESULT_CACHE
rm result_cache.html.bck
cat $RESULT_CACHE >> index.html
echo "</body></html>" >> index.html
scp index.html $TOCI_RESULTS_USER@$TOCI_RESULTS_HOST:$TOCI_RESULTS_PATH/index.html
ssh $TOCI_RESULTS_USER@$TOCI_RESULTS_HOST "chmod -R 775 $TOCI_RESULTS_PATH/*"
fi
# Send a irc message
if [ -n "$TOCI_IRC" -a $STATUS != 0 ] ; then
send_irc $TOCI_IRC ERROR during toci run, see http://$TOCI_RESULTS_HOST/toci/$(basename $TOCI_LOG_DIR)/
fi
if [ $TOCI_REMOVE == 1 ] ; then
rm -rf $TOCI_WORKING_DIR $TOCI_LOG_DIR
fi
if [ $STATUS != 0 ] ; then
echo ERROR
fi
exit $STATUS

View File

@ -1,12 +0,0 @@
#!/usr/bin/env bash
set -x
for NAME in $(sudo virsh list --name --all | grep "^\(seed\|bootstrap\|baremetal_.*\)$"); do
sudo virsh destroy $NAME
sudo virsh undefine --remove-all-storage $NAME
done
for NAME in $(virsh vol-list default | grep /var/ | awk '{print $1}' | grep "^\(seed\|bootstrap\|baremetal-\)" ); do
sudo virsh vol-delete --pool default $NAME
done

View File

@ -2,8 +2,6 @@
set -e
. toci_functions.sh
if [ ! -e "$TE_DATAFILE" ] ; then
echo "Couldn't find data file"
exit 1

View File

@ -1,108 +0,0 @@
#!/usr/bin/env bash
get_get_repo(){
CACHEDIR=$TOCI_WORKING_DIR/${1/[^\/]*\//}
repo_basename=${1#*/}
if [ ! -e $CACHEDIR ] ; then
git clone https://github.com/$1.git $CACHEDIR
REF_VAR=TOCI_REPOREF_${repo_basename//-/_}
REF=${!REF_VAR}
if [ -n "$REF" ]; then
pushd $CACHEDIR
git reset --hard $REF
popd
fi
else
pushd $CACHEDIR
# Repositories in $TOCI_WORKING_DIR aren't updated but we do fetch origin
# this fetch will make it a little more obvious to a user that upstream has changed
git fetch
popd
fi
apply_patches ${repo_basename} ${repo_basename}*
}
ssh_noprompt(){
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=QUIET -o PasswordAuthentication=no $@
}
scp_noprompt(){
scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=QUIET -o PasswordAuthentication=no $@
}
wait_for(){
LOOPS=$1
SLEEPTIME=$2
shift ; shift
i=0
while [ $i -lt $LOOPS ] ; do
i=$((i + 1))
eval "$@" && return 0 || true
sleep $SLEEPTIME
done
return 1
}
apply_patches(){
pushd $TOCI_WORKING_DIR/$1
if [ -d "$TOCI_SOURCE_DIR/patches/" ]; then
for PATCH in $(find $TOCI_SOURCE_DIR/patches/ -name "$2") ; do
patch -p1 -N < $PATCH || echo Error : could not apply $PATCH >> $TOCI_LOG_DIR/error-applying-patches.log
done
fi
popd
}
mark_time(){
echo $(date) : $@
}
# Get config files and logs from a host for debuging purposes
get_state_from_host(){
ssh_noprompt $1@$2 "( set -x ; ps -ef ; df -h ; uptime ; sudo netstat -lpn ; sudo iptables-save ; sudo ovs-vsctl show ; ip addr ; dpkg -l || rpm -qa) > /var/log/host_info.txt 2>&1 ;
sudo tar -czf - --exclude=udev/hwdb.bin --exclude=selinux/targeted /var/log /etc || true" > $TOCI_LOG_DIR/$2.tgz
}
# On Exit write relevant toci env to a rc file
get_tocienv(){
declare | grep -v '\<declare\>' | grep -e "^PATH=" -e "^http.*proxy" -e "^TOCI_" -e '^DIB_' -e 'CLOUD_ADMIN_PASSWORD' | sed -e 's/^/export /g' > $TOCI_WORKING_DIR/toci_env
# Some IP we don't want to proxy
echo 'export no_proxy=$($TOCI_WORKING_DIR/tripleo-incubator/scripts/get-vm-ip seed),192.0.2.1,192.0.2.2,192.0.2.5,192.0.2.6,192.0.2.7,192.0.2.8' >> $TOCI_WORKING_DIR/toci_env
}
# Sends a message to a freenode irc channel
send_irc(){
exec 3<>/dev/tcp/irc.freenode.net/6667
CHANNEL=$1
shift
MESSAGE=$@
echo "Nick toci-bot" >&3
echo "User toci-bot -i * : hi" >&3
sleep 2
echo "JOIN #$CHANNEL" >&3
echo "PRIVMSG #$CHANNEL :$@" >&3
echo "QUIT" >&3
cat <&3 > /dev/null
}
ERROR(){
echo $@
exit 1
}
# Check for some dependencies
function check_dependencies(){
commands=("patch" "make" "tar" "ssh" "arp" "busybox")
for cmd in "${commands[@]}"; do
which "${cmd}" > /dev/null 2>&1 || ERROR "$cmd: command not found"
done
python -c 'import yaml' > /dev/null 2>&1 || ERROR "Please install PyYAML"
# TODO : why do I need to do this, heat client complains without it
python -c 'import keystoneclient' || ERROR "Please install python-keystoneclient"
export PYTHONPATH=$(python -c 'import keystoneclient; print keystoneclient.__file__.rsplit("/", 1)[0]'):$PYTHONPATH
}

View File

@ -1,48 +0,0 @@
#!/usr/bin/env bash
set -xe
. toci_functions.sh
# Get the tripleO repo's
for repo in 'openstack/tripleo-incubator' 'openstack/diskimage-builder' 'openstack/tripleo-image-elements' 'openstack/tripleo-heat-templates' ; do
if [ ${TOCI_GIT_CHECKOUT:-1} == 1 ] ; then
get_get_repo $repo
else
if [ ! -d "$TOCI_WORKING_DIR/$repo" ]; then
echo "Please checkout $repo to $TOCI_WORKING_DIR or enable TOCI_GIT_CHECKOUT."
fi
fi
done
# Get a local copy of each of the git repositories referenced in
REGEX="^([^ ]+) ([^ ]+) ([/~][^ ]+) ([^ ]+) ?([^ ]*)$"
for sr in $TOCI_WORKING_DIR/*/elements/*/source-repository* ; do
while read line ; do
# ignore blank lines and lines begining in '#'
[[ "$line" == \#* ]] || [[ -z "$line" ]] && continue
if [[ "$line" =~ $REGEX ]] ; then
REPONAME=${BASH_REMATCH[1]//-/_}
REPOTYPE=${BASH_REMATCH[2]}
REPOLOCATION=${BASH_REMATCH[4]}
REPOREF=${BASH_REMATCH[5]:-master}
REPOREF_OVERRIDE=TOCI_REPOREF_$REPONAME
REPOREF=${!REPOREF_OVERRIDE:-$REPOREF}
REPO_DIRECTORY=$TOCI_WORKING_DIR/$REPONAME
if [ $REPOTYPE = git ] ; then
if [ ! -e $REPO_DIRECTORY ] ; then
git clone $REPOLOCATION $REPO_DIRECTORY
pushd $REPO_DIRECTORY
git reset --hard $REPOREF
popd
fi
else
echo "Unsupported repository type"
fi
else
echo "Couldn't parse '$line' as a source repository"
fi
done < $sr
done

View File

@ -1,49 +0,0 @@
#!/usr/bin/env bash
set -xe
. toci_functions.sh
cd $TOCI_WORKING_DIR
# Were going to cache images here
mkdir -p $TOCI_WORKING_DIR/image_cache
# install deps on host machine, this script also restarts libvirt so we have to wait for it to be ready
install-dependencies
wait_for 3 3 ls /var/run/libvirt/libvirt-sock
setup-network
id | grep libvirt || ( echo "You have been added to the libvirt group, this script will now exit but will succeed if run again in a new shell" ; exit 1 )
if [ -f /etc/init.d/libvirt-bin ]; then
sudo service libvirt-bin restart
else
sudo service libvirtd restart
fi
# set default arch for flavors in boot-stack
if [ "$TOCI_DIB_ARCH" != "i386" ]; then
sed -i "s/\"arch\":.*,/\"arch\": \"$TOCI_DIB_ARCH\",/" $TOCI_WORKING_DIR/tripleo-image-elements/elements/seed-stack-config/config.json
fi
# custom power driver config
if [ -n "$TOCI_PM_DRIVER" ]; then
sed -i "s/\"power_manager\":.*,/\"power_manager\": \"$TOCI_PM_DRIVER\",/" $TOCI_WORKING_DIR/tripleo-image-elements/elements/seed-stack-config/config.json
fi
sed -i "s/\"user\": \"stack\",/\"user\": \"`whoami`\",/" $TOCI_WORKING_DIR/tripleo-image-elements/elements/seed-stack-config/config.json
# Create a deployment ramdisk + kernel
$TOCI_WORKING_DIR/diskimage-builder/bin/ramdisk-image-create -x -a $TOCI_DIB_ARCH $NODE_DIST deploy -o deploy-ramdisk
# Make the tripleo image elements accessible to diskimage-builder
export ELEMENTS_PATH=$TOCI_WORKING_DIR/diskimage-builder/elements:$TOCI_WORKING_DIR/tripleo-image-elements/elements
# Setup a seed vm
$TOCI_WORKING_DIR/tripleo-incubator/scripts/setup-seed-vm -a $TOCI_DIB_ARCH
# Boot a seed vm
$TOCI_WORKING_DIR/tripleo-incubator/scripts/boot-seed-vm -a $TOCI_DIB_ARCH $NODE_DIST $TOCI_SEED_EXTRA_ELEMENTS neutron-dhcp-agent
$TOCI_WORKING_DIR/diskimage-builder/bin/disk-image-create $NODE_DIST $TOCI_UNDERCLOUD_EXTRA_ELEMENTS -a $TOCI_DIB_ARCH -o $TOCI_WORKING_DIR/undercloud boot-stack nova-baremetal os-collect-config stackuser local-config neutron-dhcp-agent

View File

@ -1,225 +0,0 @@
#!/usr/bin/env bash
set -xe
. toci_functions.sh
cd $TOCI_WORKING_DIR
SEED_IP=`$TOCI_WORKING_DIR/tripleo-incubator/scripts/get-vm-ip seed`
# Get logs from the node on exit
trap "get_state_from_host root $SEED_IP" EXIT
# Add a route to the baremetal bridge via the seed node
sudo ip route del 192.0.2.0/24 dev virbr0 || true
sudo ip route add 192.0.2.0/24 dev virbr0 via $SEED_IP
cp $TOCI_WORKING_DIR/tripleo-incubator/seedrc $TOCI_WORKING_DIR/seedrc
sed -i "s/\$SEED_IP/$SEED_IP/" $TOCI_WORKING_DIR/seedrc
source $TOCI_WORKING_DIR/seedrc
export no_proxy=$no_proxy,$SEED_IP
# wait for a successful os-refresh-config
if [[ "$NODE_DIST" =~ (.*)ubuntu(.*) ]]; then
wait_for 60 10 ssh_noprompt root@$SEED_IP grep 'Completed phase post-configure' /var/log/upstart/os-collect-config.log
else
wait_for 60 10 ssh_noprompt root@$SEED_IP journalctl -u os-collect-config \| grep \'Completed phase post-configure\'
fi
# init keystone / setup endpoints
init-keystone -p unset unset 192.0.2.1 admin@example.com root@192.0.2.1
setup-endpoints 192.0.2.1 --glance-password unset --heat-password unset --neutron-password unset --nova-password unset
# Make sure nova has had a chance to start responding to requests
wait_for 10 5 nova list
user-config #Adds nova keypair
if [ -n "$TOCI_MACS" ]; then
# For the seed VM we use only the first MAC and power management setting
setup-baremetal $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH "${TOCI_MACS%% *}" seed "${TOCI_PM_IPS%% *}" "${TOCI_PM_USERS%% *}" "${TOCI_PM_PASSWORDS%% *}"
else
export SEED_MACS=$(create-nodes $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH 1)
setup-baremetal $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH "$SEED_MACS" seed
# If MAC's weren't provided then we're using virtual nodes
OVERCLOUD_LIBVIRT_TYPE=${OVERCLOUD_LIBVIRT_TYPE:-";NovaComputeLibvirtType=qemu"}
fi
setup-neutron 192.0.2.2 192.0.2.3 192.0.2.0/24 192.0.2.1 192.0.2.1 ctlplane
# Load images into glance
export DIB_PATH=$TOCI_WORKING_DIR/diskimage-builder
$TOCI_WORKING_DIR/tripleo-incubator/scripts/load-image undercloud.qcow2
keystone role-create --name heat_stack_user
# place the bootstrap public key on host so that it can admin virt
ssh_noprompt root@$SEED_IP "cat /opt/stack/boot-stack/virtual-power-key.pub" >> ~/.ssh/authorized_keys
# Now we have to wait for the bm poseur to appear on the compute node and for the compute node to then
# update the scheduler
if [[ "$NODE_DIST" =~ (.*)ubuntu(.*) ]]; then
wait_for 40 10 ssh_noprompt root@$SEED_IP grep 'Free VCPUS: [^0]' /var/log/upstart/nova-compute.log
else
wait_for 40 10 ssh_noprompt root@$SEED_IP journalctl -u nova-compute -u openstack-nova-compute \| grep \'Free VCPUS: [^0]\'
fi
if [ -n "$TOCI_PM_DRIVER" ]; then
UNDERCLOUD_POWER_MANAGER=${UNDERCLOUD_POWER_MANAGER:-";PowerManager=${TOCI_PM_DRIVER}"}
fi
make -C $TOCI_WORKING_DIR/tripleo-heat-templates undercloud-vm.yaml
heat stack-create -f $TOCI_WORKING_DIR/tripleo-heat-templates/undercloud-vm.yaml -P "PowerUserName=$(whoami);AdminToken=${TOCI_ADMIN_TOKEN};AdminPassword=${UNDERCLOUD_ADMIN_PASSWORD};GlancePassword=${UNDERCLOUD_ADMIN_PASSWORD};HeatPassword=${UNDERCLOUD_ADMIN_PASSWORD};NeutronPassword=${UNDERCLOUD_ADMIN_PASSWORD};NovaPassword=${UNDERCLOUD_ADMIN_PASSWORD};BaremetalArch=${TOCI_DIB_ARCH}${UNDERCLOUD_POWER_MANAGER}" undercloud
# Just sleeping here so that we don't fill the logs with so many loops
sleep 180
heat list
wait_for 40 20 heat list \| grep CREATE_COMPLETE
# Delete the rule that prevent the Fedora bootstrap vm from forwarding
# packets. If the rule doesn't exist just do nothing...
ssh_noprompt root@$SEED_IP iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited || true
wait_for 20 15 ping -c 1 $(nova list | grep undercloud | sed -e "s/.*=\(.*\) .*/\1/g")
export UNDERCLOUD_IP=$(nova list | grep ctlplane | sed -e "s/.*=\([0-9.]*\).*/\1/")
cp $TOCI_WORKING_DIR/tripleo-incubator/undercloudrc $TOCI_WORKING_DIR/undercloudrc
sed -i -e "s/\$UNDERCLOUD_IP/$UNDERCLOUD_IP/g" $TOCI_WORKING_DIR/undercloudrc
source $TOCI_WORKING_DIR/undercloudrc
export no_proxy=$no_proxy,$UNDERCLOUD_IP
# Make the tripleo image elements accessible to diskimage-builder
export ELEMENTS_PATH=$TOCI_WORKING_DIR/diskimage-builder/elements:$TOCI_WORKING_DIR/tripleo-image-elements/elements
if [ "$TOCI_DO_OVERCLOUD" = "1" ] ; then
if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "1" ] ; then
$TOCI_WORKING_DIR/diskimage-builder/bin/disk-image-create $NODE_DIST $TOCI_OVERCLOUD_EXTRA_ELEMENTS -a $TOCI_DIB_ARCH -o overcloud-all-in-one boot-stack cinder nova-compute nova-kvm neutron-openvswitch-agent os-collect-config stackuser neutron-network-node dhcp-all-interfaces swift-proxy swift-storage local-config
else
$TOCI_WORKING_DIR/diskimage-builder/bin/disk-image-create $NODE_DIST $TOCI_OVERCLOUD_EXTRA_ELEMENTS -a $TOCI_DIB_ARCH -o overcloud-control boot-stack cinder os-collect-config neutron-network-node dhcp-all-interfaces stackuser swift-proxy swift-storage local-config
fi
fi
# Also get undercloud logs
trap "get_state_from_host root $SEED_IP ; get_state_from_host heat-admin $UNDERCLOUD_IP" EXIT
# wait for a successful os-refresh-config
if [[ "$NODE_DIST" =~ (.*)ubuntu(.*) ]]; then
wait_for 60 10 ssh_noprompt heat-admin@$UNDERCLOUD_IP grep 'Completed phase post-configure' /var/log/upstart/os-collect-config.log
else
wait_for 60 10 ssh_noprompt heat-admin@$UNDERCLOUD_IP sudo journalctl -u os-collect-config \| grep \'Completed phase post-configure\'
fi
# setup keystone endpoints
init-keystone -p $UNDERCLOUD_ADMIN_PASSWORD $TOCI_ADMIN_TOKEN $UNDERCLOUD_IP admin@example.com heat-admin@$UNDERCLOUD_IP
setup-endpoints $UNDERCLOUD_IP --glance-password $UNDERCLOUD_ADMIN_PASSWORD --heat-password $UNDERCLOUD_ADMIN_PASSWORD --neutron-password $UNDERCLOUD_ADMIN_PASSWORD --nova-password $UNDERCLOUD_ADMIN_PASSWORD
# Make sure nova has had a chance to start responding to requests
wait_for 10 5 nova list
if [ "$TOCI_DO_OVERCLOUD" != "1" ] ; then
exit 0
fi
user-config
if [ -n "$TOCI_MACS" ]; then
# For the undercloud we pop off the first MAC and power management settings
# since they have already been used by the seed VM
setup-baremetal $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH "${TOCI_MACS#[^ ]* }" undercloud "${TOCI_PM_IPS#[^ ]* }" "${TOCI_PM_USERS#[^ ]* }" "${TOCI_PM_PASSWORDS#[^ ]* }"
else
export UNDERCLOUD_MACS=$(create-nodes $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH 2)
setup-baremetal $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH "$UNDERCLOUD_MACS" undercloud
fi
setup-neutron 192.0.2.5 192.0.2.24 192.0.2.0/24 192.0.2.1 $UNDERCLOUD_IP ctlplane
ssh_noprompt heat-admin@$UNDERCLOUD_IP "cat /opt/stack/boot-stack/virtual-power-key.pub" >> ~/.ssh/authorized_keys
if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "0" ] ; then
$TOCI_WORKING_DIR/diskimage-builder/bin/disk-image-create $NODE_DIST $TOCI_OVERCLOUD_EXTRA_ELEMENTS -a $TOCI_DIB_ARCH -o overcloud-compute nova-compute nova-kvm neutron-openvswitch-agent os-collect-config stackuser local-config
fi
if [ -d /var/log/upstart ]; then
wait_for 40 10 ssh_noprompt heat-admin@$UNDERCLOUD_IP grep 'Free VCPUS: [^0]' /var/log/upstart/nova-compute.log
else
wait_for 40 10 ssh_noprompt heat-admin@$UNDERCLOUD_IP sudo journalctl -u nova-compute -u openstack-nova-compute \| grep \'Free VCPUS: [^0]\'
fi
if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "1" ] ; then
load-image overcloud-all-in-one.qcow2
else
load-image overcloud-control.qcow2
load-image overcloud-compute.qcow2
fi
if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "1" ] ; then
heat stack-create -f $TOCI_WORKING_DIR/tripleo-heat-templates/overcloud-all-in-one.yaml -P "AdminToken=${TOCI_ADMIN_TOKEN};AdminPassword=${OVERCLOUD_ADMIN_PASSWORD};CinderPassword=${OVERCLOUD_ADMIN_PASSWORD};GlancePassword=${OVERCLOUD_ADMIN_PASSWORD};HeatPassword=${OVERCLOUD_ADMIN_PASSWORD};NeutronPassword=${OVERCLOUD_ADMIN_PASSWORD};NovaPassword=${OVERCLOUD_ADMIN_PASSWORD};SwiftPassword=${OVERCLOUD_ADMIN_PASSWORD};Image=overcloud-all-in-one${OVERCLOUD_LIBVIRT_TYPE}" overcloud
else
make -C $TOCI_WORKING_DIR/tripleo-heat-templates overcloud.yaml
heat stack-create -f $TOCI_WORKING_DIR/tripleo-heat-templates/overcloud.yaml -P "AdminToken=${TOCI_ADMIN_TOKEN};AdminPassword=${OVERCLOUD_ADMIN_PASSWORD};CinderPassword=${OVERCLOUD_ADMIN_PASSWORD};GlancePassword=${OVERCLOUD_ADMIN_PASSWORD};HeatPassword=${OVERCLOUD_ADMIN_PASSWORD};NeutronPassword=${OVERCLOUD_ADMIN_PASSWORD};NovaPassword=${OVERCLOUD_ADMIN_PASSWORD};SwiftPassword=${OVERCLOUD_ADMIN_PASSWORD};notcomputeImage=overcloud-control${OVERCLOUD_LIBVIRT_TYPE}" overcloud
fi
sleep 161
wait_for 50 20 heat list \| grep CREATE_COMPLETE
if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "1" ] ; then
export OVERCLOUD_IP=$(nova list | grep ctlplane | grep overcloud-allinone | sed -e "s/.*=\([0-9.]*\).*/\1/")
else
export OVERCLOUD_IP=$(nova list | grep ctlplane | grep notcompute | sed -e "s/.*=\([0-9.]*\).*/\1/")
fi
cp $TOCI_WORKING_DIR/tripleo-incubator/overcloudrc $TOCI_WORKING_DIR/overcloudrc
sed -i -e "s/\$OVERCLOUD_IP/$OVERCLOUD_IP/g" $TOCI_WORKING_DIR/overcloudrc
source $TOCI_WORKING_DIR/overcloudrc
export no_proxy=$no_proxy,$OVERCLOUD_IP
# Also get overcloud logs
trap "get_state_from_host root $SEED_IP ; get_state_from_host heat-admin $UNDERCLOUD_IP ; get_state_from_host heat-admin $OVERCLOUD_IP" EXIT
# wait for a successful os-refresh-config
ssh_noprompt heat-admin@$UNDERCLOUD_IP sudo iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited || true
if [[ "$NODE_DIST" =~ (.*)ubuntu(.*) ]]; then
wait_for 60 10 ssh_noprompt heat-admin@$OVERCLOUD_IP grep 'Completed phase post-configure' /var/log/upstart/os-collect-config.log
else
wait_for 60 10 ssh_noprompt heat-admin@$OVERCLOUD_IP sudo journalctl -u os-collect-config \| grep \'Completed phase post-configure\'
fi
# setup keystone endpoints
init-keystone -p $OVERCLOUD_ADMIN_PASSWORD $TOCI_ADMIN_TOKEN $OVERCLOUD_IP admin@example.com heat-admin@$OVERCLOUD_IP
setup-endpoints $OVERCLOUD_IP --glance-password $OVERCLOUD_ADMIN_PASSWORD --heat-password $OVERCLOUD_ADMIN_PASSWORD --neutron-password $OVERCLOUD_ADMIN_PASSWORD --nova-password $OVERCLOUD_ADMIN_PASSWORD --swift-password $OVERCLOUD_ADMIN_PASSWORD
keystone role-create --name heat_stack_user
user-config
setup-neutron "" "" 10.0.0.0/8 "" "" "" 192.0.2.45 192.0.2.64 192.0.2.0/24
# Make sure nova has had a chance to start responding to requests
wait_for 10 5 nova list
# Lets add a cirros image to the overcloud
curl -L https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-i386-disk.img | glance image-create --name cirros --disk-format qcow2 --container-format bare --is-public 1
# create demo user
os-adduser -p $OVERCLOUD_DEMO_PASSWORD demo demo@example.com
source $TRIPLEO_ROOT/tripleo-incubator/overcloudrc-user
user-config
# Start and test a image on the overcloud
nova boot --key-name default --flavor m1.tiny --image cirros --key_name default demo
sleep 20 # give the port a chance to appear
PORT=$(neutron port-list -f csv -c id --quote none | tail -n1)
neutron security-group-rule-create default --protocol tcp --direction ingress --port-range-min 22 --port-range-max 22
neutron security-group-rule-create default --protocol icmp --direction ingress
IP=$(neutron floatingip-create ext-net --port-id "${PORT//[[:space:]]/}" | grep -o -P "192[0-9.]*")
wait_for 10 5 ping -c 1 $IP

View File

@ -1,18 +0,0 @@
#!/usr/bin/env bash
. toci_functions.sh
LOCKFILE=/var/tmp/toci.lock
GITREF=${1:-origin/master}
export TOCI_LOG_DIR=${TOCI_LOG_DIR:-$(mktemp -d --tmpdir toci_logs_XXXXXXX)}
RUNLOG=$TOCI_LOG_DIR/run.out
cd $(dirname $0)
git fetch origin | tee -a $RUNLOG 2>&1
# Exit if there is a script already running, otherwise update repo
# TODO : fix small race condition here (probably not a problem)
flock -x -n $LOCKFILE git reset --hard $GITREF | tee -a $RUNLOG 2>&1 || exit 0
flock -x -n $LOCKFILE ./toci.sh | tee -a $RUNLOG 2>&1