diff --git a/FAQ.md b/FAQ.md deleted file mode 100644 index 918819ffc..000000000 --- a/FAQ.md +++ /dev/null @@ -1,94 +0,0 @@ -Before Running any commands below ensure you have sourced the toci_env file - - $ cd /opt/toci - $ . toci_env - -**Q. How can I ssh to the seed vm** - -``` -$ ssh root@$($TOCI_WORKING_DIR/incubator/scripts/get-vm-ip seed) -``` - -**Q. How can I ssh to the undercloud controller** -``` -$ ssh heat-admin@192.0.2.2 -``` - -**Q. How do I list the nodes making up the overcloud** -``` - $ . undercloudrc - $ nova list -``` - -**Q. How can I create more bm_poseur nodes and add them to my undercloud** -``` - $ . undercloudrc - # - $ create-nodes 1 1024 10 i386 5 - $ export MACS=$($TOCI_WORKING_DIR/bm_poseur/bm_poseur get-macs) - $ setup-baremetal 1 768 10 i386 all -``` - -**Q. How can I create a new image, using the elements fedora, - selinux-permissive and stackuser** -``` - $ ./diskimage-builder/bin/disk-image-create -a i386 -o myimage fedora selinux-permissive stackuser -``` - -**Q. I have a heat template, how can I start a stack on the undercloud using - it** -``` - $ . undercloudrc - $ heat stack-create -f /path/to/myheattemplate.yaml mystack -``` - -**Q. I want to rerun toci without having to build a new set of images each - time.** - -Yes, toci can apply a patch to disk-imagebuilder that will cause it place -build images in a /opt/toci/image_cache, all other runs of toci will use -these files instead of building a new one. To use the patch move it into -the patches directory where you clone toci to. -``` - $ cp patches_dev/diskimage-builder-0001-Save-images-in-a-toci-cache-file-or-use-if-present.patch patches -``` - -Do not forget to remove images from /opt/toci/image_cache if you change -anything that would require a new image build - -**Q. There is a new version of diskimage-builder I want to update to and use it - to rebuild images** -``` - $ rm -rf /opt/toci/diskimage-builder - $ ./toci.sh -``` - -**Q. I would like to redeploy tripleo with a patch that has been submitted to - nova** -``` - $ cd /opt/toci/nova - # You can get the git reference from the patchset in gerrit - $ git fetch https://review.openstack.org/openstack/nova refs/changes/36/45536/1 - $ git reset --hard FETCH_HEAD - $ cd /path/to/toci - - # Remove /opt/toci/image_cache/* if you are using the image caching described above - - # Run toci - $ ./toci.sh -``` -**Q. My toci run failed which logfile should I be looking in?** - -Toci gathers various log file into a single directory (ouput at the start -of the run), the content of this directory typically looks like this. -``` -./git.out - output from various git commands as they clone the repositories -./error-applying-patches.log - if this exists a patch from the patches directory failed to apply -./setup.out - output from toci_setup.sh -./test.out - output from toci_test.sh -./cleanup.out - output from tcoi_cleanup.sh -./192.168.122.218.tgz - tarball of /var/logs and /etc on seed VM -./192.0.2.2.tgz - tarball of /var/logs and /etc on undercloud -./192.0.2.5.tgz - tarball of /var/logs and /etc on overcloud notcompute -``` -Some of these files may not exist if toci failed to complete diff --git a/README.md b/README.md index 1a1c3ce54..f96a83536 100644 --- a/README.md +++ b/README.md @@ -1,109 +1,17 @@ toci ==== - Description ----------- TripleO CI test framework. -By default toci builds images for seed, undercloud and overcloud hosts, it -then uses bare metal poseur nodes to set up a virtualized TripleO environment. - -Options also exist so you can specify hosts to setup TripleO on a real -baremetal environment. - -Configuration -------------- - -If using toci to setup TripleO on a virtualized environment we recommend you -setup a proxy server for http traffic - -edit ~/.toci and add a value for -```bash -export http_proxy=http://1.2.3.4:3128 -``` - -See toci-defaults for a list of additional environment variables that can be -defined in ~/.toci in order to control things like -* Changing the architecture to amd64 -* Deploying TripleO images on real baremetal hosts -* Increasing the resources allocated to bm_poseur nodes -* notifying an irc channel and uploading toci results to a server (used if - running toci as a CI framwork) - -Using Toci to setup a dev environment -------------------------------------- - -I usually do this as root, in theory it will also work as a non privileged -user. - - $ git clone https://github.com/openstack-infra/tripleo-ci.git toci - $ cd toci - $ vi ~/.toci # Will work without a proxy but can be a lot slower - export http_proxy=http://1.2.3.4:8080 - -To deploy toci run the command command - - $ ./toci.sh - -Toci will start with a line outputting the working and log directories e.g. -Starting run Wed 3 Jul 11:46:39 IST 2013 ( /opt/toci /tmp/toci_logs_nGnrhLN ) - -Once it ran successfully (ERROR wasn't echo'd to the terminal) you should have -* seed vm -* undercloud vm -* overcloud controller vm -* overcloud compute vm - -NOTE: toci will now have cloned the dependency git repositories to /opt/toci, -If you rerun toci it will NOT re-clone these again, if you would like it to -reclone the most recent version of any of these repositories you can simply -delete it before running toci. - -If you would like to test a specific change locally in TripleO you can simply -edit the repository locally and commit this change to its master branch and -rerun toci. See the FAQ if you would like to do this without rebuilding all of -the images (e.g. For speed reasons, you would only like to rebuild the -overcloud images and reuse the previously built seed and undercloud image) - -See FAQ.md for more information on how to use the TripleO deployment - -Using Toci as a CI framework for TripleO ----------------------------------------- - -If running toci as part of a automated CI job several environment variables -can be defined to help make toci more suitable e.g. - -*So toci cleans up after itself* -```bash -export TOCI_UPLOAD=1 -export TOCI_REMOVE=1 -``` - -*scp logs to a server when finished* -```bash -TOCI_RESULTS_DST=user@1.2.3.4:/var/www/html/toci -``` - -*Notify a freenode irc channel upon error* -```bash -TOCI_IRC=channeltonotify -``` - -*Only build and deploy the seed and undercloud* -```bash -export TOCI_DO_OVERCLOUD=0 -``` - - -Use toci to deploy on real baremetal ------------------------------------ -```bash -export TOCI_PM_DRIVER="nova.virt.baremetal.ipmi.IPMI" -#Space delimited, aligned in order -export TOCI_MACS="84:2b:22:11:11:11 84:2b:22:11:11:12 84:2b:22:11:11:13" -export TOCI_PM_IPS="10.16.111.111 10.16.111.112 10.16.111.113" -export TOCI_PM_USERS="root root root" -export TOCI_PM_PASSWORDS="passwd passwd passwd" -``` +Tools to help run CI jobs for TripleO. Includes things like: +* Shell scripts to help execute jobs on CI slave nodes (Jenkins slaves) +* A test environment broker framework which uses a client-server + model to execute jobs on a remote bare metal machine in an isolated + test environment (using VMs). +* Image elements to help build images for the test environment + broker nodes. +* Heat templates to help deploy and maintain test environment nodes + using an undercloud. diff --git a/patches/README b/patches/README deleted file mode 100644 index 586efc246..000000000 --- a/patches/README +++ /dev/null @@ -1,2 +0,0 @@ -Put patches in here that need to be applied to the tripleo repositories -for more info see toci_git.sh diff --git a/patches/diskimage-builder-Allow-heat-admin-sudo-without-tty.patch b/patches/diskimage-builder-Allow-heat-admin-sudo-without-tty.patch deleted file mode 100644 index 4712d41c0..000000000 --- a/patches/diskimage-builder-Allow-heat-admin-sudo-without-tty.patch +++ /dev/null @@ -1,24 +0,0 @@ -From 76fb773073a02d9d6c39d0b0150947573c6de4e9 Mon Sep 17 00:00:00 2001 -From: Derek Higgins -Date: Fri, 19 Jul 2013 10:21:27 +0100 -Subject: [PATCH] Allow heat-admin sudo without tty - ---- - elements/rpm-distro/pre-install.d/00-allow-root-sudo | 5 +++++ - 1 file changed, 5 insertions(+) - -diff --git a/elements/rpm-distro/pre-install.d/00-allow-root-sudo b/elements/rpm-distro/pre-install.d/00-allow-root-sudo -index 4736ac8..ff59cfb 100755 ---- a/elements/rpm-distro/pre-install.d/00-allow-root-sudo -+++ b/elements/rpm-distro/pre-install.d/00-allow-root-sudo -@@ -5,3 +5,8 @@ set -e - echo "Defaults:root !requiretty" >> /etc/sudoers.d/root-notty - chmod 0440 /etc/sudoers.d/root-notty - visudo -c -+ -+# For toci we need heat-admin to sudo without tty -+echo "Defaults:heat-admin !requiretty" >> /etc/sudoers.d/heat-admin-notty -+chmod 0440 /etc/sudoers.d/heat-admin-notty -+visudo -c --- -1.8.1.4 diff --git a/patches_dev/diskimage-builder-Save-images-in-a-toci-cache-file-or-use-if-present.patch b/patches_dev/diskimage-builder-Save-images-in-a-toci-cache-file-or-use-if-present.patch deleted file mode 100644 index c39ecb3bf..000000000 --- a/patches_dev/diskimage-builder-Save-images-in-a-toci-cache-file-or-use-if-present.patch +++ /dev/null @@ -1,44 +0,0 @@ -From 448fbaa131521e6060f9d87c83098c8cb90696ea Mon Sep 17 00:00:00 2001 -From: Derek Higgins -Date: Fri, 19 Jul 2013 13:55:41 +0100 -Subject: [PATCH] Save images in a toci cache file or use if present - -FOR DEV ONLY -If you want to -o skip building images in place of a cached version, -o mv first-boot scripts instead of rm - -place this file in patches. ---- - bin/disk-image-create | 8 ++++++++ - 1 file changed, 8 insertions(+) - -diff --git a/bin/disk-image-create b/bin/disk-image-create -index de14433..fba1e1e 100755 ---- a/bin/disk-image-create -+++ b/bin/disk-image-create -@@ -105,6 +105,13 @@ arg_to_elements "$@" - - IMAGE_NAME=${IMAGE_NAME%%\.${IMAGE_TYPE}} - -+CACHEFILE=$TOCI_WORKING_DIR/image_cache/$(basename $IMAGE_NAME.$IMAGE_TYPE) -+if [ -e $CACHEFILE ] ; then -+ echo Using $CACHEFILE -+ cp $CACHEFILE $IMAGE_NAME.$IMAGE_TYPE -+ exit 0 -+fi -+ - mk_build_dir - create_base - run_d extra-data -@@ -151,6 +158,7 @@ unmount_image - - if [ "$IS_RAMDISK" == "0" ]; then - compress_and_save_image $IMAGE_NAME.$IMAGE_TYPE -+ cp $IMAGE_NAME.$IMAGE_TYPE $TOCI_WORKING_DIR/image_cache/$(basename $IMAGE_NAME.$IMAGE_TYPE) - else - remove_image - fi --- -1.8.1.4 - diff --git a/toci-defaults b/toci-defaults deleted file mode 100644 index 2339e58d7..000000000 --- a/toci-defaults +++ /dev/null @@ -1,71 +0,0 @@ -#!/usr/bin/bash - -# Time the script was started -export STARTTIME=$(date) - -# All toci working files should go here -export TOCI_WORKING_DIR=${TOCI_WORKING_DIR:-/opt/toci} -export TRIPLEO_ROOT=$TOCI_WORKING_DIR # some scripts are expecting this - -# Should toci get tripleo repositories -export TOCI_GIT_CHECKOUT=${TOCI_GIT_CHECKOUT:-1} - -# Files that should be cached between runs should go in here -# e.g. downloaded images, git repo's etc... -export TOCI_CACHE_DIR=${TOCI_CACHE_DIR:-/var/tmp/toci_cache} - -# Any files to be uploaded to results server go here -export TOCI_LOG_DIR=${TOCI_LOG_DIR:-$(mktemp -d --tmpdir toci_logs_XXXXXXX)} - -# Set TOCI_ARCH to x86_64 to build 64 bit arch -export TOCI_ARCH=${TOCI_ARCH:-'i386'} -export TOCI_DIB_ARCH='i386' -if [ "$TOCI_ARCH" == 'x86_64' ]; then - export TOCI_DIB_ARCH='amd64' -fi - -# This gets passed into diskimage builder -export NODE_DIST=${NODE_DIST:-'pip-cache fedora selinux-permissive'} - -# custom diskimage builder elements for the seed -export TOCI_SEED_EXTRA_ELEMENTS=${TOCI_SEED_EXTRA_ELEMENTS:-''} - -# custom diskimage builder elements for the undercloud -export TOCI_UNDERCLOUD_EXTRA_ELEMENTS=${TOCI_UNDERCLOUD_EXTRA_ELEMENTS:-''} - -# custom diskimage builder elements for the overcloud -export TOCI_OVERCLOUD_EXTRA_ELEMENTS=${TOCI_OVERCLOUD_EXTRA_ELEMENTS:-''} - -# Set to 0 if you don't want an overcloud to be started -export TOCI_DO_OVERCLOUD=${TOCI_DO_OVERCLOUD:-1} - -# Set to 1 if you want a single machine overcloud (uses less resources) -export TOCI_OVERCLOUD_ALL_IN_ONE=${TOCI_OVERCLOUD_ALL_IN_ONE:-0} - -# This is needed for help build the index.html if upload results to server -export RESULT_CACHE=${RESULT_CACHE:-$TOCI_CACHE_DIR/results_cache.html} - -# Add incubator scripts to path -export PATH=$PATH:$TOCI_WORKING_DIR/tripleo-incubator/scripts - -# Should toci remove VM's when finished -export TOCI_CLEANUP=${TOCI_CLEANUP:-0} - -# Should toci remove WORKING and LOG dirs when finished -export TOCI_REMOVE=${TOCI_REMOVE:-0} - -# Should toci upload results to a server -export TOCI_UPLOAD=${TOCI_UPLOAD:-0} - -# irc channel (on freenode) to notify on error -export TOCI_IRC - -# bm_poser host resources, use by various scripts -export TOCI_NODE_CPU=${TOCI_NODE_CPU:-1} # Number of cpu's -export TOCI_NODE_MEM=${TOCI_NODE_MEM:-2048} # RAM MB -export TOCI_NODE_DISK=${TOCI_NODE_DISK:-30} # Disk size - -export TOCI_ADMIN_TOKEN='admin' -export UNDERCLOUD_ADMIN_PASSWORD='stack' -export OVERCLOUD_ADMIN_PASSWORD='stack' -export OVERCLOUD_DEMO_PASSWORD='stack' diff --git a/toci.sh b/toci.sh deleted file mode 100755 index 783970337..000000000 --- a/toci.sh +++ /dev/null @@ -1,95 +0,0 @@ -#!/usr/bin/env bash - -export TOCI_SOURCE_DIR=$(realpath $(dirname $0)) - -# setup toci env variables -[ -e ~/.toci ] && source ~/.toci # your local toci env setup -source $TOCI_SOURCE_DIR/toci-defaults # defaults for env variables toci expects -[ -e $TOCI_SOURCE_DIR/tocirc ] && source $TOCI_SOURCE_DIR/tocirc # env variables you may want to setup for this run - -. $TOCI_SOURCE_DIR/toci_functions.sh - -check_dependencies - -mkdir -p $TOCI_WORKING_DIR $TOCI_LOG_DIR $TOCI_CACHE_DIR - -echo "Starting run $STARTTIME ( $TOCI_WORKING_DIR $TOCI_LOG_DIR )" - -# On Exit write relevant toci env to a rc file -trap get_tocienv EXIT - -STATUS=0 - -mark_time Starting git -./toci_git.sh > $TOCI_LOG_DIR/git.out 2>&1 || STATUS=1 - -# set d-i-b env variables to fetch git repositories from local caches -for repo in $TOCI_WORKING_DIR/*/.git ; do - repo_dir=$(dirname $repo) - repo_name=$(basename $repo_dir) - if [[ "^(tripleo-incubator|diskimage-builder|tripleo-image-elements|tripleo-heat-templates)$" =~ "$repo_name" ]] ; then - continue - fi - export DIB_REPOLOCATION_$repo_name=$repo_dir -done - -mark_time Starting pre-cleanup -./toci_cleanup.sh > $TOCI_LOG_DIR/cleanup.out 2>&1 -if [ $STATUS == 0 ] ; then - mark_time Starting setup - ./toci_setup.sh > $TOCI_LOG_DIR/setup.out 2>&1 || STATUS=1 -fi - -if [ $STATUS == 0 ] ; then - mark_time Starting test - ./toci_test.sh > $TOCI_LOG_DIR/test.out 2>&1 || STATUS=1 -fi - -if [ $TOCI_CLEANUP == 1 ] ; then - mark_time Starting cleanup - ./toci_cleanup.sh >> $TOCI_LOG_DIR/cleanup.out 2>&1 || STATUS=1 -fi - -mark_time Finished - -if [ $TOCI_UPLOAD == 1 ] ; then - cd $(dirname $TOCI_LOG_DIR) - - [[ "$TOCI_RESULTS_DST" =~ (.*)@(.*):(.*) ]] || ( echo "Couldn't parse '$TOCI_RESULTS_DST'" && exit 1 ) - TOCI_RESULTS_USER=${BASH_REMATCH[1]} - TOCI_RESULTS_HOST=${BASH_REMATCH[2]} - TOCI_RESULTS_PATH=${BASH_REMATCH[3]} - - tar -czf - $(basename $TOCI_LOG_DIR) | ssh $TOCI_RESULTS_USER@$TOCI_RESULTS_HOST tar -C $TOCI_RESULTS_PATH -xzf - - touch $RESULT_CACHE - mv $RESULT_CACHE result_cache.html.bck - echo "" > index.html - if [ $STATUS == 0 ] ; then - echo "$STARTTIME : OK
" > $RESULT_CACHE - else - echo "$STARTTIME : ERR
" > $RESULT_CACHE - fi - # keep only the last 100 runs - head -n 100 result_cache.html.bck >> $RESULT_CACHE - rm result_cache.html.bck - cat $RESULT_CACHE >> index.html - echo "" >> index.html - - scp index.html $TOCI_RESULTS_USER@$TOCI_RESULTS_HOST:$TOCI_RESULTS_PATH/index.html - ssh $TOCI_RESULTS_USER@$TOCI_RESULTS_HOST "chmod -R 775 $TOCI_RESULTS_PATH/*" - -fi - -# Send a irc message -if [ -n "$TOCI_IRC" -a $STATUS != 0 ] ; then - send_irc $TOCI_IRC ERROR during toci run, see http://$TOCI_RESULTS_HOST/toci/$(basename $TOCI_LOG_DIR)/ -fi - -if [ $TOCI_REMOVE == 1 ] ; then - rm -rf $TOCI_WORKING_DIR $TOCI_LOG_DIR -fi - -if [ $STATUS != 0 ] ; then - echo ERROR -fi -exit $STATUS diff --git a/toci_cleanup.sh b/toci_cleanup.sh deleted file mode 100755 index dd2efcdbb..000000000 --- a/toci_cleanup.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env bash - -set -x - -for NAME in $(sudo virsh list --name --all | grep "^\(seed\|bootstrap\|baremetal_.*\)$"); do - sudo virsh destroy $NAME - sudo virsh undefine --remove-all-storage $NAME -done - -for NAME in $(virsh vol-list default | grep /var/ | awk '{print $1}' | grep "^\(seed\|bootstrap\|baremetal-\)" ); do - sudo virsh vol-delete --pool default $NAME -done diff --git a/toci_devtest.sh b/toci_devtest.sh index ff1fadc41..f07e50e1d 100755 --- a/toci_devtest.sh +++ b/toci_devtest.sh @@ -2,8 +2,6 @@ set -e -. toci_functions.sh - if [ ! -e "$TE_DATAFILE" ] ; then echo "Couldn't find data file" exit 1 diff --git a/toci_functions.sh b/toci_functions.sh deleted file mode 100644 index 4c2eebcf4..000000000 --- a/toci_functions.sh +++ /dev/null @@ -1,108 +0,0 @@ -#!/usr/bin/env bash - -get_get_repo(){ - CACHEDIR=$TOCI_WORKING_DIR/${1/[^\/]*\//} - repo_basename=${1#*/} - if [ ! -e $CACHEDIR ] ; then - git clone https://github.com/$1.git $CACHEDIR - REF_VAR=TOCI_REPOREF_${repo_basename//-/_} - REF=${!REF_VAR} - if [ -n "$REF" ]; then - pushd $CACHEDIR - git reset --hard $REF - popd - fi - else - pushd $CACHEDIR - # Repositories in $TOCI_WORKING_DIR aren't updated but we do fetch origin - # this fetch will make it a little more obvious to a user that upstream has changed - git fetch - popd - fi - apply_patches ${repo_basename} ${repo_basename}* -} - -ssh_noprompt(){ - ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=QUIET -o PasswordAuthentication=no $@ -} - -scp_noprompt(){ - scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=QUIET -o PasswordAuthentication=no $@ -} - -wait_for(){ - LOOPS=$1 - SLEEPTIME=$2 - shift ; shift - i=0 - while [ $i -lt $LOOPS ] ; do - i=$((i + 1)) - eval "$@" && return 0 || true - sleep $SLEEPTIME - done - return 1 -} - -apply_patches(){ - pushd $TOCI_WORKING_DIR/$1 - if [ -d "$TOCI_SOURCE_DIR/patches/" ]; then - for PATCH in $(find $TOCI_SOURCE_DIR/patches/ -name "$2") ; do - patch -p1 -N < $PATCH || echo Error : could not apply $PATCH >> $TOCI_LOG_DIR/error-applying-patches.log - done - fi - popd -} - -mark_time(){ - echo $(date) : $@ -} - -# Get config files and logs from a host for debuging purposes -get_state_from_host(){ - ssh_noprompt $1@$2 "( set -x ; ps -ef ; df -h ; uptime ; sudo netstat -lpn ; sudo iptables-save ; sudo ovs-vsctl show ; ip addr ; dpkg -l || rpm -qa) > /var/log/host_info.txt 2>&1 ; - sudo tar -czf - --exclude=udev/hwdb.bin --exclude=selinux/targeted /var/log /etc || true" > $TOCI_LOG_DIR/$2.tgz -} - -# On Exit write relevant toci env to a rc file -get_tocienv(){ - declare | grep -v '\' | grep -e "^PATH=" -e "^http.*proxy" -e "^TOCI_" -e '^DIB_' -e 'CLOUD_ADMIN_PASSWORD' | sed -e 's/^/export /g' > $TOCI_WORKING_DIR/toci_env - # Some IP we don't want to proxy - echo 'export no_proxy=$($TOCI_WORKING_DIR/tripleo-incubator/scripts/get-vm-ip seed),192.0.2.1,192.0.2.2,192.0.2.5,192.0.2.6,192.0.2.7,192.0.2.8' >> $TOCI_WORKING_DIR/toci_env -} - -# Sends a message to a freenode irc channel -send_irc(){ - exec 3<>/dev/tcp/irc.freenode.net/6667 - - CHANNEL=$1 - shift - MESSAGE=$@ - - echo "Nick toci-bot" >&3 - echo "User toci-bot -i * : hi" >&3 - sleep 2 - echo "JOIN #$CHANNEL" >&3 - echo "PRIVMSG #$CHANNEL :$@" >&3 - echo "QUIT" >&3 - - cat <&3 > /dev/null -} - -ERROR(){ - echo $@ - exit 1 -} - -# Check for some dependencies -function check_dependencies(){ - commands=("patch" "make" "tar" "ssh" "arp" "busybox") - for cmd in "${commands[@]}"; do - which "${cmd}" > /dev/null 2>&1 || ERROR "$cmd: command not found" - done - - python -c 'import yaml' > /dev/null 2>&1 || ERROR "Please install PyYAML" - - # TODO : why do I need to do this, heat client complains without it - python -c 'import keystoneclient' || ERROR "Please install python-keystoneclient" - export PYTHONPATH=$(python -c 'import keystoneclient; print keystoneclient.__file__.rsplit("/", 1)[0]'):$PYTHONPATH -} diff --git a/toci_git.sh b/toci_git.sh deleted file mode 100755 index 86c8a6be7..000000000 --- a/toci_git.sh +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env bash - -set -xe -. toci_functions.sh - -# Get the tripleO repo's -for repo in 'openstack/tripleo-incubator' 'openstack/diskimage-builder' 'openstack/tripleo-image-elements' 'openstack/tripleo-heat-templates' ; do - if [ ${TOCI_GIT_CHECKOUT:-1} == 1 ] ; then - get_get_repo $repo - else - if [ ! -d "$TOCI_WORKING_DIR/$repo" ]; then - echo "Please checkout $repo to $TOCI_WORKING_DIR or enable TOCI_GIT_CHECKOUT." - fi - fi -done - -# Get a local copy of each of the git repositories referenced in -REGEX="^([^ ]+) ([^ ]+) ([/~][^ ]+) ([^ ]+) ?([^ ]*)$" -for sr in $TOCI_WORKING_DIR/*/elements/*/source-repository* ; do - while read line ; do - # ignore blank lines and lines begining in '#' - [[ "$line" == \#* ]] || [[ -z "$line" ]] && continue - if [[ "$line" =~ $REGEX ]] ; then - REPONAME=${BASH_REMATCH[1]//-/_} - REPOTYPE=${BASH_REMATCH[2]} - REPOLOCATION=${BASH_REMATCH[4]} - REPOREF=${BASH_REMATCH[5]:-master} - - REPOREF_OVERRIDE=TOCI_REPOREF_$REPONAME - REPOREF=${!REPOREF_OVERRIDE:-$REPOREF} - - REPO_DIRECTORY=$TOCI_WORKING_DIR/$REPONAME - - if [ $REPOTYPE = git ] ; then - if [ ! -e $REPO_DIRECTORY ] ; then - git clone $REPOLOCATION $REPO_DIRECTORY - pushd $REPO_DIRECTORY - git reset --hard $REPOREF - popd - fi - else - echo "Unsupported repository type" - fi - else - echo "Couldn't parse '$line' as a source repository" - fi - done < $sr -done diff --git a/toci_setup.sh b/toci_setup.sh deleted file mode 100755 index 6d1cd12b2..000000000 --- a/toci_setup.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/usr/bin/env bash - -set -xe -. toci_functions.sh - -cd $TOCI_WORKING_DIR - -# Were going to cache images here -mkdir -p $TOCI_WORKING_DIR/image_cache - -# install deps on host machine, this script also restarts libvirt so we have to wait for it to be ready -install-dependencies -wait_for 3 3 ls /var/run/libvirt/libvirt-sock - -setup-network - -id | grep libvirt || ( echo "You have been added to the libvirt group, this script will now exit but will succeed if run again in a new shell" ; exit 1 ) - -if [ -f /etc/init.d/libvirt-bin ]; then - sudo service libvirt-bin restart -else - sudo service libvirtd restart -fi - -# set default arch for flavors in boot-stack -if [ "$TOCI_DIB_ARCH" != "i386" ]; then - sed -i "s/\"arch\":.*,/\"arch\": \"$TOCI_DIB_ARCH\",/" $TOCI_WORKING_DIR/tripleo-image-elements/elements/seed-stack-config/config.json -fi - -# custom power driver config -if [ -n "$TOCI_PM_DRIVER" ]; then - sed -i "s/\"power_manager\":.*,/\"power_manager\": \"$TOCI_PM_DRIVER\",/" $TOCI_WORKING_DIR/tripleo-image-elements/elements/seed-stack-config/config.json -fi - -sed -i "s/\"user\": \"stack\",/\"user\": \"`whoami`\",/" $TOCI_WORKING_DIR/tripleo-image-elements/elements/seed-stack-config/config.json - -# Create a deployment ramdisk + kernel -$TOCI_WORKING_DIR/diskimage-builder/bin/ramdisk-image-create -x -a $TOCI_DIB_ARCH $NODE_DIST deploy -o deploy-ramdisk - -# Make the tripleo image elements accessible to diskimage-builder -export ELEMENTS_PATH=$TOCI_WORKING_DIR/diskimage-builder/elements:$TOCI_WORKING_DIR/tripleo-image-elements/elements - -# Setup a seed vm -$TOCI_WORKING_DIR/tripleo-incubator/scripts/setup-seed-vm -a $TOCI_DIB_ARCH - -# Boot a seed vm -$TOCI_WORKING_DIR/tripleo-incubator/scripts/boot-seed-vm -a $TOCI_DIB_ARCH $NODE_DIST $TOCI_SEED_EXTRA_ELEMENTS neutron-dhcp-agent - -$TOCI_WORKING_DIR/diskimage-builder/bin/disk-image-create $NODE_DIST $TOCI_UNDERCLOUD_EXTRA_ELEMENTS -a $TOCI_DIB_ARCH -o $TOCI_WORKING_DIR/undercloud boot-stack nova-baremetal os-collect-config stackuser local-config neutron-dhcp-agent diff --git a/toci_test.sh b/toci_test.sh deleted file mode 100755 index 80180478a..000000000 --- a/toci_test.sh +++ /dev/null @@ -1,225 +0,0 @@ -#!/usr/bin/env bash - -set -xe -. toci_functions.sh - -cd $TOCI_WORKING_DIR -SEED_IP=`$TOCI_WORKING_DIR/tripleo-incubator/scripts/get-vm-ip seed` - -# Get logs from the node on exit -trap "get_state_from_host root $SEED_IP" EXIT - -# Add a route to the baremetal bridge via the seed node -sudo ip route del 192.0.2.0/24 dev virbr0 || true -sudo ip route add 192.0.2.0/24 dev virbr0 via $SEED_IP - -cp $TOCI_WORKING_DIR/tripleo-incubator/seedrc $TOCI_WORKING_DIR/seedrc -sed -i "s/\$SEED_IP/$SEED_IP/" $TOCI_WORKING_DIR/seedrc -source $TOCI_WORKING_DIR/seedrc - -export no_proxy=$no_proxy,$SEED_IP - -# wait for a successful os-refresh-config -if [[ "$NODE_DIST" =~ (.*)ubuntu(.*) ]]; then - wait_for 60 10 ssh_noprompt root@$SEED_IP grep 'Completed phase post-configure' /var/log/upstart/os-collect-config.log -else - wait_for 60 10 ssh_noprompt root@$SEED_IP journalctl -u os-collect-config \| grep \'Completed phase post-configure\' -fi - -# init keystone / setup endpoints -init-keystone -p unset unset 192.0.2.1 admin@example.com root@192.0.2.1 -setup-endpoints 192.0.2.1 --glance-password unset --heat-password unset --neutron-password unset --nova-password unset - -# Make sure nova has had a chance to start responding to requests -wait_for 10 5 nova list -user-config #Adds nova keypair - -if [ -n "$TOCI_MACS" ]; then - - # For the seed VM we use only the first MAC and power management setting - setup-baremetal $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH "${TOCI_MACS%% *}" seed "${TOCI_PM_IPS%% *}" "${TOCI_PM_USERS%% *}" "${TOCI_PM_PASSWORDS%% *}" - -else - - export SEED_MACS=$(create-nodes $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH 1) - setup-baremetal $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH "$SEED_MACS" seed - - # If MAC's weren't provided then we're using virtual nodes - OVERCLOUD_LIBVIRT_TYPE=${OVERCLOUD_LIBVIRT_TYPE:-";NovaComputeLibvirtType=qemu"} - -fi - -setup-neutron 192.0.2.2 192.0.2.3 192.0.2.0/24 192.0.2.1 192.0.2.1 ctlplane - -# Load images into glance -export DIB_PATH=$TOCI_WORKING_DIR/diskimage-builder -$TOCI_WORKING_DIR/tripleo-incubator/scripts/load-image undercloud.qcow2 - -keystone role-create --name heat_stack_user - -# place the bootstrap public key on host so that it can admin virt -ssh_noprompt root@$SEED_IP "cat /opt/stack/boot-stack/virtual-power-key.pub" >> ~/.ssh/authorized_keys - -# Now we have to wait for the bm poseur to appear on the compute node and for the compute node to then -# update the scheduler -if [[ "$NODE_DIST" =~ (.*)ubuntu(.*) ]]; then - wait_for 40 10 ssh_noprompt root@$SEED_IP grep 'Free VCPUS: [^0]' /var/log/upstart/nova-compute.log -else - wait_for 40 10 ssh_noprompt root@$SEED_IP journalctl -u nova-compute -u openstack-nova-compute \| grep \'Free VCPUS: [^0]\' -fi - -if [ -n "$TOCI_PM_DRIVER" ]; then - UNDERCLOUD_POWER_MANAGER=${UNDERCLOUD_POWER_MANAGER:-";PowerManager=${TOCI_PM_DRIVER}"} -fi - -make -C $TOCI_WORKING_DIR/tripleo-heat-templates undercloud-vm.yaml -heat stack-create -f $TOCI_WORKING_DIR/tripleo-heat-templates/undercloud-vm.yaml -P "PowerUserName=$(whoami);AdminToken=${TOCI_ADMIN_TOKEN};AdminPassword=${UNDERCLOUD_ADMIN_PASSWORD};GlancePassword=${UNDERCLOUD_ADMIN_PASSWORD};HeatPassword=${UNDERCLOUD_ADMIN_PASSWORD};NeutronPassword=${UNDERCLOUD_ADMIN_PASSWORD};NovaPassword=${UNDERCLOUD_ADMIN_PASSWORD};BaremetalArch=${TOCI_DIB_ARCH}${UNDERCLOUD_POWER_MANAGER}" undercloud - -# Just sleeping here so that we don't fill the logs with so many loops -sleep 180 - -heat list - -wait_for 40 20 heat list \| grep CREATE_COMPLETE - -# Delete the rule that prevent the Fedora bootstrap vm from forwarding -# packets. If the rule doesn't exist just do nothing... -ssh_noprompt root@$SEED_IP iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited || true -wait_for 20 15 ping -c 1 $(nova list | grep undercloud | sed -e "s/.*=\(.*\) .*/\1/g") - -export UNDERCLOUD_IP=$(nova list | grep ctlplane | sed -e "s/.*=\([0-9.]*\).*/\1/") -cp $TOCI_WORKING_DIR/tripleo-incubator/undercloudrc $TOCI_WORKING_DIR/undercloudrc -sed -i -e "s/\$UNDERCLOUD_IP/$UNDERCLOUD_IP/g" $TOCI_WORKING_DIR/undercloudrc -source $TOCI_WORKING_DIR/undercloudrc -export no_proxy=$no_proxy,$UNDERCLOUD_IP - -# Make the tripleo image elements accessible to diskimage-builder -export ELEMENTS_PATH=$TOCI_WORKING_DIR/diskimage-builder/elements:$TOCI_WORKING_DIR/tripleo-image-elements/elements - -if [ "$TOCI_DO_OVERCLOUD" = "1" ] ; then - if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "1" ] ; then - $TOCI_WORKING_DIR/diskimage-builder/bin/disk-image-create $NODE_DIST $TOCI_OVERCLOUD_EXTRA_ELEMENTS -a $TOCI_DIB_ARCH -o overcloud-all-in-one boot-stack cinder nova-compute nova-kvm neutron-openvswitch-agent os-collect-config stackuser neutron-network-node dhcp-all-interfaces swift-proxy swift-storage local-config - else - $TOCI_WORKING_DIR/diskimage-builder/bin/disk-image-create $NODE_DIST $TOCI_OVERCLOUD_EXTRA_ELEMENTS -a $TOCI_DIB_ARCH -o overcloud-control boot-stack cinder os-collect-config neutron-network-node dhcp-all-interfaces stackuser swift-proxy swift-storage local-config - fi -fi - -# Also get undercloud logs -trap "get_state_from_host root $SEED_IP ; get_state_from_host heat-admin $UNDERCLOUD_IP" EXIT - -# wait for a successful os-refresh-config -if [[ "$NODE_DIST" =~ (.*)ubuntu(.*) ]]; then - wait_for 60 10 ssh_noprompt heat-admin@$UNDERCLOUD_IP grep 'Completed phase post-configure' /var/log/upstart/os-collect-config.log -else - wait_for 60 10 ssh_noprompt heat-admin@$UNDERCLOUD_IP sudo journalctl -u os-collect-config \| grep \'Completed phase post-configure\' -fi - -# setup keystone endpoints -init-keystone -p $UNDERCLOUD_ADMIN_PASSWORD $TOCI_ADMIN_TOKEN $UNDERCLOUD_IP admin@example.com heat-admin@$UNDERCLOUD_IP -setup-endpoints $UNDERCLOUD_IP --glance-password $UNDERCLOUD_ADMIN_PASSWORD --heat-password $UNDERCLOUD_ADMIN_PASSWORD --neutron-password $UNDERCLOUD_ADMIN_PASSWORD --nova-password $UNDERCLOUD_ADMIN_PASSWORD - -# Make sure nova has had a chance to start responding to requests -wait_for 10 5 nova list - -if [ "$TOCI_DO_OVERCLOUD" != "1" ] ; then - exit 0 -fi - -user-config - -if [ -n "$TOCI_MACS" ]; then - - # For the undercloud we pop off the first MAC and power management settings - # since they have already been used by the seed VM - setup-baremetal $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH "${TOCI_MACS#[^ ]* }" undercloud "${TOCI_PM_IPS#[^ ]* }" "${TOCI_PM_USERS#[^ ]* }" "${TOCI_PM_PASSWORDS#[^ ]* }" - -else - - export UNDERCLOUD_MACS=$(create-nodes $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH 2) - setup-baremetal $TOCI_NODE_CPU $TOCI_NODE_MEM $TOCI_NODE_DISK $TOCI_DIB_ARCH "$UNDERCLOUD_MACS" undercloud - -fi - -setup-neutron 192.0.2.5 192.0.2.24 192.0.2.0/24 192.0.2.1 $UNDERCLOUD_IP ctlplane -ssh_noprompt heat-admin@$UNDERCLOUD_IP "cat /opt/stack/boot-stack/virtual-power-key.pub" >> ~/.ssh/authorized_keys - -if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "0" ] ; then - $TOCI_WORKING_DIR/diskimage-builder/bin/disk-image-create $NODE_DIST $TOCI_OVERCLOUD_EXTRA_ELEMENTS -a $TOCI_DIB_ARCH -o overcloud-compute nova-compute nova-kvm neutron-openvswitch-agent os-collect-config stackuser local-config -fi - -if [ -d /var/log/upstart ]; then - wait_for 40 10 ssh_noprompt heat-admin@$UNDERCLOUD_IP grep 'Free VCPUS: [^0]' /var/log/upstart/nova-compute.log -else - wait_for 40 10 ssh_noprompt heat-admin@$UNDERCLOUD_IP sudo journalctl -u nova-compute -u openstack-nova-compute \| grep \'Free VCPUS: [^0]\' -fi - -if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "1" ] ; then - load-image overcloud-all-in-one.qcow2 -else - load-image overcloud-control.qcow2 - load-image overcloud-compute.qcow2 -fi - - -if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "1" ] ; then - heat stack-create -f $TOCI_WORKING_DIR/tripleo-heat-templates/overcloud-all-in-one.yaml -P "AdminToken=${TOCI_ADMIN_TOKEN};AdminPassword=${OVERCLOUD_ADMIN_PASSWORD};CinderPassword=${OVERCLOUD_ADMIN_PASSWORD};GlancePassword=${OVERCLOUD_ADMIN_PASSWORD};HeatPassword=${OVERCLOUD_ADMIN_PASSWORD};NeutronPassword=${OVERCLOUD_ADMIN_PASSWORD};NovaPassword=${OVERCLOUD_ADMIN_PASSWORD};SwiftPassword=${OVERCLOUD_ADMIN_PASSWORD};Image=overcloud-all-in-one${OVERCLOUD_LIBVIRT_TYPE}" overcloud -else - make -C $TOCI_WORKING_DIR/tripleo-heat-templates overcloud.yaml - heat stack-create -f $TOCI_WORKING_DIR/tripleo-heat-templates/overcloud.yaml -P "AdminToken=${TOCI_ADMIN_TOKEN};AdminPassword=${OVERCLOUD_ADMIN_PASSWORD};CinderPassword=${OVERCLOUD_ADMIN_PASSWORD};GlancePassword=${OVERCLOUD_ADMIN_PASSWORD};HeatPassword=${OVERCLOUD_ADMIN_PASSWORD};NeutronPassword=${OVERCLOUD_ADMIN_PASSWORD};NovaPassword=${OVERCLOUD_ADMIN_PASSWORD};SwiftPassword=${OVERCLOUD_ADMIN_PASSWORD};notcomputeImage=overcloud-control${OVERCLOUD_LIBVIRT_TYPE}" overcloud -fi - -sleep 161 - -wait_for 50 20 heat list \| grep CREATE_COMPLETE - -if [ "$TOCI_OVERCLOUD_ALL_IN_ONE" = "1" ] ; then - export OVERCLOUD_IP=$(nova list | grep ctlplane | grep overcloud-allinone | sed -e "s/.*=\([0-9.]*\).*/\1/") -else - export OVERCLOUD_IP=$(nova list | grep ctlplane | grep notcompute | sed -e "s/.*=\([0-9.]*\).*/\1/") -fi - -cp $TOCI_WORKING_DIR/tripleo-incubator/overcloudrc $TOCI_WORKING_DIR/overcloudrc -sed -i -e "s/\$OVERCLOUD_IP/$OVERCLOUD_IP/g" $TOCI_WORKING_DIR/overcloudrc -source $TOCI_WORKING_DIR/overcloudrc -export no_proxy=$no_proxy,$OVERCLOUD_IP - -# Also get overcloud logs -trap "get_state_from_host root $SEED_IP ; get_state_from_host heat-admin $UNDERCLOUD_IP ; get_state_from_host heat-admin $OVERCLOUD_IP" EXIT - -# wait for a successful os-refresh-config -ssh_noprompt heat-admin@$UNDERCLOUD_IP sudo iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited || true - -if [[ "$NODE_DIST" =~ (.*)ubuntu(.*) ]]; then - wait_for 60 10 ssh_noprompt heat-admin@$OVERCLOUD_IP grep 'Completed phase post-configure' /var/log/upstart/os-collect-config.log -else - wait_for 60 10 ssh_noprompt heat-admin@$OVERCLOUD_IP sudo journalctl -u os-collect-config \| grep \'Completed phase post-configure\' -fi - - -# setup keystone endpoints -init-keystone -p $OVERCLOUD_ADMIN_PASSWORD $TOCI_ADMIN_TOKEN $OVERCLOUD_IP admin@example.com heat-admin@$OVERCLOUD_IP -setup-endpoints $OVERCLOUD_IP --glance-password $OVERCLOUD_ADMIN_PASSWORD --heat-password $OVERCLOUD_ADMIN_PASSWORD --neutron-password $OVERCLOUD_ADMIN_PASSWORD --nova-password $OVERCLOUD_ADMIN_PASSWORD --swift-password $OVERCLOUD_ADMIN_PASSWORD -keystone role-create --name heat_stack_user -user-config -setup-neutron "" "" 10.0.0.0/8 "" "" "" 192.0.2.45 192.0.2.64 192.0.2.0/24 - -# Make sure nova has had a chance to start responding to requests -wait_for 10 5 nova list - -# Lets add a cirros image to the overcloud -curl -L https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-i386-disk.img | glance image-create --name cirros --disk-format qcow2 --container-format bare --is-public 1 - -# create demo user -os-adduser -p $OVERCLOUD_DEMO_PASSWORD demo demo@example.com -source $TRIPLEO_ROOT/tripleo-incubator/overcloudrc-user -user-config - -# Start and test a image on the overcloud -nova boot --key-name default --flavor m1.tiny --image cirros --key_name default demo -sleep 20 # give the port a chance to appear -PORT=$(neutron port-list -f csv -c id --quote none | tail -n1) -neutron security-group-rule-create default --protocol tcp --direction ingress --port-range-min 22 --port-range-max 22 -neutron security-group-rule-create default --protocol icmp --direction ingress -IP=$(neutron floatingip-create ext-net --port-id "${PORT//[[:space:]]/}" | grep -o -P "192[0-9.]*") - -wait_for 10 5 ping -c 1 $IP diff --git a/updated_launch.sh b/updated_launch.sh deleted file mode 100755 index 9c5d001a8..000000000 --- a/updated_launch.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/env bash - -. toci_functions.sh - -LOCKFILE=/var/tmp/toci.lock -GITREF=${1:-origin/master} - -export TOCI_LOG_DIR=${TOCI_LOG_DIR:-$(mktemp -d --tmpdir toci_logs_XXXXXXX)} -RUNLOG=$TOCI_LOG_DIR/run.out - -cd $(dirname $0) -git fetch origin | tee -a $RUNLOG 2>&1 - -# Exit if there is a script already running, otherwise update repo -# TODO : fix small race condition here (probably not a problem) -flock -x -n $LOCKFILE git reset --hard $GITREF | tee -a $RUNLOG 2>&1 || exit 0 -flock -x -n $LOCKFILE ./toci.sh | tee -a $RUNLOG 2>&1 -