Refactor redstack to use local.conf

The devstack project has moved away from a localrc file and now uses a
[[local|localrc]] section in the new local.conf file to accomplish the
same thing.  The redstack script should set up this file.
(see https://github.com/openstack-dev/devstack/commit/38e38fb )

The redstack script has been modified to set up devstack configuration
in the new format.

Backwards compatibility was maintained by deferring to an existing
localrc file in the devstack base directory.  In order to use the
new style configuration file, the localrc file must first be
deleted or renamed.  At the same time, any 'user' configuration
would need to move to USER_LOCAL_CONF (see below).

The refactor solves the important issue of when different configurations
are desired.  For example, in order to support profiling (OSProfiler),
entries have to be made in the corresponding trove configuration files.
The ability to turn profiling on and off is now possible, by using
configuration entries such as the following:

[[post-config|$TROVE_CONF]]
[profiler]
enabled = True

where the line 'enabled = True' could be generated from redstack in
the form of 'enabled = $ENABLE_PROFILER'

The refactoring also permits user specified configuration to become
more in-depth, allowing greater control over the trove/devstack
environment.  This was accomplished by including a USER_LOCAL_CONF
variable that defaults to '$HOME/.devstack.local.conf'.  All the
contents of that file are now included in devstack's local.conf
file on each invocation of 'redstack install'.

A directory 'local.conf.d' has been added to store additional
configuration files.  The existing using_vagrant and use_uuid_token
configurations have been moved there.  Stubs for the Trove
configuration files were added in 'trove.conf.rc'.  Files are loaded
by default, unless a corresponding variable is set to false.
For example, setting TROVE_CONF=false in options.rc will cause the
trove.conf.rc file to be skipped.  USING_VAGRANT=false and
USE_UUID_TOKEN=false have been added in redstack.rc to maintain
backwards compatibility.

A sample.rc file was created with detailed comments about how the
configuration can be set up (this file is ignored).

An additional file for persistent redstack options was added,
'$HOME/.redstack.options.rc', and is sourced before options.rc if it
exists.

A side effect of this refactor is that it regenerates the localrc
part each time, thus effectively fixing bug 1370946 as well.

Change-Id: I319476b29aada7f24829ae2308bf88c8c4bdafa7
Closes-Bug: #1421859
Closes-Bug: #1370946
This commit is contained in:
Peter Stachowski 2015-03-04 15:49:07 -05:00
parent f18e96715e
commit 0ac948a704
10 changed files with 300 additions and 49 deletions

2
.gitignore vendored
View File

@ -20,3 +20,5 @@ Vagrantfile
options.rc
.novaclient
*.log
.local.conf
scripts/local.conf.d/local.conf.d

View File

@ -252,3 +252,22 @@ function get_python_exec_prefix() {
echo "/usr/local/bin"
fi
}
# Returns 'true' if 'true', 'yes', 'on' or '1' - false, otherwise.
# Converts values to lower case first.
# If no default is provided, assumes false.
function get_bool() {
local VARIABLE="$1"
local DEFAULT=${2:-false}
VALUE=${!VARIABLE:-$DEFAULT}
VALUE=$(eval echo "$VALUE" | tr '[:upper:]' '[:lower:]')
if [[ "1 yes true on" =~ "$VALUE" ]]; then
VALUE=true
else
VALUE=false
fi
echo $VALUE
}

View File

@ -0,0 +1,42 @@
#
# Files in this directory are automatically added to the devstack
# local.conf file, between a specific set of tags.
#
# Filenames must end with '.rc' to be recognized; sample.rc is
# ignored.
#
# A '\' is required in front of any devstack variables since all
# .rc files are parsed first (using eval).
#
# Meta section headings must be included in each file, such as:
# [[local|localrc]]
# as the order of inserting the files is not guaranteed.
#
# All files are inherently included by default - to exclude a file,
# add a variable 'FILENAME_IN_UPPERCASE_MINUS_RC=false' in redstack.rc
# For Example: USING_VAGRANT=false (for the using_vagrant.rc file).
#
# Symbolic links are followed, so additional files can be loaded
# by placing them in an external directory and linking it in
# local.conf.d (this should allow complete flexibility in setting
# up testing options).
# For Example:
# cd /path/to/trove-integration/scripts/local.conf.d
# ln -s $HOME/local.conf.d local.conf.d
# cp /path/to/my_conf.rc $HOME/local.conf.d
[[local|localrc]]
# Put regular devstack variables under this meta section heading.
# This section is written out to a file and sourced by devstack,
# so it can contain logic as well.
# The following section types should only contain ini file style
# section headings and name=value pairs
[[post-config|\$TROVE_CONF]]
[[post-config|\$TROVE_TASKMANAGER_CONF]]
[[post-config|\$TROVE_CONDUCTOR_CONF]]
[[post-config|\$TROVE_API_PASTE_INI]]

View File

@ -0,0 +1,7 @@
[[post-config|\$TROVE_CONF]]
[[post-config|\$TROVE_TASKMANAGER_CONF]]
[[post-config|\$TROVE_CONDUCTOR_CONF]]
[[post-config|\$TROVE_API_PASTE_INI]]

View File

@ -0,0 +1,3 @@
[[local|localrc]]
KEYSTONE_TOKEN_FORMAT=UUID

View File

@ -1,3 +1,5 @@
[[local|localrc]]
# This is similar to code found at
# https://github.com/bcwaldon/vagrant_devstack/blob/master/Vagrantfile
# and seems to make instances ping'able in VirtualBox.

37
scripts/local.conf.rc Normal file
View File

@ -0,0 +1,37 @@
$TROVE_PRESENT_TAG
# Set some arguments for devstack.
#
# Note: This file contains autogenerated parts.
# All lines are removed from between the tag/end of tag
# markers (lines with '$MARKER_TOKEN' at beginning and end) and
# are replaced by trove-integration.
# Edits to these sections will not persist.
#
# See the '$USER_OPTS_TAG' section
# for ways to insert user args into this file.
#
#
# This section is for things that belong in localrc
# It comes from $DEFAULT_LOCALRC
#
[[local|localrc]]
$LOCALRC_OPTS_TAG
$LOCALRC_OPTS_TAG_END
#
# User options here were inserted from the file USER_LOCAL_CONF
# (defaults to $USERHOME/.$LOCAL_CONF)
#
$USER_OPTS_TAG
$USER_OPTS_TAG_END
#
# Additional options here were inserted by trove-integration
# automatically from files in $LOCAL_CONF_D
#
$ADD_OPTS_TAG
$ADD_OPTS_TAG_END

View File

@ -1,14 +1,14 @@
# Trove-integration
# Set some arguments for devstack.
# These passwords originally come from redstack.rc.
MYSQL_PASSWORD=$MYSQL_PASSWORD
RABBIT_PASSWORD=$RABBIT_PASSWORD
SERVICE_TOKEN=$SERVICE_TOKEN
ADMIN_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$SERVICE_PASSWORD
PUBLIC_INTERFACE=eth0
TROVE_LOGDIR=$TROVE_LOGDIR
TROVE_AUTH_CACHE_DIR=$TROVE_AUTH_CACHE_DIR
# Enable Trove, Swift, and Heat
ENABLED_SERVICES+=,trove,tr-api,tr-tmgr,tr-cond
ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
@ -21,28 +21,32 @@ fi
# Use Git repositories for client components
LIBS_FROM_GIT=python-troveclient
LIBS_FROM_GIT+=,python-ceilometerclient,python-cinderclient
LIBS_FROM_GIT+=,python-glanceclient,python-heatclient
LIBS_FROM_GIT+=,python-keystoneclient
if [[ $ENABLE_NEUTRON = true ]]; then
LIBS_FROM_GIT+=,python-neutronclient
if [[ $LIBS_FROM_GIT_ALL_CLIENTS = true ]]; then
LIBS_FROM_GIT+=,python-ceilometerclient,python-cinderclient
LIBS_FROM_GIT+=,python-glanceclient,python-heatclient
LIBS_FROM_GIT+=,python-keystoneclient
if [[ $ENABLE_NEUTRON = true ]]; then
LIBS_FROM_GIT+=,python-neutronclient
fi
LIBS_FROM_GIT+=,python-novaclient,python-openstackclient
LIBS_FROM_GIT+=,python-saharaclient,python-swiftclient
fi
LIBS_FROM_GIT+=,python-novaclient,python-openstackclient
LIBS_FROM_GIT+=,python-saharaclient,python-swiftclient
# Add Git repositories for libraries
LIBS_FROM_GIT+=,cliff,cookiecutter
LIBS_FROM_GIT+=,oslo.concurrency,oslo.context
LIBS_FROM_GIT+=,oslo.config,oslo-cookiecutter
LIBS_FROM_GIT+=,oslo.db,oslo.i18n
LIBS_FROM_GIT+=,oslo.log,oslo.messaging
LIBS_FROM_GIT+=,oslo.middleware,oslo.rootwrap
LIBS_FROM_GIT+=,oslo.serialization,oslosphinx
LIBS_FROM_GIT+=,oslotest,oslo.utils
LIBS_FROM_GIT+=,oslo.version,oslo.vmware
LIBS_FROM_GIT+=,pylockfile,hacking
LIBS_FROM_GIT+=,pbr,pyCADF
LIBS_FROM_GIT+=,stevedore,taskflow
LIBS_FROM_GIT+=,tooz
if [[ $LIBS_FROM_GIT_ALL_OSLO = true ]]; then
LIBS_FROM_GIT+=,cliff,cookiecutter
LIBS_FROM_GIT+=,oslo.concurrency,oslo.context
LIBS_FROM_GIT+=,oslo.config,oslo-cookiecutter
LIBS_FROM_GIT+=,oslo.db,oslo.i18n
LIBS_FROM_GIT+=,oslo.log,oslo.messaging
LIBS_FROM_GIT+=,oslo.middleware,oslo.rootwrap
LIBS_FROM_GIT+=,oslo.serialization,oslosphinx
LIBS_FROM_GIT+=,oslotest,oslo.utils
LIBS_FROM_GIT+=,oslo.version,oslo.vmware
LIBS_FROM_GIT+=,pylockfile,hacking
LIBS_FROM_GIT+=,pbr,pyCADF
LIBS_FROM_GIT+=,stevedore,taskflow
LIBS_FROM_GIT+=,tooz
fi
NOVNC_FROM_PACKAGE=false
SWIFT_HASH=$SWIFT_HASH

View File

@ -12,8 +12,17 @@
REDSTACK_SCRIPTS=${REDSTACK_SCRIPTS:-`pwd`}
REDSTACK_TESTS=$REDSTACK_SCRIPTS/../tests/
DEFAULT_LOCAL_CONF=local.conf.rc
DEFAULT_LOCALRC=localrc.rc
LOCAL_CONF=local.conf
LOCALRC=localrc
USER_LOCAL_CONF_NAME=.devstack.$LOCAL_CONF
USERHOME=$HOME
# Load options not checked into VCS.
if [ -f $USERHOME/.redstack.options.rc ]; then
. $USERHOME/.redstack.options.rc
fi
if [ -f $REDSTACK_SCRIPTS/options.rc ]; then
. $REDSTACK_SCRIPTS/options.rc
fi
@ -27,6 +36,10 @@ function get_default_host_ip() {
echo `LC_ALL=C ip -f inet addr show ${host_iface} | awk '/inet/ {split($2,parts,"/"); print parts[1]}' | head -1`
}
# Load functions devstack style
. $REDSTACK_SCRIPTS/functions
. $REDSTACK_SCRIPTS/functions_qemu
# Load global configuration variables.
. $REDSTACK_SCRIPTS/redstack.rc
. $REDSTACK_SCRIPTS/reviews.rc
@ -34,7 +47,7 @@ function get_default_host_ip() {
# allow overrides from devstack if already set
[[ -f $PATH_DEVSTACK_SRC/functions-common ]] && source $PATH_DEVSTACK_SRC/functions-common
[[ -f $PATH_DEVSTACK_SRC/functions ]] && source $PATH_DEVSTACK_SRC/functions
[[ -f $PATH_DEVSTACK_SRC/localrc ]] && source $PATH_DEVSTACK_SRC/localrc
[[ -f $PATH_DEVSTACK_SRC/$LOCALRC ]] && source $PATH_DEVSTACK_SRC/$LOCALRC
# Set up variables for the CONF files - this has to happen after loading redstack.rc, since
# TROVE_CONF_DIR is defined there - these will be used by devstack too
@ -71,10 +84,6 @@ TROVE_LOGDIR=${TROVE_LOGDIR:-$DEST/logs}
# filesystem that contains the datastore packages.
DATASTORE_PKG_LOCATION=${DATASTORE_PKG_LOCATION:-}
# Load functions devstack style
. $REDSTACK_SCRIPTS/functions
. $REDSTACK_SCRIPTS/functions_qemu
# Support entry points installation of console scripts
if [[ -d $PATH_TROVE/bin ]]; then
TROVE_BIN_DIR=$PATH_TROVE/bin
@ -229,36 +238,150 @@ function fixup_broken_devstack() {
:
}
# Delete all the lines from FILE_NAME between START_TAG and END_TAG
# Tags must appear at the beginning of a line
function clear_file_lines() {
local FILE_NAME=$1
local START_TAG=$2
local END_TAG=$3
sed -i "/^$START_TAG$/,/^$END_TAG$/{/^$START_TAG/!{/^$END_TAG/!d;}}" "$FILE_NAME"
}
# Checks to see if a variable with the same name as FILE_NAME exists.
# Returns 'true' if no varable exists or if the value of the variable
# is set to 'true' - returns the VAR_NAME to set otherwise.
# FILE_NAME is first converted to uppercase, the extension is removed
# and all remaining '.' and spaces are replaced with '_'
function check_filename_var() {
local FILE_NAME=$1
DEREF_VALUE=false
if [ -f "$FILE_NAME" ]; then
VAR_NAME=$(basename "$FILE_NAME" ".rc" | tr '[:lower:][:blank:][:punct:]' '[:upper:]__')
DEREF_VALUE=$(get_bool "$VAR_NAME" "true")
if [ "$DEREF_VALUE" != "true" ]; then
DEREF_VALUE=$VAR_NAME
fi
fi
echo "$DEREF_VALUE"
}
# Add the contents of one file to another, after the given tag
# Run through 'eval' if PARSE_FILE is true (defaults to true)
# Start with a blank line if BLANK_LINE_TO_START is true (defaults to false)
function add_file_contents() {
local FILE_NAME=$1
local FILE_TO_ADD=$2
local TAG=$3
local PARSE_FILE=${4:-true}
local BLANK_LINE_TO_START=${5:-false}
TEMP_FILE=".redstack.$$"
rm -f "$TEMP_FILE"
if [ "$BLANK_LINE_TO_START" = "true" ]; then
echo "" > "$TEMP_FILE"
fi
if [ -f "$FILE_TO_ADD" ]; then
echo "Adding $FILE_TO_ADD to $FILE_NAME"
echo "# Contents from $FILE_TO_ADD" >> "$TEMP_FILE"
if [ "$PARSE_FILE" = "true" ]; then
eval echo "\"$(cat "$FILE_TO_ADD")\"" >> "$TEMP_FILE"
else
cat "$FILE_TO_ADD" >> "$TEMP_FILE"
fi
echo "# End Of Contents from $FILE_TO_ADD" >> "$TEMP_FILE"
fi
echo "" >> "$TEMP_FILE"
sed -i "/^$TAG/r $TEMP_FILE" "$FILE_NAME"
rm -f "$TEMP_FILE"
}
function run_devstack() {
exclaim "Running devstack..."
# (Re)Creating this lock directory seems sure-fire.
rm -rf $USERHOME/nova_locks
mkdir -p $USERHOME/nova_locks
rm -rf "$USERHOME/nova_locks"
mkdir -p "$USERHOME/nova_locks"
pushd $PATH_DEVSTACK_SRC
if [ -f localrc ]; then
#Check if we have already configured localrc
is_in_localrc=`grep "# Trove-integration" localrc | wc -l`
if [ $is_in_localrc == 0 ]; then
#Otherwise append them to the existing file
eval echo "\"$(cat $REDSTACK_SCRIPTS/localrc.rc)\"" >> localrc
TROVE_PRESENT_TAG="# Trove-integration"
LOCAL_CONF_D=local.conf.d
CONF_MATCH="*.rc"
MARKER_TOKEN="#####"
USER_LOCAL_CONF=$(readlink -f "${USER_LOCAL_CONF:-$USERHOME/$USER_LOCAL_CONF_NAME}")
LOCALRC_OPTS_TAG="$MARKER_TOKEN Redstack Localrc Options $MARKER_TOKEN"
LOCALRC_OPTS_TAG_END="$MARKER_TOKEN End Of Redstack Localrc Options $MARKER_TOKEN"
USER_OPTS_TAG="$MARKER_TOKEN User Specified Options $MARKER_TOKEN"
USER_OPTS_TAG_END="$MARKER_TOKEN End Of User Specified Options $MARKER_TOKEN"
ADD_OPTS_TAG="$MARKER_TOKEN Additional Options $MARKER_TOKEN"
ADD_OPTS_TAG_END="$MARKER_TOKEN End Of Additional Options $MARKER_TOKEN"
pushd "$PATH_DEVSTACK_SRC"
DEVSTACK_LOCAL_CONF=$LOCAL_CONF
# remain backwards compatible with existing localrc files
if [ -f "$LOCALRC" ]; then
DEVSTACK_LOCAL_CONF=$LOCALRC
echo "Old-style devstack config file $PATH_DEVSTACK_SRC/$DEVSTACK_LOCAL_CONF found."
echo "Consider removing to generate the preferred-sytle config file $LOCAL_CONF."
fi
if [ -f "$DEVSTACK_LOCAL_CONF" ]; then
# Check if we have already configured the devstack config file
already_in_conf=$(grep "$TROVE_PRESENT_TAG" "$DEVSTACK_LOCAL_CONF" | wc -l)
if [ "$already_in_conf" == 0 ]; then
# We can no longer append to an existing old-style localrc file
if [ "$DEVSTACK_LOCAL_CONF" == "$LOCALRC" ]; then
echo "The devstack config file $PATH_DEVSTACK_SRC/$DEVSTACK_LOCAL_CONF is too old to append to."
echo "Please remove and try again."
exit 1
fi
# Otherwise append the redstack version to the existing file
eval echo "\"$(cat "$REDSTACK_SCRIPTS/$DEFAULT_LOCALRC")\"" >> "$DEVSTACK_LOCAL_CONF"
fi
else
# If a devstack config file doesn't exist, create it
eval echo "\"$(cat "$REDSTACK_SCRIPTS/$DEFAULT_LOCAL_CONF")\"" > "$DEVSTACK_LOCAL_CONF"
fi
# We can only replace sections from the LOCAL_CONF style files
if [ "$DEVSTACK_LOCAL_CONF" == "$LOCAL_CONF" ]; then
# Clear out all the options
clear_file_lines "$DEVSTACK_LOCAL_CONF" "$LOCALRC_OPTS_TAG" "$LOCALRC_OPTS_TAG_END"
clear_file_lines "$DEVSTACK_LOCAL_CONF" "$USER_OPTS_TAG" "$USER_OPTS_TAG_END"
clear_file_lines "$DEVSTACK_LOCAL_CONF" "$ADD_OPTS_TAG" "$ADD_OPTS_TAG_END"
# Add the main localrc file
PARSE_FILE="true"
BLANK_LINE_TO_START="true"
if [ -f "$REDSTACK_SCRIPTS/$DEFAULT_LOCALRC" ]; then
add_file_contents "$DEVSTACK_LOCAL_CONF" "$REDSTACK_SCRIPTS/$DEFAULT_LOCALRC" "$LOCALRC_OPTS_TAG" "$PARSE_FILE" "$BLANK_LINE_TO_START"
fi
else
#If localrc doesn't exist, create it
eval echo "\"$(cat $REDSTACK_SCRIPTS/localrc.rc)\"" > localrc
fi
# Add any user options
PARSE_FILE="false"
BLANK_LINE_TO_START="true"
if [ -f "$USER_LOCAL_CONF" ]; then
add_file_contents "$DEVSTACK_LOCAL_CONF" "$USER_LOCAL_CONF" "$USER_OPTS_TAG" "$PARSE_FILE" "$BLANK_LINE_TO_START"
fi
if [ -n "$USE_UUID_TOKEN" ]; then
echo "KEYSTONE_TOKEN_FORMAT=UUID" >> localrc
# Add all the files in the LOCAL_CONF_D directory that match CONF_MATCH (except for sample files)
# and that aren't excluded. Files are excluded by having a variable
# 'FILENAME_IN_UPPERCASE_MINUS_RC=false' in redstack.rc
# For Example: USING_VAGRANT=false (for the using_vagrant.rc file).
PARSE_FILE="true"
BLANK_LINE_TO_START="false"
while IFS= read -r -d '' CONF_FILE
do
FILE_NAME_VAR=$(check_filename_var "$CONF_FILE")
if [ "$FILE_NAME_VAR" = "true" ]; then
add_file_contents "$DEVSTACK_LOCAL_CONF" "$CONF_FILE" "$ADD_OPTS_TAG" "$PARSE_FILE" "$BLANK_LINE_TO_START"
else
echo "Skipping $CONF_FILE"
echo "Use $FILE_NAME_VAR=true to include"
fi
done < <(find "$REDSTACK_SCRIPTS/${LOCAL_CONF_D}" -name "${CONF_MATCH}" -follow -not -name "sample*.rc" -type f -print0)
# this is to add a blank line for readability
add_file_contents "$DEVSTACK_LOCAL_CONF" "" "$ADD_OPTS_TAG"
fi
if [ -n "$USING_VAGRANT" ]
then
eval echo "\"$(cat $REDSTACK_SCRIPTS/vagrant.rc)\"" >> localrc
fi
export MYSQL_HOST=$SERVICE_HOST
./stack.sh
popd
@ -898,6 +1021,9 @@ function print_usage() {
Commands :
--setup environment--
install - Install all the required dependencies and bring up tr-api and tr-tmgr
- devstack config can be altered by using a USER_LOCAL_CONF file
which will be copied into devstack/local.conf on each 'install' run
(defaults to \$HOME/$USER_LOCAL_CONF_NAME)
test-init - Configure the test configuration files and add keystone test users
build-image - Builds the vm image for the trove guest
initialize - Reinitialize the trove database, users, services, and test config
@ -975,6 +1101,7 @@ function run_command() {
"wipe-queues" ) shift; cmd_wipe_queues $@;;
"example-tests" ) shift; cmd_example_tests $@;;
"repl" ) shift; cmd_repl $@;;
"help" ) print_usage;;
* )
echo "'$1' not a valid command"
exit 1

View File

@ -6,7 +6,15 @@
[ -z $TROVE_CONF_DIR ] && TROVE_CONF_DIR=/etc/trove
# Enable neutron instead of nova-network
[ -z $ENABLE_NEUTRON ] && ENABLE_NEUTRON=false
ENABLE_NEUTRON=$(get_bool ENABLE_NEUTRON false)
# Enable LIBS_FROM_GIT
LIBS_FROM_GIT_ALL_CLIENTS=$(get_bool LIBS_FROM_GIT_ALL_CLIENTS true)
LIBS_FROM_GIT_ALL_OSLO=$(get_bool LIBS_FROM_GIT_ALL_OSLO true)
# Don't include certain .rc files in local.conf.d by default
USING_VAGRANT=$(get_bool USING_VAGRANT false)
USE_UUID_TOKEN=$(get_bool USE_UUID_TOKEN false)
# Paths for various OpenStack components
PATH_DEVSTACK_OUTPUT=$DEST