Add support for installation and testing

Moved files to a new directory to support the installation
of the molteniron package.  Renamed one file to be a
command line tool and added a new file to start/stop
the daemon.

Added tox support for testing.  This needed the ability
to start and stop the daemon.  And it also needed the
ability to delete the database entries to ensure a clean
test run.

Change-Id: I1b0fadd68b9598b715c639b962cf8586f4db5da8
This commit is contained in:
Mark Hamzy 2016-09-14 10:26:04 -05:00
parent 1535754268
commit dcd5fc4279
9 changed files with 473 additions and 121 deletions

35
README
View File

@ -1,35 +0,0 @@
MoltenIron maintains a pool of bare metal nodes.
==============================================================================
Before starting the server for the first time, the createDB.py
script must be run.
To start or restart the server, run moltenIronD.py.
==============================================================================
Use the molteniron client (molteniron.py) to communicate with the server. For
usage information type './molteniron.py -h'. For usage of a specific command
use ./molteniron.py [command] -h (ie - molteniron.py add -h)
==============================================================================
Configuration of MoltenIron is specified in the file conf.yaml.
"B)" means that this configuration option is required for both the client and
the server. "C)" means that it is required only for the client. "S)" means
it is only required for the server.
B) mi_port: - the port that the server uses to respond to
commands.
C) serverIP: - The IP address of the server. This is only used by
clients.
S) maxTime: - The maximum amount of time, in seconds, that a node
is allowed to be allocated to a particular BM node.
S) logdir: - The path to the directory where the logs should be
stored.
S) maxLogDays: - The amount of time, in days, to keep old logs.
S) sqlUser: - The username to use for the MI server. This user
will automatically be generated when createDB.py is run.
S) sqlPass: - The password of sqlUser

81
README.md Normal file
View File

@ -0,0 +1,81 @@
MoltenIron overview
===================
MoltenIron maintains a pool of bare metal nodes.
Starting
--------
Before starting the server for the first time, the createDB.py
script must be run.
To start the server:
```bash
moltenirond-helper start
```
To stop the server:
```bash
moltenirond-helper stop
```
MoltenIron client
-----------------
Use the molteniron client (molteniron) to communicate with the server. For
usage information type:
```bash
molteniron -h
```
For usage of a specific command use:
```bash
molteniron [command] -h
```
MoltenIron commands
-------------------
command | description
------- | -----------
add | Add a node
allocate | Allocate a node
release | Release a node
get_field | Get a specific field in a node
set_field | Set a specific field with a value in a node
status | Return the status of every node
delete_db | Delete every database entry
Configuration of MoltenIron
---------------------------
Configuration of MoltenIron is specified in the file conf.yaml.
"B)" means that this configuration option is required for both the client and
the server. "C)" means that it is required only for the client. "S)" means
it is only required for the server.
usage | key | description
----- | --- | -----------
B) | mi_port | the port that the server uses to respond to commands.
C) | serverIP | The IP address of the server. This is only used by
| | clients.
S) | maxTime | The maximum amount of time, in seconds, that a node
| | is allowed to be allocated to a particular BM node.
S) | logdir | The path to the directory where the logs should be
| | stored.
S) | maxLogDays | The amount of time, in days, to keep old logs.
S) | sqlUser | The username to use for the MI server. This user
| | will automatically be generated when createDB.py is run.
S) | sqlPass | The password of sqlUser
Running testcases
-----------------
The suite of testcases is checked by tox. But, before you can run tox, you
need to change the local yaml configuration file to point to the log
directory.
```bash
(LOG=$(pwd)/testenv/log; sed -i -r -e 's,^(logdir: )(.*)$,\1'${LOG}',' conf.yaml; rm -rf testenv/; tox -e testenv)
```

0
molteniron/__init__.py Normal file
View File

View File

@ -19,21 +19,44 @@ import argparse
import httplib
import json
import sys
import os
import yaml
import argparse
DEBUG = False
def split_commandline_args(argv):
front = []
back = []
command_found = False
for elm in argv:
if command_found:
back.append(elm)
else:
front.append(elm)
if elm[0] != '-':
command_found = True
return (front, back)
class MoltenIron(object):
def __init__(self):
self.conf = self.read_conf()
def __init__(self, conf, argv):
self.conf = conf
(argv, rest_argv) = split_commandline_args(argv)
# Parse the arguments and generate a request
parser = argparse.ArgumentParser()
parser.add_argument('command', help='Subcommand to run')
args = parser.parse_args(sys.argv[1:2])
request = getattr(self, args.command)()
(args, unknown_args) = parser.parse_known_args (argv)
unknown_args += rest_argv
request = getattr(self, args.command)(unknown_args)
# Send the request and print the response
self.response_str = self.send(request)
@ -57,7 +80,7 @@ class MoltenIron(object):
"""Returns the response from the server """
return self.response_json
def add(self):
def add(self, argv):
"""Generate a request to add a node to the MoltenIron database """
parser = argparse.ArgumentParser(
description='Add a node to the micli')
@ -80,12 +103,12 @@ class MoltenIron(object):
parser.add_argument('disk_gb', type=int, help="Amount of disk (in GiB)"
" that the node has")
args = parser.parse_args(sys.argv[2:])
args = parser.parse_args(argv)
request = vars(args)
request['method'] = 'add'
return request
def allocate(self):
def allocate(self, argv):
"""Generate request to checkout a node from the MoltenIron database """
parser = argparse.ArgumentParser(
description="Checkout a node in molteniron. Returns the node's"
@ -94,12 +117,12 @@ class MoltenIron(object):
parser.add_argument('number_of_nodes', type=int, help="How many nodes"
" to reserve")
args = parser.parse_args(sys.argv[2:])
args = parser.parse_args(argv)
request = vars(args)
request['method'] = 'allocate'
return request
def release(self):
def release(self, argv):
"""Generate a request to release an allocated node from the MoltenIron
database
"""
@ -108,13 +131,13 @@ class MoltenIron(object):
" returning it to the available state")
parser.add_argument('owner_name', help="Name of the owner who"
" currently owns the nodes to be released")
args = parser.parse_args(sys.argv[2:])
args = parser.parse_args(argv)
request = vars(args)
request['method'] = 'release'
return request
def get_field(self):
def get_field(self, argv):
"""Generate a request to return a field of data from an owned node from
the MoltenIron database
"""
@ -127,12 +150,12 @@ class MoltenIron(object):
parser.add_argument('field_name', help="Name of the field to retrieve"
" the value from")
args = parser.parse_args(sys.argv[2:])
args = parser.parse_args(argv)
request = vars(args)
request['method'] = 'get_field'
return request
def set_field(self):
def set_field(self, argv):
"""Generate request to set a field of data from an id in the MoltenIron
database
"""
@ -143,44 +166,72 @@ class MoltenIron(object):
parser.add_argument('key', help='Field name to set')
parser.add_argument('value', help='Field value to set')
args = parser.parse_args(sys.argv[2:])
args = parser.parse_args(argv)
request = vars(args)
request['method'] = 'set_field'
return request
def status(self):
def status(self, argv):
"""Return status """
parser = argparse.ArgumentParser(
description="Return a list of current MoltenIron Node database"
" entries")
args = parser.parse_args(sys.argv[2:])
args = parser.parse_args(argv)
request = vars(args)
request['method'] = 'status'
return request
def read_conf(self):
"""Read ./conf.yaml in """
path = sys.argv[0]
dirs = path.split("/")
newPath = "/".join(dirs[:-1]) + "/"
def delete_db(self, argv):
"""Delete all database entries"""
parser = argparse.ArgumentParser(
description="Delete every entry in the MoltenIron Node database")
fobj = open(newPath + "conf.yaml", "r")
conf = yaml.load(fobj)
return conf
args = parser.parse_args(argv)
request = vars(args)
request['method'] = 'delete_db'
return request
if __name__ == "__main__":
mi = MoltenIron()
parser = argparse.ArgumentParser(description="Molteniron command line tool")
parser.add_argument("-c",
"--conf-dir",
action="store",
type=str,
dest="conf_dir",
help="The directory where configuration is stored")
print(mi.get_response())
# We want individual command help later. So we need to remove a --help
# found at the end of the command line as long as there is at least
# one argument that doesn't start with '-'.
(argv, rest_argv) = split_commandline_args(list(sys.argv[1:]))
(args, unknown_args) = parser.parse_known_args (argv)
unknown_args += rest_argv
try:
rc = mi.get_response_map()['status']
except KeyError:
print("Error: Server returned: %s" % (mi.get_response_map(),))
rc = 444
if args.conf_dir:
if not os.path.isdir (args.conf_dir):
msg = "Error: %s is not a valid directory" % (args.conf_dir, )
print >> sys.stderr, msg
sys.exit(1)
if rc == 200:
exit(0)
yaml_file = os.path.realpath("%s/conf.yaml" % (args.conf_dir, ))
else:
exit(1)
yaml_file = "/usr/local/etc/molteniron/conf.yaml"
with open(yaml_file, "r") as fobj:
conf = yaml.load(fobj)
mi = MoltenIron(conf, unknown_args)
print(mi.get_response())
try:
rc = mi.get_response_map()['status']
except KeyError:
print("Error: Server returned: %s" % (mi.get_response_map(),))
rc = 444
if rc == 200:
exit(0)
else:
exit(1)

124
molteniron/moltenirond-helper Executable file
View File

@ -0,0 +1,124 @@
#!/usr/bin/env python
# Copyright (c) 2016 IBM Corporation.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import sys
import os
import signal
import yaml
from daemonize import Daemonize
from molteniron import moltenirond
PID = "/var/run/moltenirond.pid"
YAML_CONF = "/usr/local/etc/molteniron/conf.yaml"
def get_moltenirond_pid():
if not os.path.isfile(PID):
return -1
with open(PID) as fobj:
lines = fobj.readlines()
try:
pid = int(lines[0])
try:
# Send harmless kill signal in order to test existance
os.kill(pid, 0)
except Exception:
return -1
return pid
except Exception:
return -1
def moltenirond_main():
with open(YAML_CONF, "r") as fobj:
conf = yaml.load(fobj)
moltenirond.listener(conf)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Molteniron daemon helper")
parser.add_argument("-c",
"--conf-dir",
action="store",
type=str,
dest="conf_dir",
help="The directory where configuration is stored")
parser.add_argument("-p",
"--pid-dir",
action="store",
type=str,
dest="pid_dir",
help="The directory where PID information is stored")
parser.add_argument("-v",
"--verbose",
action="store",
type=bool,
dest="verbose",
help="Set a verbose information mode")
parser.add_argument("command", type=str, nargs=1, help="the command")
args = parser.parse_args ()
if args.conf_dir:
if not os.path.isdir (args.conf_dir):
msg = "Error: %s is not a valid directory" % (args.conf_dir, )
print >> sys.stderr, msg
sys.exit(1)
YAML_CONF = os.path.realpath("%s/conf.yaml" % (args.conf_dir, ))
if args.pid_dir:
if not os.path.isdir (args.pid_dir):
msg = "Error: %s is not a valid directory" % (args.pid_dir, )
print >> sys.stderr, msg
sys.exit(1)
PID = os.path.realpath("%s/moltenirond.pid" % (args.pid_dir, ))
if args.verbose:
print "YAML_CONF = %s" % (YAML_CONF, )
print "PID = %s" % (PID, )
if len(args.command) != 1:
msg = "Error: Expecting one command? Received: %s" % (args.command, )
print >> sys.stderr, msg
sys.exit (1)
if args.command[0].upper().lower() == "start":
pid = get_moltenirond_pid()
if pid > 0:
print >> sys.stderr, "Error: The daemon is already running"
sys.exit(1)
daemon = Daemonize(app="moltenirond",
pid=PID,
action=moltenirond_main)
daemon.start()
elif args.command[0].upper().lower() == "stop":
pid = get_moltenirond_pid()
if pid > 0:
os.remove (PID)
os.kill(pid, signal.SIGTERM)
else:
print >> sys.stderr, "Error: The daemon doesn't exist?"
sys.exit(1)
else:
msg = "Error: Unknown command: %s" % (args.command[0], )
print >> sys.stderr, msg
sys.exit (1)

View File

@ -24,6 +24,7 @@ import sys
import time
import traceback
import yaml
import argparse
from contextlib import contextmanager
@ -52,55 +53,79 @@ class JSON_encoder_with_DateTime(json.JSONEncoder):
return json.JSONEncoder.default(self, o)
class MoltenIronHandler(BaseHTTPRequestHandler):
# We need to turn BaseHTTPRequestHandler into a "new-style" class for
# Python 2.x
# NOTE: URL is over two lines :(
# http://stackoverflow.com/questions/1713038/super-fails-with-error-typeerror-
# argument-1-must-be-type-not-classobj
class OBaseHTTPRequestHandler(BaseHTTPRequestHandler, object):
pass
def do_POST(self):
self.data_string = self.rfile.read(int(self.headers['Content-Length']))
response = self.parse(self.data_string)
self.send_reply(response)
# We need to pass in conf into MoltenIronHandler, so make a class factory
# to do that
# NOTE: URL is over two lines :(
# http://stackoverflow.com/questions/21631799/how-can-i-pass-parameters-to-a-
# requesthandler
def MakeMoltenIronHandlerWithConf(conf):
class MoltenIronHandler(OBaseHTTPRequestHandler):
def __init__(self, *args, **kwargs):
# Note this *needs* to be done before call to super's class!
self.conf = conf
super(OBaseHTTPRequestHandler, self).__init__(*args, **kwargs)
def send_reply(self, response):
if DEBUG:
print("send_reply: response = %s" % (response,))
# get the status code off the response json and send it
status_code = response['status']
self.send_response(status_code)
self.send_header('Content-type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps(response, cls=JSON_encoder_with_DateTime))
def do_POST(self):
CL = 'Content-Length'
self.data_string = self.rfile.read(int(self.headers[CL]))
response = self.parse(self.data_string)
self.send_reply(response)
def parse(self, request_string):
"""Handle the request. Returns the response of the request """
try:
database = DataBase(conf)
# Try to json-ify the request_string
request = json.loads(request_string)
method = request.pop('method')
if method == 'add':
response = database.addBMNode(request)
elif method == 'allocate':
response = database.allocateBM(request['owner_name'],
request['number_of_nodes'])
elif method == 'release':
response = database.deallocateOwner(request['owner_name'])
elif method == 'get_field':
response = database.get_field(request['owner_name'],
request['field_name'])
elif method == 'set_field':
response = database.set_field(request['id'],
request['key'],
request['value'])
elif method == 'status':
response = database.status()
database.close()
del database
except Exception as e:
response = {'status': 400, 'message': str(e)}
def send_reply(self, response):
if DEBUG:
print("send_reply: response = %s" % (response,))
# get the status code off the response json and send it
status_code = response['status']
self.send_response(status_code)
self.send_header('Content-type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps(response,
cls=JSON_encoder_with_DateTime))
if DEBUG:
print("parse: response = %s" % (response,))
def parse(self, request_string):
"""Handle the request. Returns the response of the request """
try:
database = DataBase(self.conf)
# Try to json-ify the request_string
request = json.loads(request_string)
method = request.pop('method')
if method == 'add':
response = database.addBMNode(request)
elif method == 'allocate':
response = database.allocateBM(request['owner_name'],
request['number_of_nodes'])
elif method == 'release':
response = database.deallocateOwner(request['owner_name'])
elif method == 'get_field':
response = database.get_field(request['owner_name'],
request['field_name'])
elif method == 'set_field':
response = database.set_field(request['id'],
request['key'],
request['value'])
elif method == 'status':
response = database.status()
elif method == 'delete_db':
response = database.delete_db()
database.close()
del database
except Exception as e:
response = {'status': 400, 'message': str(e)}
return response
if DEBUG:
print("parse: response = %s" % (response,))
return response
return MoltenIronHandler
class Nodes(declarative_base()):
@ -362,6 +387,8 @@ class DataBase():
# Nodes.__table__.drop(self.engine, checkfirst=True)
metadata.drop_all(self.engine, checkfirst=True)
return {'status': 200}
def create_metadata(self):
# Instead of:
# Nodes.__table__.create(self.engine, checkfirst=True)
@ -975,9 +1002,9 @@ class DataBase():
def listener(conf):
mi_addr = str(conf['serverIP'])
mi_port = int(conf['mi_port'])
handler = MoltenIronHandler
handler_class = MakeMoltenIronHandlerWithConf(conf)
print('Listening... to %s:%d' % (mi_addr, mi_port,))
moltenirond = HTTPServer((mi_addr, mi_port), handler)
moltenirond = HTTPServer((mi_addr, mi_port), handler_class)
moltenirond.serve_forever()
@ -1052,11 +1079,27 @@ def cleanLogs(conf):
if __name__ == "__main__":
path = sys.argv[0]
dirs = path.split("/")
newPath = "/".join(dirs[:-1]) + "/"
parser = argparse.ArgumentParser(description="Molteniron daemon")
parser.add_argument("-c",
"--conf-dir",
action="store",
type=str,
dest="conf_dir",
help="The directory where configuration is stored")
fobj = open(newPath + "conf.yaml", "r")
conf = yaml.load(fobj)
args = parser.parse_args ()
listener(conf)
if args.conf_dir:
if not os.path.isdir (args.conf_dir):
msg = "Error: %s is not a valid directory" % (args.conf_dir, )
print >> sys.stderr, msg
sys.exit(1)
yaml_file = os.path.realpath("%s/conf.yaml" % (args.conf_dir, ))
else:
yaml_file = "/usr/local/etc/molteniron/conf.yaml"
with open(yaml_file, "r") as fobj:
conf = yaml.load(fobj)
listener(conf)

5
requirements.txt Normal file
View File

@ -0,0 +1,5 @@
daemonize
MySQL-python
pyyaml
sqlalchemy
sqlalchemy_utils

27
setup.py Executable file
View File

@ -0,0 +1,27 @@
#!/usr/bin/env python
# Copyright (c) 2016 IBM Corporation.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from distutils.core import setup
setup(name="molteniron",
version="1.0",
description="database for Ironic Baremetal services",
url="https://github.com/openstack/third-party-ci-tools",
py_modules=["molteniron/__init__", "molteniron/moltenirond"],
scripts=["molteniron/moltenirond-helper", "molteniron/molteniron"],
data_files=[("etc/molteniron/", ["conf.yaml"])]
)

56
tox.ini Normal file
View File

@ -0,0 +1,56 @@
[tox]
envlist = py27
[testenv:devenv]
envdir = devenv
basepython = python2.7
# usedevelop = True
# will create a devenv/lib/python2.7/site-packages/molteniron.egg-link which
# will point back to the git directory.
# Instead, we want the module installed in the virtual environment.
usedevelop = False
deps = -rrequirements.txt
[testenv:testenv]
envdir = testenv
basepython = python2.7
# usedevelop = True
# will create a testenv/lib/python2.7/site-packages/molteniron.egg-link which
# will point back to the git directory.
# Instead, we want the module installed in the virtual environment.
usedevelop = False
# Skip automatic tarballing of source distribution. We will manually run
# setup.py later...
skipsdist = True
# Don't worry about installing bash commands in the virtual environment.
whitelist_externals = mkdir
deps = -rrequirements.txt
commands = mkdir -p testenv/var/run/
python setup.py \
install \
--install-data=testenv/ \
--install-scripts=testenv/bin/ \
--install-purelib=testenv/lib/python2.7/site-packages/
moltenirond-helper \
--conf-dir=testenv/etc/molteniron/ \
--pid-dir=testenv/var/run/ \
start
molteniron \
--conf-dir=testenv/etc/molteniron/ \
delete_db
molteniron \
--conf-dir=testenv/etc/molteniron/ \
add test 10.1.2.1 user password 10.1.2.3,10.1.2.4 de:ad:be:ef:00:01 ppc64el 8 2048 32
molteniron \
--conf-dir=testenv/etc/molteniron/ \
allocate hamzy 1
molteniron \
--conf-dir=testenv/etc/molteniron/ \
get_field hamzy port_hwaddr
molteniron \
--conf-dir=testenv/etc/molteniron/ \
release hamzy
moltenirond-helper \
--conf-dir=testenv/etc/molteniron/ \
--pid-dir=testenv/var/run/ \
stop