diff --git a/README.md b/README.md index e91e1b4..47cb447 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,6 @@ -# Picasso: Functions-as-a-Service (FaaS) on OpenStack +# Picasso + +Functions-as-a-Service (FaaS) on OpenStack ## Mission @@ -37,13 +39,13 @@ then the benefits are different, but related. * Scaling is simply adding more IronFunctions nodes -### System requirements +## System requirements * Operating system: Linux/MacOS * Python version: 3.5 or greater * Database: MySQL 5.7 or greater -### Quick-start guide +## Quick-start guide * Install DevStack with [IronFunctions enabled](https://github.com/iron-io/picasso/blob/master/devstack/README.rst) * Clone the [Picasso source](https://github.com/iron-io/picasso) @@ -53,7 +55,7 @@ Create a Python3.5 virtualenv $ virtualenv -p python3.5 .venv $ source .venv/bin/activate -Install dependencies +Install Picasso dependencies $ pip install -r requirements.txt -r test-requirements.txt @@ -72,7 +74,7 @@ set the following environment variable: export PICASSO_MIGRATIONS_DB=mysql+pymysql://root:root@localhost/functions -Use `alembic` to apply the migrations: +Then use `alembic` to apply the migrations $ alembic upgrade head @@ -103,22 +105,21 @@ The following are the minimum required options to start the Picasso API service: ### Building and Running Picasso in Docker -From the Picasso repo, build a Docker image: +Install [Docker engine](https://docs.docker.com/engine/installation/) if you haven't already. + +From the Picasso repo, build a Docker image using the following commands: export DOCKER_HOST=tcp://: docker build -t picasso-api -f Dockerfile . -To start the container, pass in the required env vars, by +To start the container, pass in the required env vars with `--env-file` or by entering all required +values in `-e =` format to the `docker run` command. -`--env-file` example [Dockerfile.env](Dockerfile.env.example) +Example [Dockerfile.env](Dockerfile.env.example) - docker run -d -p 10001:10001 --env-file Dockerfile.env picasso-api + docker run -d -p 10001:10001 --env-file Dockerfile.env picasso-api -or by entering all values in `-e =` format. - -Once the container is started, check if the service in running: - -In your web browser navigate to: +Once the container is started, check if the service in running. In your web browser navigate to: :10001/api @@ -126,32 +127,16 @@ or using the CLI: curl -X GET http://:10001/api/swagger.json | python -mjson.tool -### Examining the API - -In [examples](examples/) folder you can find a script that examines available API endpoints. - -Note that this script depends on the following env vars: - -* `PICASSO_API_URL` - Picasso API endpoint -* `OS_AUTH_URL` - OpenStack Auth URL -* `OS_PROJECT_ID` - it can be found in OpenStack Dashboard or in CLI -* `OS_USERNAME` - OpenStack project-aligned username -* `OS_PASSWORD` - OpenStack project-aligned user password -* `OS_DOMAIN` - OpenStack project domain name -* `OS_PROJECT_NAME` - OpenStack project name - -To run the script: - - OS_AUTH_URL=http://192.168.0.112:5000/v3 OS_PROJECT_ID=8fb76785313a4500ac5367eb44a31677 OS_USERNAME=admin OS_PASSWORD=root OS_DOMAIN=default OS_PROJECT_NAME=admin ./examples/hello-lambda.sh - -Please note that values provided are project-specific, so they can't be reused. - ### API docs API docs are discoverable via Swagger. Just launch the Picasso API and browse to: http://:/api +### Testing Picasso + +See [Testing.md](TESTING.md) + ### Support -Join us on [Slack](https://open-iron.herokuapp.com/)! \ No newline at end of file +Join us on [Slack](https://open-iron.slack.com/)! diff --git a/TESTING.md b/TESTING.md new file mode 100644 index 0000000..bbbf5bb --- /dev/null +++ b/TESTING.md @@ -0,0 +1,93 @@ +# Testing + +## Requirements + +* Install `Tox` + + $ pip install tox + +* MySQL instance with database migrations applied (refer to quick start guide) + + $ export TEST_DB_URI=mysql://:@:/ + +### PEP8 style checks + + $ tox -e pep8 + + +### Functional testing + + $ tox -e py35-functional + +Pros: + +* lightweight (controllers and DB models testing) +* no OpenStack required +* no IronFunctions required + +Cons: + +* MySQL server required +* OpenStack authentication is not tested +* IronFunctions API stubbed with fake implementation + +### Integration tests + +The following env variables are required: + +* TEST_DB_URI - similar to functional tests, database endpoint +* FUNCTIONS_API_URL - IronFunctions API URL (default value - `http://localhost:8080/v1`) +* OS_AUTH_URL - OpenStack Identity endpoint +* OS_PROJECT_NAME - OpenStack user-specific project name +* OS_USERNAME - OpenStack user name +* OS_PASSWORD - OpenStack user user password + +```bash +export TEST_DB_URI=mysql://:@:/ +export FUNCTIONS_API_URL=://:/ +export OS_AUTH_URL=://:/ +export OS_PROJECT_NAME= +export OS_USERNAME= +export OS_PASSWORD= +tox -epy35-integration +``` + +### Testing: Docker-build + +The following operations are performed: + +* builds an image +* deletes all artifacts (Python3.5 image and recently built image) + +```bash +export DOCKER_HOST=tcp://:> +export TEST_DB_URI=mysql://:@:/ +export FUNCTIONS_API_URL=://:/ +export OS_AUTH_URL=://:/ +tox -e docker-build +``` + +### Testing Docker-full + +The following operations are performed: + +* build container from source code +* run container with exposed ports +* request Swagger API doc to see if API is responsive +* destroy running container + +```bash +export DOCKER_HOST=tcp://:> +export TEST_DB_URI=mysql://:@:/ +export FUNCTIONS_API_URL=://:/ +export OS_AUTH_URL=://:/ +tox -e docker-full +``` + +### Coverage regression testing + + $ tox -e py35-functional-regression + +### Static code analysis with Bandit + + $ tox -e bandit diff --git a/devstack/README.rst b/devstack/README.md similarity index 78% rename from devstack/README.rst rename to devstack/README.md index b0e7372..8d98457 100644 --- a/devstack/README.rst +++ b/devstack/README.md @@ -1,28 +1,21 @@ -Enabling Picasso (Functions-as-a-Service) in DevStack -===================================================== +# Enabling Picasso (Functions-as-a-Service) in DevStack -Installing Glide -================ +## Install Glide -Note that your machine should have Glide installed. +It is required to install Glide on the system in which you plan to run DevStack on, as +the Functions service is written in Go and we must fetch dependencies during install. See more info at https://github.com/Masterminds/glide -Download DevStack -================= - -.. sourcecode:: bash +## Download DevStack export DEVSTACK_DIR=~/devstack git clone git://git.openstack.org/openstack-dev/devstack.git $DEVSTACK_DIR -Enable the FaaS plugin -====================== +## Enable the Functions plugin Enable the plugin by adding the following section to ``$DEVSTACK_DIR/local.conf`` -.. sourcecode:: bash - [[local|localrc]] enable_plugin picasso git@github.com:iron-io/picasso.git @@ -40,7 +33,7 @@ Enable the plugin by adding the following section to ``$DEVSTACK_DIR/local.conf` PICASSO_CLIENT_DIR=${PICASSO_CLIENT_DIR:-${DEST}/python-picassoclient} PICASSO_CLIENT_BRANCH=${PICASSO_CLIENT_BRANCH:-master} - # IronFunctions parameters + # Functions parameters FUNCTIONS_REPO=${FUNCTIONS_REPO:-git@github.com:iron-io/functions.git} FUNCTIONS_BRANCH=${FUNCTIONS_BRANCH:-master} FUNCTIONS_PORT=${FUNCTIONS_PORT:-10501} @@ -50,10 +43,7 @@ Enable the plugin by adding the following section to ``$DEVSTACK_DIR/local.conf` DOCKERD_OPTS=${DOCKERD_OPTS:---dns 8.8.8.8 --dns 8.8.4.4 --storage-driver=overlay2 -H fd://} -Run the DevStack utility -======================== - -.. sourcecode:: bash +## Run the DevStack utility cd $DEVSTACK_DIR ./stack.sh diff --git a/examples/hello-lambda.sh b/examples/hello-lambda.sh deleted file mode 100755 index 88ee696..0000000 --- a/examples/hello-lambda.sh +++ /dev/null @@ -1,96 +0,0 @@ -#!/usr/bin/env bash - -set +x -set +e - -export LAOS_API_URL=${LAOS_API_URL:-http://localhost:10001} - -export OS_AUTH_URL=${OS_AUTH_URL:-http://localhost:5000/v3} -export OS_USERNAME=${OS_USERNAME:-admin} -export OS_PASSOWRD=${OS_PASSWORD:-root} -export OS_DOMAIN=${OS_DOMAIN:-default} -export OS_PROJECT_ID=${OS_PROJECT_ID:-"dummy_project_id"} - - -rm -fr examples/token_request.json -echo -e "{ - \"auth\": { - \"identity\": { - \"methods\": [\"password\"], - \"password\": { - \"user\": { - \"name\": \"${OS_USERNAME:-admin}\", - \"domain\": { \"id\": \"${OS_DOMAIN:-default}\" }, - \"password\": \"${OS_PASSWORD:-root}\" - } - } - }, - \"scope\": { - \"project\": { - \"name\": \"${OS_PROJECT_NAME:-admin}\", - \"domain\": {\"id\": \"${OS_DOMAIN:-default}\" } - } - } - } -}" >> examples/token_request.json - - -export OS_TOKEN=`curl -si -d @examples/token_request.json -H "Content-type: application/json" ${OS_AUTH_URL}/auth/tokens | awk '/X-Subject-Token/ {print $2}'` - -echo -e "Listing apps\n" -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Creating app\n" -curl -X POST -d '{"app":{"name": "testapp"}}' ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Listing apps\n" -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Showing app info\n" -export raw_app_info=`curl localhost:10001/v1/${OS_PROJECT_ID}/apps -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool | grep name | awk '{print $2}'` -export app_name=${raw_app_info:1:30} -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name} -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Listing app routes\n" -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Creating app sync private route\n" -curl -X POST -d '{"route":{"type": "sync", "path": "/hello-sync-private", "image": "iron/hello", "is_public": "false" }}' ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${APP_NAME}/routes -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Creating app sync public route\n" -curl -X POST -d '{"route":{"type": "sync", "path": "/hello-sync-public", "image": "iron/hello", "is_public": "true" }}' ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${APP_NAME}/routes -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Listing app routes\n" -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Show app private route\n" -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes/hello-sync-private -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Show app public route\n" -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes/hello-sync-public -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Running app sync private route\n" -curl -X POST -d '{"name": "Johnny"}' ${LAOS_API_URL}/v1/r/${OS_PROJECT_ID}/${app_name}/hello-sync-private -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Running app sync public route\n" -curl -X POST -d '{"name": "Johnny"}' ${LAOS_API_URL}/r/${app_name}/hello-sync-public -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Creating app async route\n" -curl -X POST -d '{"route":{"type": "async", "path": "/hello-async-private", "image": "iron/hello", "is_public": "false"}}' ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Running app async route\n" -curl -X POST -d '{"name": "Johnny"}' ${LAOS_API_URL}/v1/r/${OS_PROJECT_ID}/${app_name}/hello-async-private -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Deleting app route\n" -curl -X DELETE ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes/hello-sync-public -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool -curl -X DELETE ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes/hello-sync-private -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool -curl -X DELETE ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes/hello-async-private -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Listing app routes\n" -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name}/routes -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Deleting app\n" -curl -X DELETE ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps/${app_name} -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool - -echo -e "Listing apps\n" -curl ${LAOS_API_URL}/v1/${OS_PROJECT_ID}/apps -H "X-Auth-Token:${OS_TOKEN}" -H "Content-Type: application/json" | python3 -mjson.tool diff --git a/testing.md b/testing.md deleted file mode 100644 index 4b2c346..0000000 --- a/testing.md +++ /dev/null @@ -1,112 +0,0 @@ -Testing -------- - -In order to run tests you need to install `Tox`: - - $ pip install tox - -Also, you will need a running MySQL instance with the database migrations applied from the previous step. -Tests are dependent on pre-created MySQL database for persistence. -Please set env var - - $ export TEST_DB_URI=mysql://:@:/ - -PEP8 style checks ------------------ - -In order to run `PEP8` style checks run following command: - - $ tox -e pep8 - - -Functional testing ------------------- - -In order to run `functional` tests run following command: - - $ tox -e py35-functional - -Pros: - -* lightweight (controllers and DB models testing) -* no OpenStack required -* no IronFunctions required - -Cons: - -* MySQL server required -* OpenStack authentication is not tested -* IronFunctions API stubbed with fake implementation - -Integration integrations ------------------------- - -Integration tests are dependent on following env variables: - -* TEST_DB_URI - similar to functional tests, database endpoint -* FUNCTIONS_API_URL - IronFunctions API URL (default value - `http://localhost:8080/v1`) -* OS_AUTH_URL - OpenStack Identity endpoint -* OS_PROJECT_NAME - OpenStack user-specific project name -* OS_USERNAME - OpenStack user name -* OS_PASSWORD - OpenStack user user password - -To run tests use following command: - - export TEST_DB_URI=mysql://:@:/ - export FUNCTIONS_API_URL=://:/ - export OS_AUTH_URL=://:/ - export OS_PROJECT_NAME= - export OS_USERNAME= - export OS_PASSWORD= - tox -epy35-integration - -Testing: Docker-build ---------------------- - -This type of testing allows to ensure if code can be build inside docker container with no problems. -In order to run this check use following commands:: - - export DOCKER_HOST=tcp://:> - export TEST_DB_URI=mysql://:@:/ - export FUNCTIONS_API_URL=://:/ - export OS_AUTH_URL=://:/ - tox -e docker-build - -During this check Tox: - -* builds an image -* deletes all artifacts (Python3.5 image and recently built image) - -Testing Docker-full -------------------- - -This type of testing allows to ensure if code code can be build and run successfully inside docker container with no problems. -In order to run this check use following commands:: - - export DOCKER_HOST=tcp://:> - export TEST_DB_URI=mysql://:@:/ - export FUNCTIONS_API_URL=://:/ - export OS_AUTH_URL=://:/ - tox -e docker-full - -During this check following operations are performed:: - -* build container from source code -* run container with exposed ports -* request Swagger API doc to see if API is responsive -* tear-down running container - - -Coverage regression testing ---------------------------- - -In order to build quality software it is necessary to keep test coverage at its highest point. -So, as part of `Tox` testing new check was added - functional test coverage regression. -In order to run it use following command: - - $ tox -e py35-functional-regression - -Static code analysis with Bandit -================================ - - $ tox -e bandit