Support storage of introspection data in Nginx

As an operator I want to be able to persist raw and processed
introspection data so that I am able to view it at a later date.

As an operator I want to be able to persist raw and processed
introspection data so that I am able to reprocess the data after the
initial inspection process has completed.

In the absence of swift, we can use the bifrost nginx web server -
masquerading as an object store - to store raw and processed
introspection data for nodes.

This allows introspection data to be retrieved and reprocessed after the
initial inspection has completed. This can be useful when the processing
pipeline or introspection rules are changed.

Change-Id: Ia2bd16080594e854054f380d4f7670eaea98e82b
Closes-Bug: #1685879
This commit is contained in:
Mark Goddard 2017-04-24 19:42:03 +01:00
parent 8c52981580
commit 507228a228
13 changed files with 289 additions and 0 deletions

View File

@ -228,6 +228,14 @@ inspector_processing_hooks: String value containing a comma-separated list,
non-default list of comma-separated processing
hooks for inspector.
inspector_store_data_in_nginx: Boolean value, default true. If true, this
enables data gathered during introspection to be
stored in the local Nginx web server. In this
mode, Nginx masquerades as an unauthenticated
'Swift' object storage service. Nginx is
configured to only allow the required operations
on the 'ironic-inspector' object container.
### Virtual Environment Install
Bifrost can install ironic into a python virtual environment using the

View File

@ -165,6 +165,14 @@ inspector_keep_ports: "present"
# list of processing hooks for inspector.
#inspector_processing_hooks:
# Whether to store introspection data using the local Nginx web server as an
# object storage service.
inspector_store_data_in_nginx: true
# When inspector_store_data_in_nginx is true, this is the URL of the Nginx
# 'Swift' API endpoint.
inspector_store_data_url: "http://localhost:{{ file_url_port }}"
# Inspector defaults
inspector:
discovery:

View File

@ -294,6 +294,15 @@
testing | bool == true
- name: "Deploy nginx configuration file for serving HTTP requests"
template: src=nginx.conf.j2 dest=/etc/nginx/nginx.conf
- name: "Ensure inspector object storage directory exists"
file:
path: "{{ http_boot_folder }}/ironic-inspector"
state: directory
owner: "{{ nginx_user }}"
group: "{{ nginx_user }}"
when:
- enable_inspector | bool
- inspector_store_data_in_nginx | bool
- name: "Download Ironic Python Agent kernel & image"
include: download_ipa_image.yml
when: create_ipa_image | bool == false and download_ipa | bool == true
@ -329,6 +338,15 @@
setype: httpd_sys_content_t
state: present
- name: "Add proper context on inspector data store"
sefcontext:
target: "{{ http_boot_folder }}/ironic-inspector(/.*)?"
setype: httpd_sys_rw_content_t
state: present
when:
- enable_inspector | bool
- inspector_store_data_in_nginx | bool
- name: Copy ironic policy file to temporary directory
copy:
src: ironic_policy.te

View File

@ -63,9 +63,19 @@ always_store_ramdisk_logs = {{ inspector_store_ramdisk_logs | default('true') |
{% if inspector_processing_hooks is defined %}
processing_hooks = {{ inspector_processing_hooks }}
{% endif %}
{% if inspector_store_data_in_nginx | bool %}
store_data = swift
{% endif %}
{% if inspector.discovery.enabled == true %}
node_not_found_hook = enroll
[discovery]
enroll_node_driver = {{ inspector.discovery.default_node_driver }}
{% endif %}
{% if inspector_store_data_in_nginx | bool %}
[swift]
# Use the local nginx web server as a Swift-list object storage service.
auth_type = none
endpoint = {{ inspector_store_data_url }}
{% endif %}

View File

@ -46,6 +46,14 @@ http {
location {{ http_boot_folder }}/ {
alias {{ http_boot_folder }}/;
}
{% if inspector_store_data_in_nginx | bool %}
location /ironic-inspector {
return 200 "";
}
location /ironic-inspector/ {
dav_methods PUT DELETE;
}
{% endif %}
}
include /etc/nginx/conf.d/bifrost*.conf;
}

View File

@ -0,0 +1,53 @@
bifrost-test-inspection
=======================
Tests nodes that have been inspected by ironic inspector.
Requirements
------------
None at this time. See Dependencies.
Role Variables
--------------
None at this time. See Dependencies.
Dependencies
------------
This role is intended to be executed as part of bifrost, after the
ironic-inspect-node role, as part of the test sequence.
Example Playbook
----------------
hosts: baremetal
name: "Tests inspection of baremetal nodes"
connection: local
become: no
gather_facts: no
roles:
- role: bifrost-test-inspection
License
-------
Copyright (c) 2018 StackHPC Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Author Information
------------------
Ironic Developers

View File

@ -0,0 +1,12 @@
---
# defaults file for bifrost-test-inspection
file_url_port: "8080"
# Whether to store introspection data using the local Nginx web server as an
# object storage service.
inspector_store_data_in_nginx: true
# When inspector_store_data_in_nginx is true, this is the URL of the Nginx
# 'Swift' API endpoint.
inspector_store_data_url: "http://localhost:{{ file_url_port }}"

View File

@ -0,0 +1,2 @@
---
# handlers file for bifrost-test-inspection

View File

@ -0,0 +1,116 @@
---
galaxy_info:
author: Ironic Developers
description: Tests inspection of nodes created by Bifrost.
company: OpenStack
license: Apache
min_ansible_version: 1.9
#
# Below are all platforms currently available. Just uncomment
# the ones that apply to your role. If you don't see your
# platform on this list, let us know and we'll get it added!
#
platforms:
#- name: EL
# versions:
# - all
# - 5
# - 6
# - 7
#- name: GenericUNIX
# versions:
# - all
# - any
#- name: Fedora
# versions:
# - all
# - 16
# - 17
# - 18
# - 19
# - 20
#- name: SmartOS
# versions:
# - all
# - any
#- name: opensuse
# versions:
# - all
# - 12.1
# - 12.2
# - 12.3
# - 13.1
# - 13.2
#- name: Amazon
# versions:
# - all
# - 2013.03
# - 2013.09
#- name: GenericBSD
# versions:
# - all
# - any
#- name: FreeBSD
# versions:
# - all
# - 8.0
# - 8.1
# - 8.2
# - 8.3
# - 8.4
# - 9.0
# - 9.1
# - 9.1
# - 9.2
#- name: Ubuntu
# versions:
# - all
# - lucid
# - maverick
# - natty
# - oneiric
# - precise
# - quantal
# - raring
# - saucy
- trusty
#- name: SLES
# versions:
# - all
# - 10SP3
# - 10SP4
# - 11
# - 11SP1
# - 11SP2
# - 11SP3
#- name: GenericLinux
# versions:
# - all
# - any
#- name: Debian
# versions:
# - all
# - etch
# - lenny
# - squeeze
# - wheezy
#
# Below are all categories currently available. Just as with
# the platforms above, uncomment those that apply to your role.
#
categories:
- cloud
- cloud:openstack
#- cloud:gce
#- cloud:rax
#- clustering
#- database
#- database:nosql
#- database:sql
#- development
#- monitoring
#- networking
#- packaging
#- system
#- web
dependencies: []

View File

@ -0,0 +1,40 @@
# Copyright (c) 2018 StackHPC Ltd.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
# TODO(mgoddard): Ideally we would grab inspection data from ironic inspector
# rather than going direct to the web server. That would require either
# installing python-openstackclient, or creating an ansible module that uses
# python-ironic-inspector-client.
- block:
- name: Check node hardware inspection data
uri:
url: "{{ inspector_store_data_url ~ '/ironic-inspector/inspector_data-' ~ uuid }}"
method: GET
return_content: True
register: inspection_data
# TODO(mgoddard): More validation of data format and contents.
- name: Validate the inspection data format
assert:
that:
- "'inventory' in data"
- "'memory' in inventory"
- "'cpu' in inventory"
- "'bmc_address' in inventory"
- "'interfaces' in inventory"
- "'disks' in inventory"
vars:
data: "{{ inspection_data.content | from_json }}"
inventory: "{{ data.inventory }}"
when: inspector_store_data_in_nginx | bool

View File

@ -0,0 +1,2 @@
---
# vars file for bifrost-test-inspection

View File

@ -99,6 +99,9 @@
roles:
- role: ironic-enroll-dynamic
- { role: ironic-inspect-node, when: inspect_nodes | default('false') | bool == true }
- role: bifrost-test-inspection
when: inspect_nodes | default('false') | bool == true
- hosts: baremetal
name: "Create configuration drive files and deploy machines"

View File

@ -0,0 +1,9 @@
---
features:
- |
Stores introspection data in nginx.
In the absence of swift, we can now use the bifrost nginx web server -
masquerading as an object store - to store raw and processed introspection
data for nodes. This is configured via the boolean variable
``inspector_store_data_in_nginx`` and is enabled by default.