openstack
/
searchlight
Code Issues Proposed changes

Retire the Searchlight project 

As announced in openstack-discuss ML[1], Searchlight project
is retiring in Wallaby cycle.

This commit retires this repository as per process deinfed in
project-guide[2]. Anyone would like to maintain it again, please
revert back this commit and propose the re-adding of Searchlight
to governance.

The community wishes to express our thanks and appreciation to all of
those who have contributed to the Searchlight project over the years.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/764519
Needed-By: https://review.opendev.org/c/openstack/governance/+/764530

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018637.html
[2] https://docs.openstack.org/project-team-guide/repository.html#retiring-a-repository

Change-Id: I0c5c9a961cca19113f4d59e37cb16e3bc0ccf8e1
This commit is contained in:
Ghanshyam Mann 2020-11-27 21:46:40 -06:00
parent ae76091a3b
commit 7331642231
388 changed files with 8 additions and 53968 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.coveragerc
View File

@ -1,7 +0,0 @@
[run]
branch = True
source = searchlight
omit = searchlight/tests/*
[report]
ignore_errors = True

44
.gitignore vendored
View File

@ -1,44 +0,0 @@
*.pyc
*.log
.searchlight-venv
.venv
.stestr/
.tox
.coverage*
cover/*
covhtml
nosetests.xml
coverage.xml
searchlight.sqlite
AUTHORS
ChangeLog
build
.DS_Store
dist
*.egg
searchlight.egg-info
tests.sqlite
searchlight/versioninfo
etc/searchlight.conf.sample
!/.coveragerc
#Files created by releasenotes build
releasenotes/build
# Swap files range from .saa to .swp
*.s[a-w][a-p]
# generated policy file
etc/searchlight/policy.yaml.sample
# Files created by doc build
doc/source/api
_static/
# IDE files
.project
.pydevproject
.idea
.e4p
.eric5project/
.issues/
.ropeproject

4
.stestr.conf
View File

@ -1,4 +0,0 @@
[DEFAULT]
test_path=./searchlight/tests
top_dir=.

8
.zuul.yaml
View File

@ -1,8 +0,0 @@
- project:
templates:
- openstack-python3-victoria-jobs
- openstack-cover-jobs
- publish-openstack-docs-pti
- check-requirements
- openstack-lower-constraints-jobs
- release-notes-jobs-python3

21
CONTRIBUTING.rst
View File

@ -1,21 +0,0 @@
If you would like to contribute to the development of OpenStack,
you must follow the steps documented at:
https://docs.openstack.org/infra/manual/developers.html#development-workflow
Once those steps have been completed, changes to OpenStack
should be submitted for review via the Gerrit tool, following
the workflow documented at:
https://docs.openstack.org/infra/manual/developers.html#development-workflow
Pull requests submitted through GitHub will be ignored.
Bugs should be filed on Storyboard, not GitHub:
https://storyboard.openstack.org/#!/project_group/searchlight
Further documentation on feature request and bug report processes may be
found here:
https://docs.openstack.org/searchlight/latest/contributor/feature-requests-bugs.html

19
HACKING.rst
View File

@ -1,19 +0,0 @@
Searchlight Style Commandments
==============================
- Step 1: Read the OpenStack Style Commandments
https://docs.openstack.org/hacking/latest
- Step 2: Read on
Searchlight Specific Commandments
---------------------------------
- [SL316] Change assertTrue(isinstance(A, B)) by optimal assert like
assertIsInstance(A, B)
- [SL317] Change assertEqual(type(A), B) by optimal assert like
assertIsInstance(A, B)
- [SL318] Change assertEqual(A, None) or assertEqual(None, A) by optimal assert like
assertIsNone(A)
- [SL319] Validate that logs are not translated
- [SL327] Prevent use of deprecated contextlib.nested
- [SL343] Check for common double word typos

176
LICENSE
View File

@ -1,176 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

72
README.rst
View File

@ -1,66 +1,10 @@
========================
Team and repository tags
========================
This project is no longer maintained.
.. image:: https://governance.openstack.org/tc/badges/searchlight.svg
:target: https://governance.openstack.org/tc/reference/tags/index.html
:alt: Searchlight is an official OpenStack project as indicated by
the "project:official" badge
.. NOTE(rosmaita): the alt text above will have to be updated when
additional tags are asserted for Searchlight. (The SVG in the
governance repo is updated automatically.)
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
.. Change things from this point on
========
Welcome!
========
The Searchlight project provides indexing and search capabilities across
OpenStack resources. Its goal is to achieve high performance and flexible
querying combined with near real-time indexing. It uses Elasticsearch, a
real-time distributed indexing and search engine built on Apache Lucene, but
adds OpenStack authentication and Role Based Access Control to provide
appropriate protection of data.
Documentations
==============
* Installation instruction and user guides:
https://docs.openstack.org/searchlight/latest
* Searchlight wiki:
https://wiki.openstack.org/wiki/Searchlight
* Release notes: https://docs.openstack.org/releasenotes/searchlight/
* Specs: https://specs.openstack.org/openstack/searchlight-specs/
Code
====
Searchlight code is available in the following repositories:
* Searchlight API and Listener:
https://opendev.org/openstack/searchlight
* Searchlight Python client:
https://opendev.org/openstack/python-searchlightclient
* Searchlight Horizon UI:
https://opendev.org/openstack/searchlight-ui
Bugs
====
Please report bugs at https://storyboard.openstack.org/#!/project_group/searchlight
License
=======
Apache Licence 2.0
Other Resources
===============
Use the following resources to learn more:
* Storyboard project: https://storyboard.openstack.org/#!/project_group/searchlight
* Send mail to openstack-discuss@lists.openstack.org with [Searchlight]
tag for help and hacking of Searchlight
* IRC channel: #openstack-searchlight
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-dev on
Freenode.

209
api-ref/source/conf.py
View File

@ -1,209 +0,0 @@
# -*- coding: utf-8 -*-
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# searchlight api-ref build config file, copied from:
# nova documentation build configuration file, created by
# sphinx-quickstart on Sat May 1 15:17:47 2010.
#
# This file is execfile()d with the current directory set to
# its containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import os
import sys
html_theme = 'openstackdocs'
html_theme_options = {
"sidebar_mode": "toc",
}
extensions = [
'os_api_ref',
'openstackdocstheme'
]
# openstackdocstheme options
openstackdocs_repo_name = 'openstack/searchlight'
openstackdocs_bug_project = 'searchlight'
openstackdocs_bug_tag = ''
openstackdocs_auto_name = False
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('../../'))
sys.path.insert(0, os.path.abspath('../'))
sys.path.insert(0, os.path.abspath('./'))
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#
# source_encoding = 'utf-8'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'Search Service API Reference'
copyright = u'2015-present, OpenStack Foundation'
# for a list of supported languages.
#
# language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
# today = ''
# Else, today_fmt is used as the format for a strftime call.
# today_fmt = '%B %d, %Y'
# The reST default role (used for this markup: `text`) to use
# for all documents.
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = False
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'native'
# -- Options for man page output ----------------------------------------------
# Grouping the document tree for man pages.
# List of tuples 'sourcefile', 'target', u'title', u'Authors name', 'manual'
# -- Options for HTML output --------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
# html_theme = '_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
# html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
# html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
# html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static']
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
# html_additional_pages = {}
# If false, no module index is generated.
# html_use_modindex = True
# If false, no index is generated.
# html_use_index = True
# If true, the index is split into individual pages for each letter.
# html_split_index = False
# If true, links to the reST sources are added to the pages.
# html_show_sourcelink = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
# html_use_opensearch = ''
# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = ''
# Output file base name for HTML help builder.
htmlhelp_basename = 'searchlightdoc'
# -- Options for LaTeX output -------------------------------------------------
# The paper size ('letter' or 'a4').
# latex_paper_size = 'letter'
# The font size ('10pt', '11pt' or '12pt').
# latex_font_size = '10pt'
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto/manual]).
latex_documents = [
('index', 'Searchlight.tex', u'OpenStack Search Service API Documentation',
u'OpenStack Foundation', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
# latex_use_parts = False
# Additional stuff for the LaTeX preamble.
# latex_preamble = ''
# Documents to append as an appendix to all manuals.
# latex_appendices = []
# If false, no module index is generated.
# latex_use_modindex = True

55
api-ref/source/http-codes.yaml
View File

@ -1,55 +0,0 @@
200:
default: |
The response contains an entity corresponding to the requested resource.
201:
default: |
The request has been fullfilled and a new resource created.
202:
default: |
The request has been accepted for processing, but the processing has not
been completed.
204:
default: |
The request has been processed successfully.
400:
default: |
The server cannot (or will not) process the request due to a client error.
informal: |
Your bad ... something is wrong with your request.
401:
default: |
You have either forgotten to provide credentials, or the credentials you
provided are malformed or expired.
403:
default: |
You do not have sufficient permissions to perform the requested action.
404:
default: |
The resource you have requested cannot be found.
409:
default: |
The request you have made cannot be processed because it conflicts with
the current state of the resource.
410:
default: |
The requested resource is no longer available at the server and no
forwarding address is known.
413:
default: |
The payload you have submitted is too large given the configuration of the
server.
415:
default: |
The server is refusing to service the request because the entity of the
request is in a format not supported by the requested resource for the
requested method.
500:
default: |
The server encountered an unexpected condition which prevented it from
fulfilling the request.
informal: |
Our bad ... something unfortunate happened on the server.
503:
default: |
The server is currently unable to handle the request due to a temporary
overloading or maintenance of the server.

25
api-ref/source/index.rst
View File

@ -1,25 +0,0 @@
..
Copyright 2016 OpenStack Foundation
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
:tocdepth: 3
==================
Search Service API
==================
.. rest_expand_all::
.. include:: searchlight-v1.inc

6
api-ref/source/samples/create-search-admin-request.json
View File

@ -1,6 +0,0 @@
{
"query": {
"match_all": {}
},
"all_projects": true
}

11
api-ref/source/samples/create-search-aggregation-request.json
View File

@ -1,11 +0,0 @@
{
"query": {
"match_all": {}
},
"type": ["OS::Glance::Image"],
"limit": 0,
"aggregations": {
"name": {"terms": {"field": "name"}},
"container_format": {"terms": {"field": "container_format"}}
}
}

64
api-ref/source/samples/create-search-aggregation-response.json
View File

@ -1,64 +0,0 @@
{
"_shards": {
"successful": 1,
"failed": 0,
"total": 1
},
"aggregations": {
"container_format": {
"buckets": [
{
"doc_count": 101,
"key": "bare"
},
{
"doc_count": 1,
"key": "aki"
},
{
"doc_count": 1,
"key": "ami"
},
{
"doc_count": 1,
"key": "ari"
}
],
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0
},
"name": {
"buckets": [
{
"doc_count": 100,
"key": "image"
},
{
"doc_count": 3,
"key": "0.3.4"
},
{
"doc_count": 3,
"key": "cirros"
},
{
"doc_count": 3,
"key": "uec"
},
{
"doc_count": 3,
"key": "x86_64"
}
],
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0
}
},
"hits": {
"hits": [],
"max_score": 0.0,
"total": 104
},
"took": 5,
"timed_out": false
}

5
api-ref/source/samples/create-search-all-request.json
View File

@ -1,5 +0,0 @@
{
"query": {
"match_all": {}
}
}

61
api-ref/source/samples/create-search-all-response.json
View File

@ -1,61 +0,0 @@
{
"_shards": {
"failed": 0,
"successful": 2,
"total": 2
},
"hits": {
"hits": [
{
"_id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"_index": "searchlight",
"_type": "OS::Glance::Image",
"_score": 1.0,
"_source": {
"id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"members": [],
"name": "cirros-0.3.2-x86_64-uec",
"owner": "d95b27da6e9f4acc9a8031918e443e04",
"visibility": "public",
"...": "..."
}
},
{
"_id": "OS::Software::DBMS",
"_index": "searchlight",
"_type": "OS::Glance::Metadef",
"_score": 1.0,
"_source": {
"description": "A database is an ...",
"display_name": "Database Software",
"namespace": "OS::Software::DBMS",
"objects": [
{
"description": "PostgreSQL, often simply 'Postgres' ...",
"name": "PostgreSQL",
"properties": [
{
"default": "5432",
"description": "Specifies the TCP/IP port...",
"property": "sw_database_postgresql_listen_port",
"...": "..."
},
{ "...": "..." }
]
}
],
"tags": [
{
"name": "Database"
}
]
}
},
{ "...": "..." }
],
"max_score": 1.0,
"total": 8
},
"timed_out": false,
"took": 1
}

25
api-ref/source/samples/create-search-complex-boolean-request.json
View File

@ -1,25 +0,0 @@
{
"type": "OS::Glance::Image",
"query": {
"bool" : {
"must" : [
{ "term" : { "status" : "active" } },
{ "term" : { "visibility" : "public" } }
],
"must_not" : [
{ "range" : { "min_ram" : { "gt" : 4096 } } },
{ "terms" : { "disk_format" : ["aki", "ari"] } }
],
"should" : [
{ "match" : { "name" : "cirros" } },
{ "match" : { "tag" : "cirros" } }
],
"minimum_should_match" : 1
}
},
"highlight": {
"fields": {
"*":{}
}
}
}

93
api-ref/source/samples/create-search-complex-boolean-response.json
View File

@ -1,93 +0,0 @@
{
"hits": {
"hits": [
{
"_type": "OS::Glance::Image",
"_source": {
"status": "active",
"created_at": "2016-08-25T17:32:41Z",
"project_id": "98c05407d80c405aa6efa01b279385ce",
"name": "cirros-0.3.4-x86_64-uec",
"tags": [],
"image_type": "image",
"checksum": "eb9139e4942121f22bbc2afc0400b2a4",
"min_ram": 0,
"ramdisk_id": "f310fb51-29dc-47bf-8cbd-3ea1170fb4e3",
"disk_format": "ami",
"updated_at": "2016-08-25T17:32:42Z",
"visibility": "public",
"kernel_id": "34ddf1d2-b588-4a9a-a3d3-344768c04bfd",
"protected": false,
"members": [],
"container_format": "ami",
"min_disk": 0,
"owner": "98c05407d80c405aa6efa01b279385ce",
"virtual_size": null,
"id": "23fa3971-3ed8-435a-a3c6-8addea5153aa",
"size": 25165824
},
"_score": 3.3642778,
"_index": "searchlight-2016_08_25_18_48_30",
"highlight": {
"status": [
"<em>active</em>"
],
"name": [
"<em>cirros</em>-0.3.4-x86_64-uec"
],
"visibility": [
"<em>public</em>"
]
},
"_id": "23fa3971-3ed8-435a-a3c6-8addea5153aa"
},
{
"_type": "OS::Glance::Image",
"_source": {
"status": "active",
"created_at": "2016-08-25T17:32:34Z",
"project_id": "98c05407d80c405aa6efa01b279385ce",
"name": "cirros-0.3.4-x86_64-disk",
"tags": [],
"image_type": "image",
"checksum": "ee1eca47dc88f4879d8a229cc70a07c6",
"min_ram": 0,
"disk_format": "qcow2",
"updated_at": "2016-08-25T17:32:35Z",
"visibility": "public",
"owner": "98c05407d80c405aa6efa01b279385ce",
"protected": false,
"members": [],
"container_format": "bare",
"min_disk": 0,
"virtual_size": null,
"id": "f34de38c-7a5c-4b00-8015-70a7a63480e8",
"size": 13287936
},
"_score": 3.3642778,
"_index": "searchlight-2016_08_25_18_48_30",
"highlight": {
"status": [
"<em>active</em>"
],
"name": [
"<em>cirros</em>-0.3.4-x86_64-disk"
],
"visibility": [
"<em>public</em>"
]
},
"_id": "f34de38c-7a5c-4b00-8015-70a7a63480e8"
}
],
"total": 2,
"max_score": 3.3642778
},
"_shards": {
"successful": 1,
"failed": 0,
"total": 1
},
"took": 1,
"timed_out": false
}

9
api-ref/source/samples/create-search-free-text-request.json
View File

@ -1,9 +0,0 @@
{
"type": "OS::Glance::Image",
"query": {
"multi_match": {
"query":"cirros",
"fields": ["_all"]
}
}
}

45
api-ref/source/samples/create-search-free-text-response.json
View File

@ -1,45 +0,0 @@
{
"_shards": {
"failed": 0,
"successful": 2,
"total": 2
},
"hits": {
"hits": [
{
"_id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"_index": "searchlight",
"_type": "OS::Glance::Image",
"_score": 1.0,
"_source": {
"id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"members": [],
"name": "cirros-0.3.2-x86_64-uec",
"owner": "d95b27da6e9f4acc9a8031918e443e04",
"visibility": "public",
"...": "..."
}
},
{
"_id": "15c2501d-07bd-4eb1-b15f-77ef739e5419",
"_index": "searchlight",
"_type": "OS::Glance::Image",
"_score": 1.0,
"_source": {
"id": "15c2501d-07bd-4eb1-b15f-77ef739e5419",
"description": "Extra special cirros image",
"members": [],
"name": "Simply Sensational",
"owner": "d95b27da6e9f4acc9a8031918e443e04",
"visibility": "public",
"...": "..."
}
},
{ "...": "..." }
],
"max_score": 1.0,
"total": 8
},
"timed_out": false,
"took": 1
}

6
api-ref/source/samples/create-search-multiple-type-request.json
View File

@ -1,6 +0,0 @@
{
"query": {
"match_all": {}
},
"type": ["OS::Glance::Image", "OS::Glance::Metadef"]
}

8
api-ref/source/samples/create-search-phrase-field-request.json
View File

@ -1,8 +0,0 @@
{
"type": "OS::Glance::Image",
"query": {
"match_phrase": {
"description": "Preconfigured with a schmaltz to optimize the foobar"
}
}
}

45
api-ref/source/samples/create-search-phrase-field-response.json
View File

@ -1,45 +0,0 @@
{
"hits": {
"hits": [
{
"_score": 12.945373,
"_type": "OS::Glance::Image",
"_id": "7a99b58d-0b70-4138-8086-ab0cc4bd1ba3",
"_source": {
"container_format": "bare",
"min_ram": 0,
"updated_at": "2016-08-25T17:54:02Z",
"cim_pasd_processorarchitecture": "Power",
"owner": "158776c9face41a7a3026d6c1cae686a",
"id": "7a99b58d-0b70-4138-8086-ab0cc4bd1ba3",
"size": 5969,
"image_type": "image",
"disk_format": "raw",
"project_id": "158776c9face41a7a3026d6c1cae686a",
"status": "active",
"description": "Preconfigured with a schmaltz to optimize the foobar",
"tags": [],
"hw_cpu_policy": "shared",
"visibility": "public",
"members": [],
"min_disk": 0,
"virtual_size": null,
"name": "My Awesome Image",
"checksum": null,
"created_at": "2016-08-25T17:54:02Z",
"protected": false
},
"_index": "searchlight-2016_08_25_18_48_30"
}
],
"total": 1,
"max_score": 12.945373
},
"_shards": {
"successful": 1,
"failed": 0,
"total": 1
},
"took": 4,
"timed_out": false
}

6
api-ref/source/samples/create-search-single-type-request.json
View File

@ -1,6 +0,0 @@
{
"query": {
"match_all": {}
},
"type": "OS::Glance::Image"
}

44
api-ref/source/samples/create-search-single-type-response.json
View File

@ -1,44 +0,0 @@
{
"_shards": {
"failed": 0,
"successful": 2,
"total": 2
},
"hits": {
"hits": [
{
"_id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"_index": "searchlight",
"_type": "OS::Glance::Image",
"_score": 1.0,
"_source": {
"id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"members": [],
"name": "cirros-0.3.2-x86_64-uec",
"owner": "d95b27da6e9f4acc9a8031918e443e04",
"visibility": "public",
"...": "..."
}
},
{
"_id": "a701f610-b162-45ac-a5a0-69e08aba8687",
"_index": "searchlight",
"_type": "OS::Glance::Image",
"_score": 1.0,
"_source": {
"id": "a701f610-b162-45ac-a5a0-69e08aba8687",
"members": [],
"name": "Docker container image",
"owner": "d95b27da6e9f4acc9a8031918e443e04",
"visibility": "public",
"...": "..."
}
},
{ "...": "..." }
],
"max_score": 1.0,
"total": 8
},
"timed_out": false,
"took": 1
}

78
api-ref/source/samples/list-facets-all-response-exclude-options.json
View File

@ -1,78 +0,0 @@
{
"OS::Glance::Image": {
"doc_count": 50,
"facets": [
{
"name": "status",
"type": "string"
},
{
"name": "created_at",
"type": "date"
},
{
"name": "virtual_size",
"type": "long"
},
{
"name": "name",
"type": "string",
"facet_field": "name.raw"
},
{ "...": "..." }
]
},
"OS::Glance::Metadef": {
"doc_count": 25,
"facets": [
{
"name": "objects.description",
"type": "string"
},
{
"name": "objects.properties.description",
"type": "string"
},
{ "...": "..." }
]
},
"OS::Nova::Server": {
"doc_count": 100,
"facets": [
{
"name": "status",
"options": [
{
"doc_count": 1,
"key": "ACTIVE"
}
],
"type": "string"
},
{
"name": "OS-EXT-SRV-ATTR:host",
"type": "string"
},
{
"name": "name",
"type": "string",
"facet_field": "name.raw"
},
{
"name": "image.id",
"type": "string"
},
{
"name": "OS-EXT-AZ:availability_zone",
"options": [
{
"doc_count": 1,
"key": "nova"
}
],
"type": "string"
},
{ "...": "..." }
]
}
}

90
api-ref/source/samples/list-facets-all-response.json
View File

@ -1,90 +0,0 @@
{
"OS::Glance::Image": {
"doc_count": 50,
"facets": [
{
"name": "status",
"type": "string",
"options": [
{
"key": "queued",
"doc_count": 47
},
{
"key": "active",
"doc_count": 3
}
]
},
{
"name": "created_at",
"type": "date"
},
{
"name": "virtual_size",
"type": "long"
},
{
"name": "name",
"type": "string",
"facet_field": "name.raw"
},
{ "...": "..." }
]
},
"OS::Glance::Metadef": {
"doc_count": 25,
"facets": [
{
"name": "objects.description",
"nested": true,
"type": "string"
},
{
"name": "objects.properties.description",
"nested": true,
"type": "string"
},
{ "...": "..." }
]
},
"OS::Nova::Server": {
"doc_count": 100,
"facets": [
{
"name": "status",
"options": [
{
"doc_count": 1,
"key": "ACTIVE"
}
],
"type": "string"
},
{
"name": "OS-EXT-SRV-ATTR:host",
"type": "string"
},
{
"name": "name",
"type": "string",
"facet_field": "name.raw"
},
{
"name": "image.id",
"type": "string"
},
{
"name": "OS-EXT-AZ:availability_zone",
"options": [
{
"doc_count": 1,
"key": "nova"
}
],
"type": "string"
},
{ "...": "..." }
]
}
}

11
api-ref/source/samples/list-facets-include-fields-response.json
View File

@ -1,11 +0,0 @@
{
"OS::Nova::Server": {
"doc_count": 7
},
"OS::Neutron::Net": {
"doc_count": 19
},
"OS::Glance::Image": {
"doc_count": 68
}
}

27
api-ref/source/samples/list-facets-type-response.json
View File

@ -1,27 +0,0 @@
{
"OS::Nova::Server": {
"doc_count": 100,
"facets": [
{
"name": "status",
"options": [
{
"doc_count": 1,
"key": "ACTIVE"
}
],
"type": "string"
},
{
"name": "OS-EXT-SRV-ATTR:host",
"type": "string"
},
{
"name": "name",
"type": "string",
"facet_field": "name.raw"
},
{ "...": "..." }
]
}
}

14
api-ref/source/samples/list-plugins-response.json
View File

@ -1,14 +0,0 @@
{
"plugins": [
{
"type": "OS::Glance::Image",
"alias-indexing": "searchlight-listener",
"alias-searching": "searchlight-search"
},
{
"type": "OS::Glance::Metadef",
"alias-indexing": "searchlight-listener",
"alias-searching": "searchlight-search"
}
]
}

170
api-ref/source/search-parameters.yaml
View File

@ -1,170 +0,0 @@
# template for entries
#name:
# description: |
# yada yada yada
# in: {header, path, query, body}
# required: {true, false}
# type: {JSON datatype: number, string, boolean, array, object, null}
#
# Note: entries must be in alphabetical order within each section
# variables in header
Content-type-json:
description: |
Mime type of the request body. Must be ``application/json``.
in: header
required: true
type: string
# variables in path
# variables in query
include-in-query:
description: |
Disable showing the facet fields. Display count only.
The default value is ``true``.
in: query
required: false
type: boolean
options-in-query:
description: |
Disable aggregating facet fields. Provides a performance
improvement. The default value is ``false``.
in: query
required: false
type: boolean
type-in-query:
description: |
The resource type whose facets you'd like to see listed.
in: query
required: false
type: string
# variables in body
all-projects:
description: |
Indicates that an administator wants to include all resources for
all projects in the search. The default is ``false``.
in: body
required: false
type: boolean
es-aggregate-obj:
description: |
A JSON object containing the aggregation results. It contains the following
fields:
- ``buckets`` - An array of aggregated results (defaults to the top ten
terms). There is a bucket for each term. Each element of the array
contains the following fields:
- ``doc_count`` - Number of times this term appears.
- ``key`` - The value of this term.
- ``doc_count_error_upper_bound`` - The upper bound on the error when
compiling the document count, needed since the count is approximate.
- ``sum_other_doc_count`` - Since the number of elements in the bucket
array may be capped (default is ten), the number of all other document
counts not included in buckets.
in: body
required: true
type: object
es-hits-obj:
description: |
A JSON object containing the search results. It contains the following
fields:
- ``hits`` - An array of search results (defaults to the first 25
documents). Each element of the array contains the following fields:
- ``_id`` - Uniquely identifies the resource within its OpenStack context
(for instance, Glance images use their Image ID)
- ``_index`` - The service to which the resource belongs (for example,
searchlight)
- ``_score`` - The relevance of a given hit. By default, this is the
field upon which results are sorted
- ``_source`` - The document originally indexed, containing all data
associated with the OpenStack resource. This field is a map,
where each key is a field whose value may be a scalar value, a list,
a nested object, or a list of nested objects.
- ``_type`` - The OpenStack resource type of the document (for example,
"OS::Neutron::Net", "OS::Glance::Image")
- ``_parent`` - *(optional)* When the resource type has a child
relationship to another resource type, the id of the specific parent
resource will appear here.
- ``_routing`` - *(optional)* Included when an Elasticsearch routing
for the document is explicitly specified.
- ``max_score`` - The highest ``_score`` of any document that matches the
query
- ``total`` - The total number of documents matching the search criteria
in: body
required: true
type: object
es-query:
description: |
A JSON object containing the requested query, expressed in the
`Elasticsearch Query DSL <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html>`_.
in: body
required: true
type: object
es-shards-obj:
description: |
A JSON object containing the following fields:
- ``failed`` - Number of shards in which the search could not be performed
- ``successful`` - Number of shards in which the search was successful
- ``total`` - The total number of shards involved in processing the query
in: body
required: true
type: object
es-timed-out-field:
description: |
Indicates whether the search timed out or not.
in: body
required: true
type: boolean
es-took-field:
description: |
The time in milliseconds it took for Elasticsearch to execute the search.
in: body
required: true
type: number
facet-list:
description: |
A JSON object pairing resource type names with a list of facets
supported for that resource type. Each facet object in the list
contains the following fields:
- name - the name of the facet
- type - the datatype of the facet
- facet_field - (optional, may be available for some string fields)
may be used to do an exact term match
in: body
required: true
type: object
multi_match-in-body:
description: |
A JSON object containing a ``query`` field whose value is an Elasticsearch
query string.
in: body
required: true
type: string or array of strings
plugin-list:
description: |
List of plugins. Each element of the list contains the following fields:
- ``alias-indexing`` - the alias this plugin uses to refer to the
Elasticsearch index in which documents are created
- ``alias-searching`` - the alias this plugin uses to refer to the
Elasticsearch index used for querying documents created by this plugin
- ``type`` - the document type associated with this plugin
in: body
required: true
type: array
type-in-body:
description: |
The name of the resource type or types to which the search should be
restricted.
in: body
required: true
type: string or array of strings

699
api-ref/source/searchlight-v1.inc
View File

@ -1,699 +0,0 @@
.. -*- rst -*-
Search
******
Provides indexing and search capabilities across OpenStack resources.
General information
~~~~~~~~~~~~~~~~~~~
The OpenStack Searchlight project provides a Search API that presents an
interface for querying about the various resources available in an OpenStack
cloud. Searchlight creates an index (using open source Elasticsearch
technology) and keeps it updated. Additionally, Searchlight's Search API
requires authentication and respects the Role Based Access Control defined by
each OpenStack service. This gives end users access to the powerful searching
facilities provided by Elasticsearch, but operators can rest assured that end
users won't have access to information they wouldn't be able to find in the
APIs exposed by each of the OpenStack services.
The Search API allows you to construct detailed queries using the
`Elasticsearch Query Domain Specific Language
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html>`_.
That's great if you are an Elasticsearch expert (or want to become one), but is
probably beyond what most end users need in order to use the Search API
effectively. So we'll provide a wide range of useful query examples below (see
`Create a general search`_).
To make effective use of the Search API, you need to know a few things about
OpenStack resource types, how Searchlight stores data about OpenStack
resources, and something about Searchlight plugins.
OpenStack resource types
------------------------
An OpenStack cloud offers users many resources, for example, servers, images,
networks. In order to eliminate ambiguity (for example, sometimes a resource
you create with Nova is called a "server", sometimes it's called an
"instance"), a standard vocabulary has developed based around that used by the
OpenStack `Heat <http://docs.openstack.org/developer/heat/>`_ project.
Heat identifies specific *resource types* that are assigned identifiers of the
form
``OS::{project_name}::{resource}``
where ``{project_name}`` and ``{resource}`` are replaced by an actual project
name and resource name. For example, a Nova server (or "instance") has the
resource type identifier ``OS::Nova::Server``.
A list of OpenStack resource type identifiers is maintained by the Heat project
at `OpenStack Resource Types
<http://docs.openstack.org/developer/heat/template_guide/openstack.html>`_.
To summarize, you need to know the identifier for each OpenStack resource type
you're interested in so that you can efficiently search for those resources.
Searchlight data
----------------
Searchlight creates a *document* to represent each individual item (server,
image, network) in an OpenStack cloud. These documents are stored in an
Elasticsearch *index*. When you make a query using the Search API, the
documents appropriate to your query (and your RBAC roles) are returned to you.
To facilitate searching, each *document* has a *document type*. Searchlight
uses the OpenStack *resource type* (described in the previous section) as the
*document type*. For example, a virtual machine image stored in Glance would
be represented by a *document* with the *document type* ``OS::Glance::Image``.
The particular *document* that corresponds to an individual OpenStack resource
requires a unique identifier. It just so happens that each OpenStack service
already assigns a unique identifier to each individual resource it controls, so
that's what Searchlight assigns as the *document ID*. For example, if you have
an image in Glance with ID ``997ec7e8-a46b-4ab7-a1d4-788e06f52abe``, then that
image will be represented in the *index* by a *document* with a *document ID*
of ``997ec7e8-a46b-4ab7-a1d4-788e06f52abe``.
The information in the document will be the same content that would be
available to you if you queried the native API that handles that particular
resource type. For example, a document with *document type*
``OS::Glance::Image`` will contain the same information about an image as you
would receive if you queried the Images API for information about that image.
The advantage to using Searchlight, of course, is that you have access to
advanced searching facilities provided by Elasticsearch.
Searchlight index
-----------------
In the previous section, you learned that each document stored in Elasticsearch
is uniquely identified by (a) what *index* it's in, (b) what *document type* it
has, and (c) what its *document ID* is. You already know how to determine the
*document type* (it's the resource type of the OpenStack resource you're
interested in) and the *document ID* (it's the same as the identifier assigned
to the resource by the OpenStack service that manages that resource). So
what's left is to figure out what index to use.
Due to careful engineering by top professionals, that's pretty easy.
Searchlight maintains its own index named 'searchlight'. Further, the Search
API automatically uses this index, so you don't need to specify it. We only
mention it here because you'll see its name in some Elasticsearch responses
that you'll receive from the Search API, so it's worth knowing what it is.
One final point. Searchlight accomplishes its magic by using *plugins* that
understand and index particular OpenStack resource types. So in order for a
resource type to be accessible via the Search API, there must be a Searchlight
plugin installed to index that kind of resource. Not all OpenStack resources
currently have Searchlight plugins associated with them, and not all OpenStack
deployers may have installed all available plugins. You can determine what's
available in a particular cloud by using the `List plugins`_ call.
.. TODO(rosmaita): include authentication section? what are other api-ref
doing?
Authentication
--------------
Like other OpenStack APIs, the Search API depends on Keystone and the
OpenStack Identity API to handle authentication. Hence, you must obtain an
authentication token from Keystone and pass it to Searchlight in API
requests with the ``X-Auth-Token`` header.
List plugins
~~~~~~~~~~~~
.. rest_method:: GET /v1/search/plugins
List the supported plugins.
In order to index OpenStack resources, Searchlight requires a *plugin* specific
to each resource type. Hence, the only resources that can be indexed in a
particular OpenStack cloud are those for which (a) a plugin exists, and (b) the
plugin is enabled by the cloud operator. You use this call to determine what
document types you can expect to be able to search in a particular cloud.
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
There are no request parameters, and this call does not take a request body.
Response
--------
.. rest_parameters:: search-parameters.yaml
- plugins: plugin-list
Response Example
----------------
.. literalinclude:: samples/list-plugins-response.json
:language: json
List facets
~~~~~~~~~~~
.. rest_method:: GET /v1/search/facets
List the supported facets.
For each registered resource type, Searchlight can provide a list of *field
names* and *values* present for those fields.
- Which fields are returned and whether values are listed is determined by
each plugin.
- Some fields or values may only be listed for administrative users.
- For some string fields, ``facet_field`` may be included in the result. If
present, it can be used to do an exact term match against facet options.
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
.. rest_parameters:: search-parameters.yaml
- exclude_options: options-in-query
- include_fields: include-in-query
- type: type-in-query
.. NOTE(rosmaita): we can force extra space below the parameters
table by inserting raw html here, but it would be better to
put up a patch on the os-api-ref project to modify the style in CSS
.. TODO(rosmaita): remove this example
.. raw:: html
<p></p>
This call does not take a request body.
Response
--------
.. rest_parameters:: search-parameters.yaml
- facets: facet-list
Response Example
----------------
.. TODO(rosmaita): decide what to do about admin-only field in
the example response
.. literalinclude:: samples/list-facets-all-response.json
:language: json
Response Example (including ``exclude_options`` query parameter)
----------------------------------------------------------------
Assume this request: ``GET /v1/search/facets?exclude_options=true``
.. literalinclude:: samples/list-facets-all-response-exclude-options.json
:language: json
Response Example (including ``type`` query parameter)
-----------------------------------------------------
Assume this request: ``GET /v1/search/facets?type=OS::Nova::Server``
.. literalinclude:: samples/list-facets-type-response.json
:language: json
Response Example (including ``include_fields`` query parameter)
---------------------------------------------------------------
Assume this request: ``GET /v1/search/facets?include_fields=false``
.. literalinclude:: samples/list-facets-include-fields-response.json
:language: json
.. _general-search:
Create a general search
~~~~~~~~~~~~~~~~~~~~~~~
.. rest_method:: POST /v1/search
Submit a search request.
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
.. rest_parameters:: search-parameters.yaml
- Content-type: Content-type-json
- query: es-query
Request Example
---------------
.. literalinclude:: samples/create-search-all-request.json
:language: json
Response
--------
.. rest_parameters:: search-parameters.yaml
- _shards: es-shards-obj
- hits: es-hits-obj
- timed_out: es-timed-out-field
- took: es-took-field
Response Example
----------------
.. literalinclude:: samples/create-search-all-response.json
:language: json
Create a type restricted search
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. rest_method:: POST /v1/search
Request a type restricted search.
By including a ``type`` field in your request, you can restrict
the results to a particular resource type or types.
The ``type`` field can take either a string value (the name of
a particular resource type) or an array of values (where each
value is the name of a resource type).
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
.. rest_parameters:: search-parameters.yaml
- Content-type: Content-type-json
- query: es-query
- type: type-in-body
Request Example (single resource type)
--------------------------------------
This example illustrates a request for a single resource type, Glance images.
.. literalinclude:: samples/create-search-single-type-request.json
:language: json
Request Example (multiple resource types)
-----------------------------------------
This example illustrates a request for multiple resource types, Glance images
and Glance metadefs.
.. literalinclude:: samples/create-search-multiple-type-request.json
:language: json
Response
--------
.. rest_parameters:: search-parameters.yaml
- _shards: es-shards-obj
- hits: es-hits-obj
- timed_out: es-timed-out-field
- took: es-took-field
Response Example (single resource type)
---------------------------------------
This example illustrates the response to a request for a single resource type,
Glance images (``OS::Glance::Image``).
.. literalinclude:: samples/create-search-single-type-response.json
:language: json
Response Example (multiple resource types)
------------------------------------------
This example illustrates the resonse to a request for multiple resource types,
Glance images (``OS::Glance::Image``) and Glance metadefs
(``OS::Glance::Metadef``).
.. literalinclude:: samples/create-search-all-response.json
:language: json
Create an administrative search
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. rest_method:: POST /v1/search
Submit an administrative search request.
By default, the results a user receives in a search response are limited
to public resources and those resources owned by (or shared with) the user's
project (or "tenant").
Administrators have the option to view all resources in the index by passing
the ``all_projects`` field in the search request body.
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
.. rest_parameters:: search-parameters.yaml
- Content-type: Content-type-json
- query: es-query
- all_projects: all-projects
Request Example
---------------
.. literalinclude:: samples/create-search-admin-request.json
:language: json
Response
--------
.. rest_parameters:: search-parameters.yaml
- _shards: es-shards-obj
- hits: es-hits-obj
- timed_out: es-timed-out-field
- took: es-took-field
Response Example
----------------
.. literalinclude:: samples/create-search-all-response.json
:language: json
Create a free text search
~~~~~~~~~~~~~~~~~~~~~~~~~
.. rest_method:: POST /v1/search
Submit a free text search request.
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
.. rest_parameters:: search-parameters.yaml
- Content-type: Content-type-json
- type: type-in-body
- multi_match: multi_match-in-body
Request Example
---------------
Suppose you want to find all images that have the word "cirros" in some field,
whether the in the name, tags, or any other image properties.
.. literalinclude:: samples/create-search-free-text-request.json
:language: json
Response
--------
.. rest_parameters:: search-parameters.yaml
- _shards: es-shards-obj
- hits: es-hits-obj
- timed_out: es-timed-out-field
- took: es-took-field
Response Example
----------------
.. literalinclude:: samples/create-search-free-text-response.json
:language: json
Create a phrase search
~~~~~~~~~~~~~~~~~~~~~~
.. rest_method:: POST /v1/search
Submit a search request that looks for a phrase in a particular field.
A common scenario is that you have custom metadata on some resources describing
the resource in natural language. For example, you may have set a property
named ``description`` on your images, where the value is a natural language
sentence describing the recommended uses for that image. You can query the
Search API for responses that match that phrase in that field.
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
Suppose you use a ``description`` property on images as described above, and
you are interested in finding any images that are described as "Preconfigured
with a schmaltz to optimize the foobar."
.. rest_parameters:: search-parameters.yaml
- Content-type: Content-type-json
- query: es-query
Request Example
---------------
.. literalinclude:: samples/create-search-phrase-field-request.json
:language: json
Response
--------
.. rest_parameters:: search-parameters.yaml
- _shards: es-shards-obj
- hits: es-hits-obj
- timed_out: es-timed-out-field
- took: es-took-field
Response Example
----------------
.. literalinclude:: samples/create-search-phrase-field-response.json
Create a compound boolean search
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. rest_method:: POST /v1/search
Submit a compound boolean search request.
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
.. rest_parameters:: search-parameters.yaml
- Content-type: Content-type-json
- query: es-query
Request Example
---------------
Suppose you want to find all the images that meet the following set of
conditions:
- The image must be *public* and *active*. (This is a *must clause*.)
- The image *cannot require more than 4096 RAM*. (This is a *must_not clause
with a range query*.)
- The image must *not* be ``aki`` or ``ari``. (This is a *must_not clause*)
- Either the *name* or the *tags* should contain "cirros". (This is a *should
clause with minimum match of 1*.)
- In the results, indicate ("highlight") why these results were included in the
request.
.. literalinclude:: samples/create-search-complex-boolean-request.json
:language: json
Response
--------
.. rest_parameters:: search-parameters.yaml
- _shards: es-shards-obj
- hits: es-hits-obj
- timed_out: es-timed-out-field
- took: es-took-field
Response Example
----------------
.. literalinclude:: samples/create-search-complex-boolean-response.json
Create an aggregated search
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. rest_method:: POST /v1/search
Submit an aggregated search request.
.. rest_status_code:: success http-codes.yaml
- 200
.. rest_status_code:: error http-codes.yaml
- 400
- 401
- 403
- 500
Request
-------
.. rest_parameters:: search-parameters.yaml
- Content-type: Content-type-json
- query: es-query
Request Example
---------------
Suppose you want to find a count of the different names and container_formats
in all images.
.. literalinclude:: samples/create-search-aggregation-request.json
:language: json
Response
--------
.. rest_parameters:: search-parameters.yaml
- _shards: es-shards-obj
- hits: es-hits-obj
- aggregations: es-aggregate-obj
- timed_out: es-timed-out-field
- took: es-took-field
Response Example
----------------
.. literalinclude:: samples/create-search-aggregation-response.json
Notes
~~~~~
The index you see in responses is not a literal ElasticSearch index.
Here are a few points to keep in mind:
- For architectural reasons (primarily to allow zero-downtime re-indexing),
Searchlight doesn't refer to the index directly, but instead uses *index
aliases*. The key impact this has on you as a consumer of the API is that
you'll see references to aliases, instead of directly to the index, in
some Search API responses.
- All accesses to ElasticSearch are done through aliases and not directly
through an index. For more details on aliases see the `List plugins`_
call. The actual indices behind the aliases are transient, since a new
one is created whenever Searchlight is re-indexed.
- By default, Searchlight uses a single index named 'searchlight'. An
operator, however, may choose to use a different name. Further, an
operator may choose to have different resource types indexed into
different indices so that, for example, "OS::Cinder::Volume" can have one
Elasticsearch index, "OS::Neutron::Net" can have a different Elasticsearch
index and "OS::Neutron::SecurityGroup" can have a third separate
ElasticSearch index. When this is done, the aliases exposed via the
Searchlight API will also change accordingly.
References
~~~~~~~~~~
- `Bool Query
<https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html>`_
- `Range Query
<https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html>`_
- `Highlighting
<https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html>`_
- `Compound Queries
<https://www.elastic.co/guide/en/elasticsearch/reference/current/compound-queries.html>`_
- `Aggregate Queries
<https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html>`_

2
bindep.txt
View File

@ -1,2 +0,0 @@
default-jre [!platform:ubuntu-bionic]
openjdk-8-jre [platform:ubuntu-bionic]

30
contrib/docker/Dockerfile
View File

@ -1,30 +0,0 @@
FROM ubuntu:18.04
MAINTAINER saphi070@gmail.com
RUN apt update
RUN apt install -y python-dev libssl-dev python-pip libxml2-dev \
libxslt-dev git git-review libffi-dev gettext \
python-tox python-memcache crudini
### Clone searchlight source code
RUN apt install wget -y
WORKDIR /home
RUN git clone https://opendev.org/openstack/searchlight
WORKDIR /home/searchlight
### Install Searchlight
RUN pip install -r requirements.txt
RUN pip install -r elasticsearch5.txt
RUN python setup.py install
RUN oslo-config-generator --config-file etc/oslo-config-generator/searchlight.conf
RUN cp etc/searchlight.conf.sample /etc/searchlight/searchlight.conf
RUN mkdir /etc/searchlight/
### Copy Configure files
RUN cp etc/api-paste.ini /etc/searchlight/
# COPY searchlight.conf /etc/searchlight/
COPY entrypoint.sh /home/
RUN chmod 775 /home/entrypoint.sh
EXPOSE 9393
WORKDIR /home
ENTRYPOINT ["./entrypoint.sh"]

76
contrib/docker/README.rst
View File

@ -1,76 +0,0 @@
Running Searchlight in containers
=================================
This contrib includes Dockerfile, docker-compose file and some configurations to run
Searchlight in Docker containers.
Requirements
------------
Follow guides in [1], [2] to install docker and docker-compose
Usage
-----
- Build docker image
::
docker-compose -f docker-compose.example.yml build
- Adjust the authentication variables in your `docker-compose.example.yml`.
Example
::
version: "3"
services:
searchlight-api:
build: .
ports:
- 9393:9393
environment:
PROCESS: api
AUTH_URL: http://192.168.53.31:5000/v3
TRANSPORT_URL: rabbitmq://openstack:openstack@192.168.53.31
SEARCHLIGHT_PASS: openstack
ELASTICSEARCH_HOST: elasticsearch:9200
searchlight-listener:
build: .
environment:
PROCESS: listener
AUTH_URL: http://192.168.53.31:5000/v3
TRANSPORT_URL: rabbitmq://openstack:openstack@192.168.53.31
SEARCHLIGHT_PASS: openstack
ELASTICSEARCH_HOST: elasticsearch:9200
elasticsearch:
image: elasticsearch:5.6
- Running
::
docker-compose -f docker-compose.example.yml up -
Environment Variables
---------------------
- TRANSPORT_URL: RabbitMQ URL. Example: rabbitmq://openstack:openstack@192.168.53.31
- REGION_NAME: Openstack Region Name. Example: RegionOne
- AUTH_URL: Keystone Auth URL. Example: http://192.168.53.31:5000
- SEARCHLIGHT_USER: Username of searchlight. Example: searchlight
- SEARCHLIGHT_PASS: Password of searchlight. Example: openstack
- PROJECT_NAME: Name of project for searchlight. Example: service
- DOMAIN_NAME: Name of domain for searchlight user and project: Example: default
- ELASTICSEARCH_HOST: Host and port of Elasticsearch. Example: 192.168.53.31:9200
References
----------
- [1] Install docker on Ubuntu: https://docs.docker.com/install/linux/docker-ce/ubuntu/
- [2] Install docker-compose: https://docs.docker.com/compose/install/

24
contrib/docker/docker-compose.example.yml
View File

@ -1,24 +0,0 @@
version: "3"
services:
searchlight-api:
build: .
ports:
- 9393:9393
environment:
PROCESS: api
AUTH_URL: http://192.168.53.31:5000/v3
TRANSPORT_URL: rabbitmq://openstack:openstack@192.168.53.31
SEARCHLIGHT_PASS: openstack
ELASTICSEARCH_HOST: elasticsearch:9200
searchlight-listener:
build: .
environment:
PROCESS: listener
AUTH_URL: http://192.168.53.31:5000/v3
TRANSPORT_URL: rabbitmq://openstack:openstack@192.168.53.31
SEARCHLIGHT_PASS: openstack
ELASTICSEARCH_HOST: elasticsearch:9200
elasticsearch:
image: elasticsearch:5.6

52
contrib/docker/entrypoint.sh
View File

@ -1,52 +0,0 @@
#!/bin/bash
echo "Configure Searhclight"
searchlight_conf=/etc/searchlight/searchlight.conf
DOMAIN_NAME=${DOMAIN_NAME:-default}
PROJECT_NAME=${PROJECT_NAME:-service}
SEARCHLIGHT_USER=${SEARCHLIGHT_USER:-searchlight}
SEARCHLIGHT_PASS=${SEARCHLIGHT_PASS:-openstack}
crudini --set $searchlight_conf DEFAULT transport_url $TRANSPORT_URL
crudini --set $searchlight_conf service_credentials os_region_name $REGION_NAME
crudini --set $searchlight_conf service_credentials auth_url $AUTH_URL
crudini --set $searchlight_conf service_credentials username $SEARCHLIGHT_USER
crudini --set $searchlight_conf service_credentials password $SEARCHLIGHT_PASS
crudini --set $searchlight_conf service_credentials project_name $PROJECT_NAME
crudini --set $searchlight_conf service_credentials user_domain_name $DOMAIN_NAME
crudini --set $searchlight_conf service_credentials project_domain_name $DOMAIN_NAME
crudini --set $searchlight_conf keystone_authtoken auth_url $AUTH_URL
crudini --set $searchlight_conf keystone_authtoken auth_type password
crudini --set $searchlight_conf keystone_authtoken project_domain_name $DOMAIN_NAME
crudini --set $searchlight_conf keystone_authtoken user_domain_name $DOMAIN_NAME
crudini --set $searchlight_conf keystone_authtoken project_name $PROJECT_NAME
crudini --set $searchlight_conf keystone_authtoken username $SEARCHLIGHT_USER
crudini --set $searchlight_conf keystone_authtoken password $SEARCHLIGHT_PASS
crudini --set $searchlight_conf elasticsearch hosts $ELASTICSEARCH_HOST
if [ ! -f /opt/searchlight-synced ];then
echo "Sync index to elasticsearch"
while true;
do
curl -s $ELASTICSEARCH_HOST > /dev/null
if [ $? -eq 0 ]; then
echo "Starting sync index to elasticsearch"
searchlight-manage index sync --force
break
fi
done
touch /opt/searchlight-synced
fi
echo "Starting searchlight"
if [ ! -d /var/log/searchlight ]; then
mkdir /var/log/searchlight
fi
searchlight-$PROCESS --log-file=/var/log/searchlight/searchlight-$PROCESS --config-file $searchlight_conf

70
contrib/docker/searchlight.conf
View File

@ -1,70 +0,0 @@
[DEFAULT]
verbose = true
transport_url = rabbit://openstack:openstack@rabbitmq
[service_credentials]
os_region_name = RegionOne
auth_url = https://keystone:35357
password = openstack
project_name = service
user_domain_name = Default
project_domain_name = Default
username = searchlight
auth_type = password
[paste_deploy]
flavor = keystone
[keystone_authtoken]
auth_version = 3
auth_url = https://keystone:5000
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = searchlight
password = openstack
[elasticsearch]
index_settings = number_of_shards:1,number_of_replicas:0
hosts = elasticsearch:9200
[service_credentials:nova]
compute_api_version = 2.1
[resource_plugin:os_nova_server]
enabled = True
[resource_plugin:os_nova_hypervisor]
enabled = True
[resource_plugin:os_nova_flavor]
notifications_topics_exchanges = versioned_notifications,nova
enabled = True
[resource_plugin:os_nova_servergroup]
enabled = True
[resource_plugin:os_glance_image]
enabled = True
[resource_plugin:os_glance_metadef]
enabled = True
[resource_plugin:os_searchlight_volume]
enabled = True
[resource_plugin:os_searchlight_snapshot]
enabled = True
[resource_plugin:os_neutron_net]
enabled = True
[resource_plugin:os_neutron_port]
enabled = True
[resource_plugin:os_neutron_floatingip]
enabled = True
[resource_plugin:os_neutron_security_group]
enabled = True

77
contrib/vagrant/Vagrantfile vendored
View File

@ -1,77 +0,0 @@
VAGRANTFILE_API_VERSION = "2"
GITCONFIG = `cat $HOME/.gitconfig`
Vagrant.require_version ">= 1.5"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.ssh.forward_agent = true
config.vm.provider "virtualbox" do |vb, override|
vb.customize ["modifyvm", :id, "--memory", "6144"]
if not RUBY_PLATFORM.downcase.include?("mswin")
vb.customize ["modifyvm", :id, "--cpus", `awk "/^processor/ {++n} END {print n}" /proc/cpuinfo 2> /dev/null || sh -c 'sysctl hw.logicalcpu 2> /dev/null || echo ": 2"' | awk \'{print \$2}\' `.chomp ]
end
vb.customize ["modifyvm", :id, "--nicpromisc3", "allow-all"]
override.vm.synced_folder "../..", "/opt/stack/searchlight"
end
config.vm.provider :libvirt do |lv, override|
lv.graphics_ip = '0.0.0.0'
lv.nested = true
lv.memory = 8192
if not RUBY_PLATFORM.downcase.include?("mswin")
lv.cpus = `awk "/^processor/ {++n} END {print n}" /proc/cpuinfo 2> /dev/null || sh -c 'sysctl hw.logicalcpu 2> /dev/null || echo ": 2"' | awk \'{print \$2}\' `.chomp
end
override.vm.synced_folder ".", "/vagrant", type: "nfs"
override.vm.synced_folder "../..", "/opt/stack/searchlight", type: "nfs"
end
$script = <<SCRIPT
set -e
# Fixup permissions on /opt/stack/
sudo chown vagrant:vagrant /opt/stack/
# Copy over git config
cat << EOF > /home/vagrant/.gitconfig
#{GITCONFIG}
EOF
# Clone DevStack
if [ ! -d "/home/vagrant/devstack" ]; then
git clone https://opendev.org/openstack/devstack.git /home/vagrant/devstack
fi
# Install Vagrant localrc sample
cd /opt/stack/searchlight/contrib/vagrant
if [ ! -f "/home/vagrant/devstack/local.conf" ]; then
cp local.conf /home/vagrant/devstack/local.conf
fi
SCRIPT
config.vm.define "searchlight-ubuntu" do |ubuntu|
ubuntu.vm.box = "ubuntu/xenial64"
ubuntu.vm.network :private_network, ip: "192.168.27.100"
ubuntu.vm.provision :shell, :privileged => true, :inline => "DEBIAN_FRONTEND=noninteractive apt-get update"
ubuntu.vm.provision :shell, :privileged => true, :inline => "DEBIAN_FRONTEND=noninteractive apt-get install --yes git lvm2"
ubuntu.vm.provision :shell, :privileged => false, :inline => $script
end
config.vm.define "searchlight-fedora" do |fedora|
fedora.vm.box = "box-cutter/fedora20"
fedora.vm.network :private_network, ip: "192.168.27.101"
fedora.vm.provision :shell, :privileged => true, :inline => "yum update -y vim-minimal" # RH Bug 1066983
fedora.vm.provision :shell, :privileged => true, :inline => "yum install -y git-core MySQL-python"
fedora.vm.provision :shell, :privileged => false, :inline => $script
end
end

81
contrib/vagrant/local.conf
View File

@ -1,81 +0,0 @@
[[local|localrc]]
#########################
# General DevStack Config
# =======================
# IP Address for services to bind to (Should match IP from Vagrantfile)
HOST_IP=192.168.27.101
SERVICE_HOST=$HOST_IP
ADMIN_PASSWORD=devstack
MYSQL_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=$ADMIN_PASSWORD
# Logging
LOGFILE=$DEST/logs/stack.sh.log
VERBOSE=True
ENABLE_DEBUG_LOG_LEVEL=True
ENABLE_VERBOSE_LOG_LEVEL=True
# ElasticSearch version
# Currently, only ElasticSearch 2.x and 5.x are supported
ELASTICSEARCH_VERSION=5.6.11
# Enable Searchlight plugin
enable_plugin searchlight https://opendev.org/openstack/searchlight master
# Enable the basic services we require
enable_service rabbit mysql key
#############################
# Searchlight Devstack Config
# ===========================
# Enable Searchlight services
enable_service searchlight-api
enable_service searchlight-listener
#######################
# Other Devstack Config
# =====================
# Optional TLS Proxy
#enable_service tls-proxy
# Optional Tempest
#enable_service tempest
# Optional Rally
#enable_service rally
# Optional Horizon
#enable_service horizon
# Optional Glance
enable_service g-api g-reg
# Optional Nova
#enable_service n-api n-cpu n-net n-cond n-sch n-novnc
# Optional Neutron
#disable_service n-net
#enable_service q-svc q-agt q-dhcp q-l3 q-meta
[[post-config|$NOVA_CONF]]
[DEFAULT]
notification_driver=messagingv2
notification_topics=searchlight_indexer
notify_on_state_change=vm_and_task_state
notify_on_any_change=True
instance_usage_audit=True
instance_usage_audit_period=hour
[[post-config|$NEUTRON_CONF]]
[DEFAULT]
notification_driver=messagingv2
notification_topics=searchlight_indexer
[[post-config|$GLANCE_API_CONF]]
[DEFAULT]
notification_driver=messagingv2
notification_topics=searchlight_indexer
rpc_backend = 'rabbit'

80
devstack/README.rst
View File

@ -1,80 +0,0 @@
..
c) Copyright 2015 Hewlett-Packard Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
=================================
Enabling Searchlight in Devstack
=================================
1. Download DevStack (git clone)
2. Update local.conf
You may follow the customization instructions below or use the example
local.conf.
3. Run ``stack.sh``
.. note::
This installs a headless JRE. If you are working on a desktop based OS
(such as Ubuntu 14.04), this may cause tools like pycharms to no longer
launch. You can switch between JREs and back: to a headed JRE version using:
"sudo update-alternatives --config java".
Full example local.conf
=======================
The example `local.conf.example <local.conf.example>`_ MAY not be up to date
with the rest of devstack.
.. note::
You will need to look through the settings and potentially customize it to
your environment, especially ``HOST_IP``.
Existing local.conf customization
=================================
1. Add this repo as an external repository::
> cat local.conf
[[local|localrc]]
enable_plugin searchlight https://opendev.org/openstack/searchlight
enable_service searchlight-api
enable_service searchlight-listener
To use stable branches, make sure devstack is on that branch, and specify
the branch name to enable_plugin, for example::
enable_plugin searchlight https://opendev.org/openstack/searchlight stable/mitaka
2. Configure desired searchlight plugins
The search service is driven using a plugin mechanism for integrating to other
services. Each integrated service may need to be specifically enabled
in devstack and may require additional configuration settings to work with
searchlight. For example, you typically will need to set the notifications
driver in each service's configuration.
Please review the plugin documentation and add configuration appropriately:
* `Searchlight Plugins <https://docs.openstack.org/searchlight/latest/configuration/plugins.html>`_
3. Customize searchlight configuration
Searchlight documentation talks about settings in ``searchlight.conf``.
To customize searchlight.conf settings, add them under the following
section in ``local.conf``::
[[post-config|$SEARCHLIGHT_CONF]]

16
devstack/gate/gate_hook.sh
View File

@ -1,16 +0,0 @@
#!/bin/bash
set -ex
pushd $BASE/new/devstack
export KEEP_LOCALRC=1
export ENABLED_SERVICES=mysql,key,searchlight,searchlight-api
# Pass through any SEARCHLIGHT_ env vars to the localrc file
env | grep -E "^SEARCHLIGHT_" >> $BASE/new/devstack/local.conf || :
popd
# Run DevStack Gate
$BASE/new/devstack-gate/devstack-vm-gate.sh

39
devstack/gate/post_test_hook.sh
View File

@ -1,39 +0,0 @@
#!/bin/bash
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# This script is executed inside post_test_hook function in devstack gate.
# source $BASE/new/devstack/openrc admin admin
function generate_test_results {
if [ -f .testrepository/0 ]; then
sudo .tox/py27-gate/bin/testr last --subunit > $WORKSPACE/testrepository.subunit
sudo mv $WORKSPACE/testrepository.subunit $BASE/logs/testrepository.subunit
sudo .tox/py27-gate/bin/python /usr/local/jenkins/slave_scripts/subunit2html.py $BASE/logs/testrepository.subunit $BASE/logs/testr_results.html
sudo gzip -9 $BASE/logs/testrepository.subunit
sudo gzip -9 $BASE/logs/testr_results.html
sudo chown $USER:$USER $BASE/logs/testrepository.subunit.gz $BASE/logs/testr_results.html.gz
sudo chmod a+r $BASE/logs/testrepository.subunit.gz $BASE/logs/testr_results.html.gz
fi
}
set -x
export SEARCHLIGHT_DIR="$BASE/new/searchlight"
sudo chown -R stack:stack $SEARCHLIGHT_DIR
cd $SEARCHLIGHT_DIR
# Collect and parse result
generate_test_results
exit $EXIT_CODE

259
devstack/local.conf.example
View File

@ -1,259 +0,0 @@
[[local|localrc]]
LOGFILE=$DEST/logs/stack.sh.log
VERBOSE=TRUE
USE_PYTHON3=True
### NETWORK SETTINGS ###
#
# Change the FLOATING_RANGE to whatever IPs VM is working in.
# In NAT mode it is subnet VMWare Fusion provides, in bridged mode it is your local network.
# But only use the top end of the network by using a /27 and starting at the 224 octet.
#FLOATING_RANGE=192.168.1.224/27
#FIXED_RANGE=10.0.0.1/24
#FIXED_NETWORK_SIZE=256
#FLAT_INTERFACE=eth0
#EXT_GW_IP=192.168.122.1
HOST_IP=127.0.0.1
SERVICE_HOST=$HOST_IP
IMAGE_HOST=$HOST_IP
IDENTITY_HOST=$HOST_IP
### SET PASSWORDS FOR SERVICES AND USERS ###
#
ADMIN_PASSWORD=devstack
MYSQL_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=$ADMIN_PASSWORD
### SET SOME INSTALL OPTIONS ###
#
# run already-installed devstack in offline mode
# Set ``OFFLINE`` to ``True`` to configure ``stack.sh`` to run cleanly without
# Internet access. ``stack.sh`` must have been previously run with Internet
# access to install prerequisites and fetch repositories.
#OFFLINE=True
GIT_BASE=${GIT_BASE:-https://opendev.org}
# Reclone will ensure any repos already present are not re-cloned
#
# To review a patch to devstack itself, set this to yes, then:
# cd /opt/stack/searchlight
# git review -d <gerrit-id>
# <devstack>/unstack.sh
# <devstack>/stack.sh
RECLONE=no
# always upgrade all Python dependencies
PIP_UPGRADE=False
# Set libraries that will be installed from git and not PyPI
# e.g. python-searchlightclient (not yet supported)
#LIBS_FROM_GIT=python-searchlightclient
### ADD SOME IMAGES TO GLANCE ###
#
# latest Ubuntu Xenial amd64 cloud image
IMAGE_URLS+=",https://cloud-images.ubuntu.com/releases/xenial/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img"
# Fedora 21 cloud image (e.g. for AWS LoadBalancer resource in Heat)
#IMAGE_URLS+=",http://download.fedoraproject.org/pub/fedora/linux/releases/21/Cloud/Images/x86_64/Fedora-Cloud-Base-20141203-21.x86_64.qcow2"
# ElasticSearch version
# Currently, only ElasticSearch 2.x and 5.x are supported
ELASTICSEARCH_VERSION=5.6.11
### CONFIGURE INSTALLED SERVICES ###
#
# Default set of components installed (as of DevStack Juno) is:
# - Nova with Nova-network
# - Keystone
# - Glance
# - Cinder with LVM backend
# - Horizon
# The below changes it.
### HEAT ###
#
# enable Heat services
enable_service h-eng h-api h-api-cfn h-api-cw heat
# (ON REVIEW) Install image for Heat's integration tests
HEAT_TEST_IMAGE=True
### CINDER ###
#
# Do not securely rewrite cinder volumes on delete
#CINDER_SECURE_DELETE=False
### HORIZON ###
#
# disable Horizon (Dashboard)
#disable_service horizon
HORIZON_BRANCH=master
### NEUTRON ###
#
# disable Nova-network and enable Neutron
disable_service n-net
enable_service q-svc q-agt q-dhcp q-l3 q-meta q-metering q-lbaas q-fwaas neutron
### CEILOMETER ###
#
# metering services
#enable_service ceilometer-acompute ceilometer-acentral ceilometer-anotification ceilometer-collector
# alarming services
#enable_service ceilometer-alarm-evaluator,ceilometer-alarm-notifier
# api services
# enable_service ceilometer-api
# set shorter sample collection interval (default is 600)
#CEILOMETER_PIPELINE_INTERVAL=60
### SWIFT ###
#
enable_service s-proxy s-object s-container s-account
# set swift hash - the hash below is result of
# echo "SWIFT_HASH" | md5sum | awk '{print $1}'
SWIFT_HASH=096d08da4f8d4cce3a724c5f6c18f055
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data/swift
### DESIGNATE ###
#
enable_plugin designate https://opendev.org/openstack/designate
# Searchlight
#
enable_plugin searchlight https://opendev.org/openstack/searchlight
enable_service searchlight-api
enable_service searchlight-listener
enable_plugin searchlight-ui https://opendev.org/openstack/searchlight-ui
### POST CONFIG STAGE SETTINGS ###
#
# fix to allow access to instance vnc console when accessing Horizon via tunnel
[[post-config|$NOVA_CONF]]
[DEFAULT]
#novncproxy_base_url = http://172.18.200.23:6127/vnc_auto.html
[notifications]
notify_on_state_change = vm_and_task_state
[oslo_messaging_notifications]
driver = messagingv2
[[post-config|$NEUTRON_CONF]]
[oslo_messaging_notifications]
driver = messagingv2
# decrease number of Heat engine workers, when too much for devstack
[[post-config|$HEAT_CONF]]
[DEFAULT]
num_engine_workers = 2
[[post-config|$GLANCE_API_CONF]]
[oslo_messaging_notifications]
driver = messagingv2
[[post-config|$CINDER_CONF]]
[oslo_messaging_notifications]
driver = messagingv2
[[post-config|$DESIGNATE_CONF]]
[oslo_messaging_notifications]
driver = messagingv2
### Searchlight Configuration ###
[[post-config|$SEARCHLIGHT_CONF]]
[elasticsearch]
index_settings = number_of_shards:1,number_of_replicas:0
[manage]
#workers = 3
[listener]
#notifications_pool = searchlight-listener
[resource_plugin]
#notifications_topic = notifications
#resource_group_name = searchlight
[service_credentials:nova]
compute_api_version = 2.1
[resource_plugin:os_nova_server]
enabled = True
#admin_only_fields = OS-EXT-STS:vm_state
[resource_plugin:os_nova_hypervisor]
enabled = True
[resource_plugin:os_nova_flavor]
notifications_topics_exchanges = versioned_notifications,nova
enabled = True
[resource_plugin:os_nova_servergroup]
enabled = True
[resource_plugin:os_glance_image]
enabled = True
[resource_plugin:os_glance_metadef]
enabled = True
[resource_plugin:os_cinder_volume]
enabled = True
[resource_plugin:os_cinder_snapshot]
enabled = True
[resource_plugin:os_designate_zone]
enabled = True
[resource_plugin:os_designate_recordset]
enabled = True
[resource_plugin:os_neutron_net]
enabled = True
[resource_plugin:os_neutron_port]
enabled = True
[resource_plugin:os_neutron_floatingip]
enabled = True
[resource_plugin:os_neutron_security_group]
enabled = True
## SWIFT SETTINGS ###
# At this time we recommend that you manually enable the Searchlight plugins
# for Swift after devstack has completed stacking. Please follow the
# instructions in the docs:
#
# https://docs.openstack.org/searchlight/latest/admin/plugins/swift.html
#
# Notifications must be configured properly for searchlight to process
# incremental updates. There is a middleware patch to provide Swift updates.
# Please see the plugin guide for swift for more information.
[resource_plugin:os_swift_account]
enabled = False
#Specify same value as in swift proxy config for reseller_prefix
reseller_prefix = AUTH_
[resource_plugin:os_swift_container]
enabled = False
[resource_plugin:os_swift_object]
enabled = False

417
devstack/plugin.sh
View File

@ -1,417 +0,0 @@
# Install and start **Searchlight** service
# To enable Searchlight services, add the following to localrc
# enable_plugin searchlight https://opendev.org/openstack/searchlight
# enable_service searchlight-api
# enable_service searchlight-listener
# stack.sh
# ---------
# install_searchlight
# configure_searchlight
# init_searchlight
# start_searchlight
# stop_searchlight
# cleanup_searchlight
# Save trace setting
XTRACE=$(set +o | grep xtrace)
set +o xtrace
# Defaults
# --------
# Set up default repos
SEARCHLIGHT_REPO=${SEARCHLIGHT_REPO:-${GIT_BASE}/openstack/searchlight.git}
SEARCHLIGHT_BRANCH=${SEARCHLIGHT_BRANCH:-master}
# Set up default paths
SEARCHLIGHT_BIN_DIR=$(get_python_exec_prefix)
SEARCHLIGHT_DIR=$DEST/searchlight
SEARCHLIGHT_CONF_DIR=/etc/searchlight
SEARCHLIGHT_STATE_PATH=${SEARCHLIGHT_STATE_PATH:=$DATA_DIR/searchlight}
SEARCHLIGHT_CONF=$SEARCHLIGHT_CONF_DIR/searchlight.conf
SEARCHLIGHT_LOG_DIR=/var/log/searchlight
SEARCHLIGHT_AUTH_CACHE_DIR=${SEARCHLIGHT_AUTH_CACHE_DIR:-/var/cache/searchlight}
SEARCHLIGHT_APIPASTE_CONF=$SEARCHLIGHT_CONF_DIR/api-paste.ini
SEARCHLIGHT_UWSGI_CONF=$SEARCHLIGHT_CONF_DIR/searchlight-uwsgi.ini
SEARCHLIGHT_UWSGI_APP=$SEARCHLIGHT_BIN_DIR/searchlight-api-wsgi
if is_service_enabled tls-proxy; then
SEARCHLIGHT_SERVICE_PROTOCOL="https"
fi
# Public IP/Port Settings
SEARCHLIGHT_SERVICE_PROTOCOL=${SEARCHLIGHT_SERVICE_PROTOCOL:-$SERVICE_PROTOCOL}
SEARCHLIGHT_SERVICE_HOST=${SEARCHLIGHT_SERVICE_HOST:-$SERVICE_HOST}
SEARCHLIGHT_SERVICE_PORT=${SEARCHLIGHT_SERVICE_PORT:-9393}
SEARCHLIGHT_SERVICE_PORT_INT=${SEARCHLIGHT_SERVICE_PORT_INT:-19393}
ELASTICSEARCH_VERSION=${ELASTICSEARCH_VERSION:-2.3.4}
# Base URL for ElasticSearch 5.x and 6.x
ELASTICSEARCH_BASEURL=https://artifacts.elastic.co/downloads/elasticsearch
# Base URL for ElasticSearch 2.x
ELASTICSEARCH_BASEURL_LEGACY=https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution
# Helper Functions
# ----------------
function setup_colorized_logging_searchlight {
local conf_file=$1
local conf_section=$2
local project_var=${3:-"project_name"}
local user_var=${4:-"user_name"}
setup_colorized_logging $conf_file $conf_section $project_var $user_var
# Override the logging_context_format_string value chosen by
# setup_colorized_logging.
iniset $conf_file $conf_section logging_context_format_string "%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [%(request_id)s %(user_identity)s%(color)s] %(instance)s%(color)s%(message)s"
}
# DevStack Plugin
# ---------------
# cleanup_searchlight - Remove residual data files, anything left over from previous
# runs that a clean run would need to clean up
function cleanup_searchlight {
_stop_elasticsearch
sudo rm -rf $SEARCHLIGHT_STATE_PATH $SEARCHLIGHT_AUTH_CACHE_DIR
sudo rm -f $(apache_site_config_for searchlight_api)
}
# configure_searchlight - Set config files, create data dirs, etc
function configure_searchlight {
[ ! -d $SEARCHLIGHT_CONF_DIR ] && sudo mkdir -m 755 -p $SEARCHLIGHT_CONF_DIR
sudo chown $STACK_USER $SEARCHLIGHT_CONF_DIR
[ ! -d $SEARCHLIGHT_LOG_DIR ] && sudo mkdir -m 755 -p $SEARCHLIGHT_LOG_DIR
sudo chown $STACK_USER $SEARCHLIGHT_LOG_DIR
# (Re)create ``searchlight.conf``
rm -f $SEARCHLIGHT_CONF
# General Configuration
iniset_rpc_backend searchlight $SEARCHLIGHT_CONF DEFAULT
iniset $SEARCHLIGHT_CONF DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL
iniset $SEARCHLIGHT_CONF DEFAULT state_path $SEARCHLIGHT_STATE_PATH
# API Configuration
sudo cp $SEARCHLIGHT_DIR/etc/api-paste.ini $SEARCHLIGHT_APIPASTE_CONF
iniset $SEARCHLIGHT_CONF api public_endpoint $SEARCHLIGHT_SERVICE_PROTOCOL://$SEARCHLIGHT_SERVICE_HOST/search
# OpenStack users
iniset $SEARCHLIGHT_CONF service_credentials auth_type password
iniset $SEARCHLIGHT_CONF service_credentials username searchlight
iniset $SEARCHLIGHT_CONF service_credentials user_domain_id default
iniset $SEARCHLIGHT_CONF service_credentials project_domain_id default
iniset $SEARCHLIGHT_CONF service_credentials password $SERVICE_PASSWORD
iniset $SEARCHLIGHT_CONF service_credentials project_name $SERVICE_PROJECT_NAME
iniset $SEARCHLIGHT_CONF service_credentials auth_url $KEYSTONE_SERVICE_URI
iniset $SEARCHLIGHT_CONF service_credentials os_region_name $REGION_NAME
# Keystone Middleware
iniset $SEARCHLIGHT_CONF paste_deploy flavor keystone
configure_auth_token_middleware $SEARCHLIGHT_CONF searchlight $SEARCHLIGHT_AUTH_CACHE_DIR
# Oslo Concurrency
iniset $SEARCHLIGHT_CONF oslo_concurrency lock_path "$SEARCHLIGHT_STATE_PATH"
# TLS Proxy Configuration
if is_service_enabled tls-proxy; then
# Set the service port for a proxy to take the original
iniset $SEARCHLIGHT_CONF api bind_port $SEARCHLIGHT_SERVICE_PORT_INT
else
iniset $SEARCHLIGHT_CONF api bind_port $SEARCHLIGHT_SERVICE_PORT
fi
# Logging Configuration
if [ "$SYSLOG" != "False" ]; then
iniset $SEARCHLIGHT_CONF DEFAULT use_syslog True
fi
# Format logging
if [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ]; then
setup_colorized_logging_searchlight $SEARCHLIGHT_CONF DEFAULT "tenant" "user"
fi
# Plugin config - disable designate by default since it's not typically installed
iniset $SEARCHLIGHT_CONF resource_plugin:os_designate_zone enabled False
iniset $SEARCHLIGHT_CONF resource_plugin:os_designate_recordset enabled False
# Plugin config - disable ironic by default since it's not typically installed
iniset $SEARCHLIGHT_CONF resource_plugin:os_ironic_chassis enabled False
iniset $SEARCHLIGHT_CONF resource_plugin:os_ironic_chassis notifications_topics_exchanges ironic_versioned_notifications,ironic
iniset $SEARCHLIGHT_CONF resource_plugin:os_ironic_node enabled False
iniset $SEARCHLIGHT_CONF resource_plugin:os_ironic_node notifications_topics_exchanges ironic_versioned_notifications,ironic
iniset $SEARCHLIGHT_CONF resource_plugin:os_ironic_port enabled False
iniset $SEARCHLIGHT_CONF resource_plugin:os_ironic_port notifications_topics_exchanges ironic_versioned_notifications,ironic
# Plugin config - enable versioned notifications for flavor
iniset $SEARCHLIGHT_CONF resource_plugin:os_nova_flavor enabled True
iniset $SEARCHLIGHT_CONF resource_plugin:os_nova_flavor notifications_topics_exchanges versioned_notifications,nova
iniset $SEARCHLIGHT_CONF resource_plugin:os_nova_server enabled True
iniset $SEARCHLIGHT_CONF resource_plugin:os_nova_server notifications_topics_exchanges versioned_notifications,nova
iniset $SEARCHLIGHT_CONF resource_plugin:os_nova_server use_versioned_notifications True
# Plugin config - disable swift by default since it's not typically installed
iniset $SEARCHLIGHT_CONF resource_plugin:os_swift_account enabled False
iniset $SEARCHLIGHT_CONF resource_plugin:os_swift_container enabled False
iniset $SEARCHLIGHT_CONF resource_plugin:os_swift_object enabled False
# uWSGI configuration
write_uwsgi_config "$SEARCHLIGHT_UWSGI_CONF" "$SEARCHLIGHT_UWSGI_APP" "/search"
}
# create_searchlight_accounts - Set up common required searchlight accounts
# Tenant User Roles
# ------------------------------------------------------------------
# service searchlight admin # if enabled
function create_searchlight_accounts {
if [[ "$ENABLED_SERVICES" =~ "searchlight-" ]]; then
create_service_user "searchlight" "admin"
if is_service_enabled searchlight-api; then
get_or_create_service "searchlight" "search" "Searchlight Service"
get_or_create_endpoint "search" \
"$REGION_NAME" \
"$SEARCHLIGHT_SERVICE_PROTOCOL://$SEARCHLIGHT_SERVICE_HOST/search" \
"$SEARCHLIGHT_SERVICE_PROTOCOL://$SEARCHLIGHT_SERVICE_HOST/search" \
"$SEARCHLIGHT_SERVICE_PROTOCOL://$SEARCHLIGHT_SERVICE_HOST/search"
fi
fi
}
# init_searchlight - Initialize etc.
function init_searchlight {
# Create cache dir
sudo mkdir -p $SEARCHLIGHT_AUTH_CACHE_DIR
sudo chown $STACK_USER $SEARCHLIGHT_AUTH_CACHE_DIR
rm -f $SEARCHLIGHT_AUTH_CACHE_DIR/*
_start_elasticsearch
$SEARCHLIGHT_BIN_DIR/searchlight-manage --config-file $SEARCHLIGHT_CONF index sync --force
}
# Install Searchlight's requirements
# See https://elasticsearch-py.readthedocs.io/en/master/#compatibility
function _setup_searchlight_dev {
setup_develop $SEARCHLIGHT_DIR
if [[ $ELASTICSEARCH_VERSION =~ ^5 ]]; then
echo "Installing python elasticsearch for ES 5.x"
$REQUIREMENTS_DIR/.venv/bin/edit-constraints $REQUIREMENTS_DIR/upper-constraints.txt elasticsearch
pip_install -U -r $SEARCHLIGHT_DIR/elasticsearch5.txt
elif [[ $ELASTICSEARCH_VERSION =~ ^6 ]]; then
echo "WARNING - Searchlight is not tested with ES 6.x!!!"
# echo "Installing python elasticsearch for ES 6.x"
# $REQUIREMENTS_DIR/.venv/bin/edit-constraints $REQUIREMENTS_DIR/upper-constraints.txt eleasticsearch
# pip install -c $REQUIREMENTS_DIR/upper-constraints.txt -U -r $SEARCHLIGHT_DIR/elasticsearch6.txt
fi
}
# install_searchlight - Collect source and prepare
function install_searchlight {
git_clone $SEARCHLIGHT_REPO $SEARCHLIGHT_DIR $SEARCHLIGHT_BRANCH
_setup_searchlight_dev
_download_elasticsearch
_install_elasticsearch
pip_install uwsgi
}
# install_searchlightclient - Collect source and prepare
function install_searchlightclient {
git_clone $SEARCHLIGHTCLIENT_REPO $SEARCHLIGHTCLIENT_DIR $SEARCHLIGHTCLIENT_BRANCH
setup_develop $SEARCHLIGHTCLIENT_DIR
}
# start_searchlight - Start running processes, including screen
function start_searchlight {
if is_service_enabled searchlight-api; then
run_process searchlight-api "$SEARCHLIGHT_BIN_DIR/uwsgi --ini $SEARCHLIGHT_UWSGI_CONF"
fi
if is_service_enabled searchlight-listener; then
run_process searchlight-listener "$SEARCHLIGHT_BIN_DIR/searchlight-listener --config-file $SEARCHLIGHT_CONF"
fi
}
# stop_searchlight - Stop running processes
function stop_searchlight {
# Kill the searchlight screen windows
stop_process searchlight-api
stop_process searchlight-listener
remove_uwsgi_config "$SEARCHLIGHT_UWSGI_CONF" "$SEARCHLIGHT_UWSGI_APP"
}
###############
# ELASTICSEARCH
# Moving this here because the devstack team has determined that only
# services supporting devstack core projects should live in devstack
function _wget_elasticsearch {
local baseurl=${1}
local file=${2}
if [ ! -f ${FILES}/${file} ]; then
wget ${baseurl}/${file} -O ${FILES}/${file}
fi
if [ ! -f ${FILES}/${file}.sha1 ]; then
# Starting with 2.0.0, sha1 files dropped the .txt extension and changed
# the format slightly; need the leading spaces to comply with sha1sum
( wget ${baseurl}/${file}.sha1 -O ${FILES}/${file}.sha1 &&
echo " ${file}" >> ${FILES}/${file}.sha1 )
fi
pushd ${FILES}; sha1sum ${file} > ${file}.sha1.gen; popd
if ! diff ${FILES}/${file}.sha1.gen ${FILES}/${file}.sha1; then
echo "Invalid elasticsearch download. Could not install."
return 1
else
echo "SHA1 for ${file} matches downloaded ${FILES}/${file}.sha1"
fi
return 0
}
function _download_elasticsearch {
if is_ubuntu; then
arch="deb"
elif is_fedora; then
arch="rpm"
else
echo "Unknown architecture; can't download ElasticSearch"
fi
ELASTICSEARCH_FILENAME=elasticsearch-${ELASTICSEARCH_VERSION}.${arch}
if [[ $ELASTICSEARCH_VERSION =~ ^2 ]]; then
ELASTICSEARCH_URL=${ELASTICSEARCH_BASEURL_LEGACY}/${arch}/elasticsearch/${ELASTICSEARCH_VERSION}
elif [[ $ELASTICSEARCH_VERSION =~ ^5 ]]; then
ELASTICSEARCH_URL=${ELASTICSEARCH_BASEURL}
else
echo "Current Searchlight only supports ElasticSearch 2.x and 5.x"
fi
echo "Downloading ElasticSearch $ELASTICSEARCH_VERSION"
echo "ElasticSearch URL is $ELASTICSEARCH_URL"
_wget_elasticsearch $ELASTICSEARCH_URL $ELASTICSEARCH_FILENAME
}
function _check_elasticsearch_ready {
# poll elasticsearch to see if it's started
if ! wait_for_service 30 http://localhost:9200; then
die $LINENO "Maximum timeout reached. Could not connect to ElasticSearch"
fi
}
function _start_elasticsearch {
echo "Starting elasticsearch"
if is_ubuntu; then
sudo /etc/init.d/elasticsearch start
_check_elasticsearch_ready
elif is_fedora; then
sudo /bin/systemctl start elasticsearch.service
_check_elasticsearch_ready
else
echo "Unsupported architecture... Can not start elasticsearch."
fi
}
function _stop_elasticsearch {
echo "Stopping elasticsearch"
if is_ubuntu; then
sudo /etc/init.d/elasticsearch stop
elif is_fedora; then
sudo /bin/systemctl stop elasticsearch.service
else
echo "Unsupported architecture...can not stop elasticsearch."
fi
}
function _install_elasticsearch {
# echo "Installing elasticsearch"
# pip_install_gr elasticsearch
if is_package_installed elasticsearch; then
echo "Note: elasticsearch was already installed."
return
fi
if is_ubuntu; then
if [[ ${DISTRO} == "bionic" ]]; then
is_package_installed openjdk-8-jdk-headless || install_package openjdk-8-jdk-headless
else
is_package_installed default-jdk-headless || install_package default-jdk-headless
fi
sudo dpkg -i ${FILES}/elasticsearch-${ELASTICSEARCH_VERSION}.deb
sudo update-rc.d elasticsearch defaults 95 10
elif is_fedora; then
is_package_installed java-1.8.0-openjdk-headless || install_package java-1.8.0-openjdk-headless
yum_install ${FILES}/elasticsearch-${ELASTICSEARCH_VERSION}.rpm
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
else
echo "Unsupported install of elasticsearch on this architecture."
fi
}
function _uninstall_elasticsearch {
echo "Uninstalling elasticsearch"
if is_package_installed elasticsearch; then
if is_ubuntu; then
sudo apt-get purge elasticsearch
elif is_fedora; then
sudo yum remove elasticsearch
else
echo "Unsupported install of elasticsearch on this architecture."
fi
fi
}
#
# END OF ELASTICSEARCH
######################
# check for service enabled
if is_service_enabled searchlight; then
if [[ "$1" == "source" ]]; then
# Initial source of lib script
source $TOP_DIR/lib/searchlight
fi
if [[ "$1" == "stack" && "$2" == "install" ]]; then
echo_summary "Installing Searchlight"
install_searchlight
echo_summary "Installing Searchlight client"
install_searchlightclient
elif [[ "$1" == "stack" && "$2" == "post-config" ]]; then
echo_summary "Configuring Searchlight"
configure_searchlight
if is_service_enabled key; then
echo_summary "Creating Searchlight Keystone Accounts"
create_searchlight_accounts
fi
elif [[ "$1" == "stack" && "$2" == "extra" ]]; then
echo_summary "Initializing Searchlight"
init_searchlight
echo_summary "Starting Searchlight"
start_searchlight
fi
if [[ "$1" == "unstack" ]]; then
stop_searchlight
fi
if [[ "$1" == "clean" ]]; then
echo_summary "Cleaning Searchlight"
cleanup_searchlight
fi
fi
# Restore xtrace
$XTRACE

9
devstack/settings
View File

@ -1,9 +0,0 @@
# Devstack settings
#Searchlight client settings
SEARCHLIGHTCLIENT_DIR=$DEST/python-searchlightclient
SEARCHLIGHTCLIENT_REPO=${SEARCHLIGHTCLIENT_REPO:-${GIT_BASE}/openstack/python-searchlightclient.git}
SEARCHLIGHTCLIENT_BRANCH=${SEARCHLIGHTCLIENT_BRANCH:-master}
# we have to add searchlight to enabled services for screen_it to work
enable_service searchlight

6
doc/requirements.txt
View File

@ -1,6 +0,0 @@
# Documentation
sphinx>=2.0.0,!=2.1.0 # BSD
openstackdocstheme>=2.2.1 # Apache-2.0
os-api-ref>=1.4.0 # Apache-2.0
reno>=3.1.0 # Apache-2.0
sphinxcontrib-svg2pdfconverter>=0.1.0 # BSD

347
doc/source/admin/architecture.rst
View File

@ -1,347 +0,0 @@
..
Copyright (c) 2016 Hewlett Packard Enterprise Development LP
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Searchlight Architecture
========================
The OpenStack Search Service
----------------------------
Searchlight is a microservice based architecture that provides advanced and
multi-tenant searching capabilities across multiple cloud services through
indexing into Elasticsearch. Once indexed, users are able to search through
all internal data associated with an OpenStack service through a single API
with a consistent interface. These searches can be quite complex, are blazingly
fast, and easily paginated. Without Searchlight it can be necessary to make
calls to multiple services to retrieve resource information.
All documents in Elasticsearch are protected by Searchlight with role-based
access control (RBAC) for data security and protection. RBAC protection
prevents unauthorized users from accessing data that they are not permitted to
view. Searchlight uses RBAC in three ways to protect the data:
1. **Network Restriction:** Elasticsearch is not directly accessible. It
resides on a restricted network behind Searchlight. The only way to access
the data within Elasticsearch is indirectly through the Searchlight.
API endpoints. Users need to be authenticated to access these endpoints.
2. **Document Separation:** Each resource is indexed twice in Elasticsearch.
Once for the admin user and once for all others. The document associated
with the admin has all fields present. The document associated with
non-admins has all admin-only fields removed. This model prevents a
non-admin from viewing any admin-only fields since these fields are not
present in the document at all.
3. **Pre-Query Protection:** Searchlight will inject RBAC-based protection
into all queries before they get sent to Elasticsearch. In general the
pre-query RBAC injection will provide gross level protection. For example,
entire projects that the user does not have access to. See Figure 2.
4. **Post-Query Protection:** Searchlight uses the RBAC-based protection to
filter the response to each query in order to remove any sensitive data
before returning the results to the eagerly waiting user. In general the
post-query RBAC filtering will provide fine level protection. For example,
specific fields within a document. See Figure 2.
The functionality provided by Searchlight differs greatly from the logging
capabilities of OpenStack. The logging services store predetermined messages
from the OpenStack services, as opposed to the dynamic, internal state of the
OpenStack services.
Primary Users
-------------
The primary users are Horizon-based. This includes users accessing their
cloud projects ("tenants") and cloud administrators (IT operations).
Searchlight Components
----------------------
Searchlight consists of multiple components. The components live in one of two
environments: externally on the end-user's system or internally on an
OpenStack deployed system. Figure 1 illustrates how all of these components
interact.
**Figure 1: Architecture Diagram**
.. image:: ../../../images/Figure1.png
Figure 1 shows the relationship between all of the Searchlight components. The
external Searchlight components are colored light-grey. The internal
Searchlight components are colored peach. The internal non-Searchlight
components are brightly colored, as are the arrows indicating the interfaces
for these components.
* **Searchlight UI:** A plugin for Horizon that provides search support. Lives
externally on the end-user's system.
* **Searchlight User CLI (OpenStack Client plugin):** Command-line interface
to the Searchlight API. Lives externally on the end-user's system.
* **Searchlight API:** OpenStack Searchlight RESTful API. A separate process
running a WSGI web server on top of the python Searchlight framework. All
external access (for example, Horizon, browser, and CLI) go through the
Searchlight API. The Searchlight API performs read-only accesses of
Elasticsearch.
* **HAProxy:** High availability load balancer and proxy server for the
Searchlight API component. HAProxy enables multiple instances of the
Searchlight API to run and allows for better performance, scalability, and
reliability of the Searchlight service.
* **Searchlight Listener:** A separate Searchlight process that is plugged
into the notification MQ for the OpenStack services. Whenever there is a
change in state for a resource type in an OpenStack service, a notification
is sent to the Searchlight Listener. This results in the OpenStack service
state being indexed into Elasticsearch as a document related to that
resource type (for example, OS::Nova::Server). The listener is never accessed
directly by any of the other Searchlight components.
* **Searchlight Admin CLI (searchlight-manage):** A separate process that
provides admin support for Searchlight. The main functionality is
re-indexing support. Connects to various OpenStack services and loads data
from them into the Elasticsearch indices. The admin needs to log in directly
to a Searchlight node and run the Searchlight Admin CLI from that node. The
Searchlight Admin CLI performs read/write accesses to Elasticsearch. Lives
internally on an OpenStack deployed system.
* **Elasticsearch:** A database service used to index information about
selected services. Elasticsearch is meant to be shared by multiple OpenStack
projects, not just Searchlight. This implies that there may be multiple
indices within Elasticsearch. All Elasticsearch indices used by Searchlight
are created in a way so as to be unique to Searchlight. This will prevent
any duplication between Searchlight and other OpenStack projects that may
be sharing Elasticsearch. Lives internally on an OpenStack deployed system.
* **RabbitMQ:** Messaging Queue used for accessing the OpenStack's OSLO
notification MQ. Each OpenStack service will place notifications for any
state change in RabbitMQ. The Searchlight listener will pull this
notifications from RabbitMQ and process them. Lives internally on an
OpenStack deployed system. All access of the notification MQ is done
through OSLO messaging.
* **OpenStack Services:** Various OpenStack Services from which Searchlight
gathers information through internal plugins. The current list of supported
OpenStack services are: Cinder, Designate, Glance, Neutron, Nova, Swift.
Searchlight accesses the services as an admin, not as a regular user.
This admin access is required for the extra privileges that are needed,
for example, to access objects across projects ("tenants"). The credentials
for the admin are stored in the config files.
* **OpenStack Service Plugins:** Each OpenStack service mentioned above has
one or more associated plugins. These plugins live in Searchlight and not in
the OpenStack service. The plugins are based on a resource type and not on
the OpenStack service. For example, the Cinder Service has two resource types
used by Searchlight: OS::Cinder::Volume and OS::Cinder::Snapshot. This
results in two plugins for the Cinder service. The architecture is the same
for each plugin. The plugins are not a separate process. The plugins live
within the Searchlight listener and the Searchlight Admin CLI components.
The numbered arrows represent the interfaces between the components. These
interfaces are fully described in Table 1.
Searchlight Eco-System
----------------------
The Searchlight Eco-system is illuminated by Figure 2. We will step through
each stage of the Searchlight life-cycle to further explore how the various
components interact with each other.
**Deployment Stage:** This stage is when Searchlight gets instantiated by the
admin dev ops. Even though the steps are mentioned as if they are manual steps,
they are all done within a deployment process. More deployment stage options
are discussed in Figure 4.
1. The OpenStack services (for example, Nova, Neutron) are started. These
services are used by Searchlight, but are not a part of the Searchlight
Eco-system. The admin will set up the configuration files for all services
that are used by Searchlight, including enabling the notifications that will
later be consumed by Searchlight.
2. Elasticsearch is started. This analytics engine is used by Searchlight, but
it is not a part of the Searchlight Eco-system. As a note, other services
may also be using Elasticsearch.
3. The admin sets up the Searchlight configuration file.
4. The initial indexing for Searchlight occurs (component 2 in Figure 2). This
happens when the admin runs the command ``searchlight-manage index sync`` on
a Searchlight node. Searchlight will pull the data from the OpenStack
service's API and index it into Elasticsearch.
5. The Searchlight listeners are started (component 3 in Figure 2). This
happens when the admin runs the command ``searchlight-listener`` At this
point, Searchlight is notified of any state changes that occurs in the
services. The OpenStack services will push the data to Searchlight, which
in turn will index that data into Elasticsearch.
6. The Searchlight API service is started (component 1 in Figure 2). This
happens when the admin runs the command ``searchlight-api`` At this point,
the Searchlight API is live and anyone can start making queries to
Searchlight.
**Running Stage (automatic):** This stage consumes the vast majority of the
Searchlight life cycle. The running stage is further highlighted in Figure 3.
1. The Searchlight listeners are actively receiving notifications from the
OpenStack services whenever there is a state change.
2. The Searchlight API service is actively receiving queries from Horizon apps
and users.
**Maintenance Stage (manual):** This stage occurs when the admin needs to
analyze the state of Searchlight Eco-system or make corrections to it.
1. To verify there are no orphaned Elasticsearch indices, the admin can view
the Elasticsearch indices being used by Searchlight using the
``searchlight-manage`` command. Any orphaned indices will be taken
care of through Elasticsearch and not through Searchlight.
See :ref:`ES-Index-Cleanup`.
2. If the Elasticsearch data is no longer coherent with the OpenStack
services, the admin will need to re-index Elasticsearch. This is done by
running the command ``searchlight-manage`` similarly to step 4 in
the deployment stage. See :ref:`ES-Bulk-Indexing`.
**Figure 2: Searchlight Eco-System**
.. image:: ../../../images/Figure2.png
Searchlight Flow
----------------
The main portion of the Eco-system, the running stage, is showcased in Figure
3. The diagram is from the viewpoint of the users of the Eco-system.
The clients will access the OpenStack services. This could include issuing
commands to create a new server, delete a volume or modify a networking subnet.
The commands can be sent directly to the OpenStack service's API or indirectly
through the Horizon dashboard. Once these commands are executed by the
OpenStack services, the OpenStack services will notify Searchlight of the
resulting state changes.
Simultaneously, the user can make queries of Searchlight to better understand
the entire OpenStack Eco-system. These queries can be made directly through the
Searchlight API or indirectly through the Horizon dashboard.
**Figure 3: Searchlight Flow**
.. image:: ../../../images/Figure3.png
Searchlight Scaling
-------------------
When deploying Searchlight, the Eco-system is scaled in multiple ways. This
scaling is done for both performance and reliability. Both the Searchlight API
and the Searchlight listener are stateless microservices. This makes it easier
to scale them. Figure 4 shows this in glorious color.
Searchlight API Services:
* Multiple instances of the Searchlight API service can be deployed and placed
behind a load balancer. The load balancer hides the details of the
Searchlight API services from the user. All user API access will go through
the load balancer. In addition, the WSGI web service has multiple threads
handling all requests.
Searchlight Listeners:
* There are multiple notification workers that are receiving messages from the
OpenStack services. The number of workers can be statically specified in the
Searchlight configuration file. In addition, each type of resource indexed by
Searchlight can be scaled independently by spawning different processes with
different configuration files.
Elasticsearch:
* Elasticsearch is deployed with multiple nodes in a cluster. Each cluster can
also have multiple shards and replicas. The number of nodes, shards and
replicas are specified in the Elasticsearch configuration file. But the
number of shards and replicas can be overwritten on a per-index basis by
Searchlight. See :ref:`Indexing-Model`
**Figure 4: Searchlight Scaling**
.. image:: ../../../images/Figure4.png
Component Interfaces
--------------------
Tables 1 and 2 provide more details with the interfaces in Figure 1. These
details are geared toward understanding the security model and analyzing
potential threats for Searchlight.
**Table 1: Interfaces**
========= ================ ===================== ==================== =================== ====================== ================= =================== ==================== =====================================
Interface Network protocol Requestor Request Request credentials Request authorization Listener Response Response credentials Description of operation
========= ================ ===================== ==================== =================== ====================== ================= =================== ==================== =====================================
1 HTTPS Searchlight UI Horizon Searches Session token Admin or User Searchlight API Search results TLS Certificate Search query from Horizon.
1 HTTPS Searchlight User CLI User Searches Session token Admin or User Searchlight API Search results TLS Certificate Search query form user.
2 HTTP Searchlight API Query None None Elasticsearch Query Results None Elasticsearch access to
query data.
2 HTTP Searchlight Listener Query/Index None None Elasticsearch Query/Index Results None Elasticsearch access to
re-index or query data.
2 HTTP Searchlight Admin CLI Query/Index None None Elasticsearch Query/Index Results None Elasticsearch access to
re-index or query data.
3 HTTPS Searchlight Listener Service State Update Session token Admin OpenStack Service State Changes TLS Certificate Querying Service state
changes.
3 HTTPS Searchlight Admin CLI Service State Update Session token Admin OpenStack Service State Changes TLS Certificate Querying Service state
changes.
4 AMQP Service Connect to MQ Service's MQ Service MQ Account RabbitMQ State Changes TLS Certificate Service connects to MQ and
pushes state changes.
4 AMQP Searchlight Listener Connect to MQ Searchlight's MQ Searchlight MQ Account RabbitMQ State Changes TLS Certificate Searchlight Listener connects
to MQ to listen for state
changes pushed by OpenStack services.
========= ================ ===================== ==================== =================== ====================== ================= =================== ==================== =====================================
**Table 2: Default Network Ports**
============ ========= ==========
Port / Range Protocol Notes
============ ========= ==========
5671 AMQPS Messaging protocol
9200 HTTP Elasticsearch protocol
9393 HTTPS Searchlight API access
============ ========= ==========
Security Controls
-----------------
Searchlight's security and threat analysis can be viewed through multiple control points.
* **Audit:** Searchlight performs logging and sends logs to the Logstash
central logging service.
* **Authentication:** Searchlight requires a valid domain, username and
password to access the services. Keystone handles authentication and
authorization via session tokens. Note: No authentication is used for
Elasticsearch.
* **Authorization:** Searchlight provides access to user information.
Authentication and authorization is handled by Keystone.
* **Availability:** Searchlight uses HAProxy. The users can deploy
Elasticsearch in a high-availability mode if needed.
* **Confidentiality:** Searchlight API is configured to use TLS, traffic
between hosts is communicated using TLS. Data and config files protected
via filesystem controls. Note: No TLS between Searchlight and Elasticsearch.
* **Integrity:** Data is stored in Elasticsearch, but not backed up. The data
is always accessible from the OpenStack services themselves. If data in
Elasticsearch ever gets corrupted or out of sync, a resync will restore the
data integrity.
Dependencies
------------
Searchlight has dependencies on other components. The OpenStack services are
"soft" dependencies. The dependency exists only if the plugin for that particular
service is enabled.
* Elasticsearch
* RabbitMQ
* HAProxy
* Cinder Service
* Designate Service
* Glance Service
* Neutron Service
* Nova Service
* Swift Service

11
doc/source/admin/index.rst
View File

@ -1,11 +0,0 @@
======================
Administration guide
======================
.. toctree::
:maxdepth: 2
:glob:
architecture
indexingservice
plugins/*

330
doc/source/admin/indexingservice.rst
View File

@ -1,330 +0,0 @@
..
Copyright (c) 2015 Hewlett-Packard Development Company, L.P.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Searchlight Indexing
====================
In order for the Searchlight API service to return results, information
must be indexed. The two primary mechanisms by which this happens are indexing
from the source (which allows a complete index rebuild) and incrementally
updating the index based on information received via notifications.
The information indexed is determined by a plugin model.
Search plugins
--------------
The search service determines the type of information that is indexed and
searchable via a plugin mechanism.
See :ref:`searchlight-plugins` for plugin installation and general
configuration information.
See each plugin below for detailed information about specific plugins:
.. toctree::
:maxdepth: 1
:glob:
plugins/*
.. _Indexing-Model:
Indexing model
--------------
The Mitaka Searchlight release introduced the ability to continue executing
search requests while reindexing operations are running. This feature is called
*zero-downtime reindexing*. In order to implement zero-downtime indexing, the
concept of a *resource group* was introduced.
A *resource group* is a collection of plugins that share an Elasticsearch
index. Since each plugin represents a *resource type*, you can think of a
resource group as a collection of resource types.
For each resource group, Searchlight creates an index whose name consists of
the resource group name appended with a timestamp. Each resource group is
referred to by a pair of Elasticsearch aliases_. One alias is used for
searching by the API (the *search alias*), and the other (the *listener alias*)
is used to index incoming events.
During reindexing, a new index is created, and the listener alias is pointed at
both the old and new indices. Incoming events are therefore indexed into both
indices. The search alias is left pointing at the old index. Once indexing is
finished, both aliases are pointed solely at the new index and the old index
is deleted.
In order to improve the performance of reindexing, index refresh of the new
index is disabled during reindexing, and turned on after reindexing is done.
As a consequence, Documents synced to the new index are not searchable until
index is refreshed, but document retrieval by IDs still works, because GET
operation in Elasticsearch is realtime.
It is important to note that zero-downtime reindexing requires that **all**
plugins in a resource group are indexed together. When it's desired to index an
individual resource type, an optimization copies existing data directly from
the old index to the new one to avoid re-harvesting the data from each service
API.
.. note::
Due to some limitations discovered during the Mitaka release, indexing into
multiple indices (multiple plugin resource groups) is disabled. The newton release
implemented full support for specifying different resource groups for different
resource types.
.. _using-resource-groups:
Sorting across resource groups
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Using multiple resource groups will impact sort behavior when sorting on fields
across resource types when all the resource types don't have the requested 'sort-by field'.
Follow the guidelines below to avoid errors:
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_missing_values
.. _aliases: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html
.. _ES-Bulk-Indexing:
Bulk indexing
-------------
To initially create the catalog index (or add new resource typs to it later),
run the following command::
$ searchlight-manage index sync
This will iterate through all registered and enabled search plugins and
request that they perform a full indexing of all data that's available to them.
It is also possible to index just a single resource, or all resources
belonging to a resource group. For instance, to index all glance images::
$ searchlight-manage index sync --type OS::Glance::Image
As described above, this will create a new index for all plugins that share a
resource group with OS::Glance::Image. The management command will retrieve
up-to-date information from the Glance API. Data for other plugins will be
bulk-copied from a preexisting index into the new one using the scroll_ and
bulk_ features of Elasticsearch.
You can use the wildcard character * at the *end* of the ``type`` argument.
For instance, the following will match all cinder plugins::
$ searchlight-manage index sync --type OS::Cinder::*
Wildcard characters are only allowed at the end of the argument; they will not
be matched anywhere else.
To index all resources in the 'searchlight' resource group::
$ searchlight-manage index sync --index searchlight
You will be prompted to confirm unless ``--force`` is provided.
To reindex resources only without notification::
$ searchlight-manage index sync --notification-less
.. note::
We *strongly* recommend putting the notification-less plugins in their own
resource group and scheduling a cron_
job to periodically re-sync the notification plugins to keep the documents
up to date.
The ``searchlight-manage index sync`` command may be re-run at any time to
perform a full re-index of the data. As described above, there should be no
or very little impact on search requests during this process.
.. _scroll: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
.. _bulk: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html
.. _cron: https://en.wikipedia.org/wiki/Cron
Parent/child relationships
--------------------------
Some plugins contain multiple resources with parent/child relationships;
the Designate plugins are an example. Because reindexing parent data independent
of child documents does not logically make sense (without orphaning them), it
is not possible to request indexing of a child resource type::
$ searchlight-manage index sync --type OS::Designate::RecordSet
'OS::Designate::RecordSet' is a child of 'OS::Designate::Zone' and cannot be indexed separately.
Indexing 'OS::Designate::Zone' will re-index all child resource types.
You can see parent/child relationships in the list of resources presented prior
to indexing::
$ searchlight-manage index sync --type OS::Designate::Zone
Resource types (and indices) matching selection:
OS::Designate::Zone (designate)
----> OS::Designate::RecordSet
Child plugins will inherit their resource group name from their parent. Any
child configuration setting for resource_group_name will be ignored.
Incremental Updates
-------------------
Once a resource has been indexed, typically you will only need to consume
incremental updates rather than re-index the entire data set again. The
preferred methodology is to set up notification listening.
Notifications
^^^^^^^^^^^^^
Many services publish notifications when there are changes to the resources
they own. The searchlight listener consumes these notifications and will
perform incremental updates to the index based on those notifications.
To start this service, run the following command::
$ searchlight-listener
Note, this will typically require that you have configured notifications
properly for the service which owns the resource. For example, the glance
service owns images and metadata definitions. Please check the plugin
documentation for each service's specific configuration requirements.
Multi-Thread Support
--------------------
The Newton Searchlight release introduced multiple thread support for
indexing. Previously when the ``searchlight-manage index sync``
command was executed, all indexing occurred in a single thread. To boost
the performance of the indexing functionality, each resource type
will now index in its own thread. Multiple indexing threads will run
concurrently.
By default, the maximum number of simultaneous threads is 3. This limit
can be modified in the Searchlight configuration file. The setting is
called ``workers`` and lives under ``[manage]``. For example, to
increase the maximum number of threads to 6, the following can be added
to the Searchlight configuration file::
[manage]
workers=6
The use of threads can also affect the parsing of the log files. The
default formatting of the log messages include only the process ID,
but no thread-specific information. This can be changed by modifying
the formatting string settings in the Searchlight configuration file.
To add the thread ID for a message, add ``%(thread)d``. To add the thread
name, add ``%(threadName)s``. For example, to add the thread ID and the
thread name after the process ID to the logging message, the following
setting can be added to the Searchlight configuration file::
logging_default_format_string = %(asctime)s.%(msecs)03d %(process)d %(thread)d %(threadName)s %(levelname)s %(name)s [-] %(instance)s%(message)s
Force Elasticsearch indexing
----------------------------
The Newton Searchlight release introduced the ability to reindex
from Elasticsearch only, bypassing the plugin APIs altogether.
This option is useful if there has been a change to the mapping
definitions or the index settings. This functionality is enabled
with the option ``--apply-mapping-changes`` for the ``index`` command.
A sample usage would be::
$ searchlight-manage index aliases --apply-mapping-changes
The ``--type`` option is not compatible with the ``--apply-mapping-changes``
option. Specifying both options on the command line will result in an error.
.. warning::
The resource group cannot be changed when using this option.
If you do change the resource group, the underlying index will
be changed and will result in an empty index.
.. _ES-Index-Cleanup:
Elasticsearch Index Cleanup
---------------------------
In some cases, there may be orphaned Searchlight indices in Elasticsearch.
An orphaned index is one that is no longer used by Searchlight, either
directly or through an alias.
To help detect which Searchlight-related indices may be orphaned in
Elasticsearch, the ``searchlight-manage`` command will display all indices
that are currently being used by Searchlight. This is the ``aliases``
option to the ``index`` command::
$ searchlight-manage index aliases
This command outputs a listing of all indices that are used by
Searchlight (based on the current configuration file). The aliases
associated with each index is also shown. A sample output will look
like this::
$ searchlight-manage index aliases
List of Elasticsearch indices (and their associated aliases) used by Searchlight.
Note:
The indices are based on the current config file.
To view indices used by other Searchlight config files, use the --config-file option.
Indices are denoted with a '*'
Aliases are denoted with a '+'
* searchlight-2016_07_13_17_09_27
+ searchlight-listener
+ searchlight-search
* sl-swift-2016_07_13_17_09_26
+ sl-swift-listener
+ sl-swift-search
The example shows that Searchlight is using two indices in Elasticsearch:
``searchlight-2016_07_13_17_09_27`` and ``sl-swift-2016_07_13_17_09_26``.
The index ``searchlight-2016_07_13_17_09_27`` has two aliases: ``searchlight-listener``
and ``searchlight-search``. The index ``sl-swift-2016_07_13_17_09_26`` has
two aliases: ``sl-swift-listener`` and ``sl-swift-search``.
Any other indices or aliases in Elasticsearch are not used by this specific
Searchlight configuration. NOTE: If there are other Searchlight
instances running with a different configuration, their indices and aliases
will not be displayed by this command. The user will need to rerun the
``index aliases`` command using other configuration files.
.. _Notifications:
Notifications
=============
Aside from periodic reindexing, Searchlight can index in close to real-time
by subscribing to oslo.messaging notifications_. To configure services to send
them, it's typically only necessary to enable a driver in their configuration
files::
[oslo_messaging_notifications]
driver = messaging
Most notifications are emitted by a service's API but some services also emit
them in scheduler or conductor components.
Searchlight subscribes to notifications using a listener pool_ so it will not
steal notifications from e.g. telemetry projects.
.. _oslo-notifications: https://wiki.openstack.org/wiki/Oslo/Messaging#Emitting_Notifications
.. _pool: https://specs.openstack.org/openstack/oslo-specs/specs/kilo/notification-listener-pools.html
RabbitMQ
--------
The rabbitmq driver is the most commonly used. The permissions_ required for
searchlight are::
* configure: '^(searchlight-listener|openstack|nova|neutron|cinder|glance)$'
* write: '^(searchlight-listener)$'
* read: '^(searchlight-listener|nova|glance|cinder|neutron|openstack)$'
.. _permissions: https://www.rabbitmq.com/access-control.html

117
doc/source/admin/plugins/cinder.rst
View File

@ -1,117 +0,0 @@
..
c) Copyright 2016 Hewlett-Packard Enterprise Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
*******************
Cinder Plugin Guide
*******************
Integration is provided via a plugin. There are multiple configuration
settings required for proper indexing and incremental updates. Some of the
settings are specified in Searchlight configuration files. Others are
provided in other service configuration files.
Searchlight Configuration
=========================
Searchlight resource configuration options are shown below with their
configuration file and default values.
See :ref:`searchlight-plugins` for common options with their default values,
general configuration information, and an example complete configuration.
.. note::
Unless you are changing to a non-default value, you do not need to
specify any of the following configuration options.
searchlight.conf
----------------
Plugin: OS::Cinder::Volume
^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_cinder_volume]
enabled = true
resource_group_name = searchlight
Plugin: OS::Cinder::Snapshot
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_cinder_snapshot]
enabled = true
resource_group_name = searchlight
Cinder Configuration
====================
Cinder sends notifications on create/update/delete actions on the
resources that it implements. Currently Searchlight supports indexing
for volumes and snapshots of volumes. Backup support will be added but
some changes to the cinder backup API is required first.
cinder.conf
-----------
Notifications must be configured properly for searchlight to process
incremental updates. Enable notifications using the following::
[oslo_messaging_notifications]
driver = messagingv2
.. note::
Restart the Cinder api service (c-api) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
The settings above may be automatically configured by ``stack.sh``
by adding them to the following post config section in devstack.
Just place the following in local.conf and copy the above settings
underneath it.::
[[post-config|$CINDER_CONF]]
[DEFAULT]
Release Notes
=============
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
The following fields are exposed to administrators only for cinder volumes:
* os-vol-mig-status-attr:*
* os-vol-host-attr:*
* migration
Additional properties can be similarly protected with the `admin_only_fields`
under each plugin's configuration section. Glob-like patterns are supported.
For instance::
[resource_plugin:os_cinder_volume]
admin_only_fields=status,bootable
See: ADMIN_ONLY_FIELDS in:
* searchlight/elasticsearch/plugins/cinder/volumes.py

150
doc/source/admin/plugins/designate.rst
View File

@ -1,150 +0,0 @@
..
c) Copyright 2015 Hewlett-Packard Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
**********************
Designate Plugin Guide
**********************
Integration is provided via a plugin. There are multiple configuration
settings required for proper indexing and incremental updates. Some of the
settings are specified in Searchlight configuration files. Others are
provided in other service configuration files.
Searchlight Configuration
=========================
Searchlight resource configuration options are shown below with their
configuration file and default values.
See :ref:`searchlight-plugins` for common options with their default values,
general configuration information, and an example complete configuration.
.. note::
Unless you are changing to a non-default value, you do not need to
specify any of the following configuration options.
searchlight.conf
----------------
Plugin: OS::Designate::Zone
^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_designate_zone]
enabled = true
resource_group_name = searchlight
Plugin: OS::Designate::RecordSet
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_designate_recordset]
enabled = true
resource_group_name = searchlight
.. warning::
*OS::Designate::Zone* documents have a parent relationship to
*OS::Designate::RecordSet* documents. Because of this you must have
both *os_designate_zone* and *os_designate_recordset* plugin
configurations enabled or disabled together.
Designate Configuration
=======================
The Designate services must be configured properly to work with searchlight.
designate.conf
--------------
Notifications must be configured properly for searchlight to process
incremental updates. Enable notifications using the following::
[oslo_messaging_notifications]
driver = messagingv2
.. note::
Restart ``designate-central``, ``designate-pool-manager``, and
``designate-zone-manager`` after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
.. note::
Designate resource types are *not* enabled by default (``enabled = false``)
in the Searchlight devstack script because Designate is not
installed by default in devstack. If you have Designate installed in
devstack, you have two options for enabling designate resource types in
Searchlight:
1. Prior to stacking: modify the searchlight post config section in
``local.conf`` by adding a ``[[post-config|$SEARCHLIGHT_CONF]]`` section.
2. After stacking: manually edit the ``searchlight.conf`` file.
The Designate plugin must be enabled and run with devstack to include Designate
with your devstack deployment. Follow the instructions here:
https://opendev.org/openstack/designate/src/branch/master/devstack
The settings above may be automatically configured by ``stack.sh``
by adding them to the following post config section in devstack.
Just place the following in local.conf and copy the above settings
underneath it.::
[[post-config|$DESIGNATE_CONF]]
[DEFAULT]
Release Notes
=============
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
The Designate notification limitations mentioned in Liberty still apply.
You no longer need to use the --no-delete option mentioned below. Zero
downtime reindexing implemented in Mitaka handles all re-indexing
transparently.
0.1.0.0 (Liberty)
-----------------
For best results, use the v2 Designate API. Using the Designate v1 API to
create domains results in the Designate service not sending all possible
status change notifications. This causes Designate record set documents to
stay in the ``Pending`` status in the search index.
The Horizon UI uses the v1 API and causes the above issue to be seen.
So in order to ensure the search index contains the correct status values
for record sets when using Horizon, you may set up a cron job to
re-index Designate data.
You should use the ``--no-delete`` option to prevent the index from
temporarily not containing any data (which otherwise would happen with a full
bulk indexing job)::
searchlight-manage index sync --type OS::Designate::Zone,OS::Designate::RecordSet --force --no-delete

143
doc/source/admin/plugins/glance.rst
View File

@ -1,143 +0,0 @@
..
c) Copyright 2015 Hewlett-Packard Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
*******************
Glance Plugin Guide
*******************
Integration is provided via a plugin. There are multiple configuration
settings required for proper indexing and incremental updates. Some of the
settings are specified in Searchlight configuration files. Others are
provided in other service configuration files.
Searchlight Configuration
=========================
Searchlight resource configuration options are shown below with their
configuration file and default values.
See :ref:`searchlight-plugins` for common options with their default values,
general configuration information, and an example complete configuration.
.. note::
Unless you are changing to a non-default value, you do not need to
specify any of the following configuration options.
searchlight.conf
----------------
Plugin: OS::Glance::Image
^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_glance_image]
enabled = true
resource_group_name = searchlight
**Glance Image Property Protections**
Glance uses a property protections mechanism to ensure that certain
properties are limited to only people with the appropriate permissions.
Searchlight includes the same functionality and must be deployed with
the same property protections files and configured to use that file. A
sample configuration file is included in the repo and may be used for testing.
To configure it, add a ``property_protection_file`` property with a path
to the file in ``searchlight.conf``. For example::
property_protection_file = /etc/searchlight/property-protections-roles.conf
See also: `Glance Property Protections <https://docs.openstack.org/glance/latest/admin/property-protections.html>`_
Plugin: OS::Glance::Metadef
^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_glance_metadef]
enabled = true
resource_group_name = searchlight
See also: `Metadata Definitions Catalog <https://docs.openstack.org/glance/latest/user/metadefs-concepts.html>`_
Glance Configuration
====================
The Glance services must be configured properly to work with searchlight.
glance-api.conf
---------------
Notifications must be configured properly for searchlight to process
incremental updates. Enable notifications using the following::
[oslo_messaging_notifications]
driver = messagingv2
.. note::
Restart the Glance api service (g-api) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
The settings above may be automatically configured by ``stack.sh``
by adding them to the following post config section in devstack.
Just place the following in local.conf and copy the above settings
underneath it.::
[[post-config|$GLANCE_API_CONF]]
[DEFAULT]
Release Notes
=============
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
0.1.0.0 (Liberty)
-----------------
Glance did not generate notifications for Image Member updates up to and
including the Liberty release.
This means that search results will include correct results when the image
visibility is ``public`` or ``private``, but ``shared`` images will only be
included in search results for the owning project without additional deployment
configuration.
The patch (https://review.opendev.org/221307) implements this feature and
will be included/merged in the Glance Mitaka release.
Searchlight developers/installers should apply the above patch in Glance when
using Searchlight with the Glance Liberty release.
Alternatively, you may set up a cron job to re-index glance images
periodically to get updated membership information.
You should use the ``--no-delete`` option to prevent the index from
temporarily not containing any data (which otherwise would happen with a full
bulk indexing job)::
searchlight-manage index sync --type OS::Glance::Image --force --no-delete

86
doc/source/admin/plugins/ironic.rst
View File

@ -1,86 +0,0 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
*******************
Ironic Plugin Guide
*******************
Integration is provided via a plugin. There are multiple configuration
settings required for proper indexing and incremental updates. Some of the
settings are specified in Searchlight configuration files. Others are
provided in other service configuration files.
Searchlight Configuration
=========================
Searchlight resource configuration options are shown below with their
configuration file.
See :ref:`searchlight-plugins` for common options with their default values,
general configuration information, and an example complete configuration.
searchlight.conf
----------------
Plugin: OS::Ironic::Chassis
^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_ironic_chassis]
notifications_topics_exchanges = ironic_versioned_notifications,ironic
.. note::
Chassis is not parent resource for node because chassis is not mandatory
for a node.
Plugin: OS::Ironic::Node
^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_ironic_node]
notifications_topics_exchanges = ironic_versioned_notifications,ironic
Plugin: OS::Ironic::Port
^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_ironic_port]
notifications_topics_exchanges = ironic_versioned_notifications,ironic
Ironic Configuration
====================
The ironic services must be configured properly to work with searchlight.
ironic.conf
-----------
Notifications must be configured properly for searchlight to process
incremental updates. Enable notifications using the following::
[DEFAULT]
notification_level = info
[oslo_messaging_notifications]
driver = messagingv2
.. note::
Restart ironic api and conductor services after making changes.
Release Notes
=============
``properties`` node's field mapped to ``node_properties`` due to limitation of
older elasticsearch versions.

196
doc/source/admin/plugins/neutron.rst
View File

@ -1,196 +0,0 @@
..
c) Copyright 2016 Hewlett-Packard Enterprise Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
********************
Neutron Plugin Guide
********************
Integration is provided via a plugin. There are multiple configuration
settings required for proper indexing and incremental updates. Some of the
settings are specified in Searchlight configuration files. Others are
provided in other service configuration files.
Searchlight Configuration
=========================
Searchlight resource configuration options are shown below with their
configuration file and default values.
See :ref:`searchlight-plugins` for common options with their default values,
general configuration information, and an example complete configuration.
.. note::
Unless you are changing to a non-default value, you do not need to
specify any of the following configuration options.
searchlight.conf
----------------
Plugin: OS::Neutron::Net
^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_neutron_net]
enabled = true
resource_group_name = searchlight
Plugin: OS::Neutron::Port
^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_neutron_port]
enabled = true
resource_group_name = searchlight
Plugin: OS::Neutron::Subnet
^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_neutron_subnet]
enabled = true
resource_group_name = searchlight
Plugin: OS::Neutron::Router
^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_neutron_router]
enabled = true
resource_group_name = searchlight
Plugin: OS::Neutron::FloatingIP
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_neutron_floatingip]
enabled = true
resource_group_name = searchlight
Plugin: OS::Neutron::SecurityGroup
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_neutron_security_group]
enabled = true
resource_group_name = searchlight
Neutron Configuration
=====================
Neutron sends notifications on create/update/delete actions on the
concepts that it implements. Currently Searchlight supports indexing
for networks, subnets, ports, routers, floating IPs and security groups.
neutron.conf
------------
Notifications must be configured properly for searchlight to process
incremental updates. Enable notifications using the following::
[oslo_messaging_notifications]
driver = messagingv2
.. note::
Restart the Neutron api service (q-svc) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
The settings above may be automatically configured by ``stack.sh``
by adding them to the following post config section in devstack.
Just place the following in local.conf and copy the above settings
underneath it.::
[[post-config|$NEUTRON_CONF]]
[DEFAULT]
Neutron RBAC Reference
======================
RBAC Filters
------------
RBAC in searchlight neutron plugin is implemented as:
Networks are visible within a tenant OR if they are shared OR if they are external.
Subnets are visible within a tenant OR if their network is shared (OR for admins if their network is external).
Ports are visible within a tenant (OR for admins if their network is shared or external).
Routers are visible within a tenant.
Floating IPs are visible within a tenant.
Security groups are visible within a tenant.
Neutron Ports Notifications
===========================
Not all Neutron ports send notifications when created/updated. Depending on the port's
``device_owner``, the notifications will be sent. The ``device_owner``'s that will
send a notification are:
* compute:*
* baremetal:*
We want the initial indexing (and subsequent re-indexings) to match the state that
Searchlight receives from the Neutron notifications. Having this mismatch will lead
to the Searchlight state being out of sync with the Neutron state. To prevent this
from happening, ``searchlight-manage`` will index only the Neutron ports that have
``device_owner`` defined above. All other ports with ``device_owner`` not listed
above will be ignored when indexing.
Release Notes
=============
1.0.0.0 (Newton)
----------------
The Neutron tenant RBAC policy functionality is supported as
part of the OS::Neutron::Net resource type.
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
DHCP ports are *not* indexed. Neutron doesn't provide a reliable way for
Searchlight to index these ports since they are created and modified
asynchronously from the subnets that they're attached to.
All provider:* properties of networks are exposed to administrators only.
All binding:* properties of ports are also visible only to administrators.
The 'distributed' and 'ha' router properties are available only to
administrators.
Additional properties can be protected similarly with the `admin_only_fields`
under each plugin's configuration section. Glob-like patterns are supported.
For instance::
[resource_plugin:os_neutron_net]
admin_only_fields=admin_state_up,status
See: ADMIN_ONLY_FIELDS in:
* searchlight/elasticsearch/plugins/neutron/networks.py
* searchlight/elasticsearch/plugins/neutron/ports.py
* searchlight/elasticsearch/plugins/neutron/routers.py
Floating IP addresses are mapped as type 'ip' which only supports ipv4 in
Elasticsearch prior to the 5.0 release. Neutron doesn't seem to support ipv6
floating IPs since translating ipv4 FIPs to ipv6 internal addresses isn't
supported and mapping ipv6->ipv6 is deemed unnecessary.

231
doc/source/admin/plugins/nova.rst
View File

@ -1,231 +0,0 @@
..
c) Copyright 2015 Hewlett-Packard Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
*****************
Nova Plugin Guide
*****************
Integration is provided via a plugin. There are multiple configuration
settings required for proper indexing and incremental updates. Some of the
settings are specified in Searchlight configuration files. Others are
provided in other service configuration files.
Searchlight Configuration
=========================
Searchlight resource configuration options are shown below with their
configuration file and default values.
See :ref:`searchlight-plugins` for common options with their default values,
general configuration information, and an example complete configuration.
.. note::
Unless you are changing to a non-default value, you do not need to
specify any of the following configuration options.
searchlight.conf
----------------
Nova microversions
^^^^^^^^^^^^^^^^^^
::
[service_credentials:nova]
compute_api_version = 2.1
.. note::
Nova adds/removes fields using microversion mechanism, check
https://docs.openstack.org/nova/latest/api_microversion_history.html
for detailed Nova microversion history.
Plugin: OS::Nova::Server
^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_nova_server]
enabled = true
resource_group_name = searchlight
notifications_topics_exchanges = versioned_notifications,nova
use_versioned_notifications = true
Plugin: OS::Nova::Hypervisor
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This plugin represents each of the Nova compute nodes.
::
[resource_plugin:os_nova_hypervisor]
enabled = true
resource_group_name = sl_without_notification
.. note::
There are no notifications from the compute nodes ("hypervisors") from
nova yet, so we recommend putting it in its own resource group and
scheduling a cron job to periodically re-sync. This will create a very
low overhead way to keep the index up to date. The index latency will be
dependent on how often you re-sync the data.
Plugin: OS::Nova::Flavor
^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_nova_flavor]
enabled = true
resource_group_name = searchlight
notifications_topics_exchanges = versioned_notifications,nova
.. note::
The notifications topic for flavors is versioned_notifications, so we
need to config notifications_topics_exchanges with value
'versioned_notifications,nova' in order to get the related versioned
notifications from nova.
Plugin: OS::Nova::SeverGroup
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_nova_servergroup]
enabled = true
resource_group_name = searchlight
.. note::
The return value of os-server-groups API from nova doesn't contain
project and user information before nova API microversion v2.13,
thus the index cannot been searched by particular project.
Nova Configuration
==================
The nova services must be configured properly to work with searchlight.
nova.conf
---------
Notifications must be configured properly for searchlight to process
incremental updates. Enable notifications using the following::
[oslo_messaging_notifications]
driver = messagingv2
[notifications]
notify_on_state_change = vm_and_task_state
# notification_format = versioned
.. note::
Restart Nova API and Nova scheduler (n-api, n-sch) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
The default setting for notification_format is 'both' which sends both
versioned and unversioned notifications. Searchlight uses
'use_versioned_notifications' to decide which to use.
local.conf (devstack)
---------------------
The settings above may be automatically configured by ``stack.sh``
by adding them to the following post config section in devstack.
Just place the following in local.conf and copy the above settings
underneath it.::
[[post-config|$NOVA_CONF]]
[DEFAULT]
Neutron Configuration
=====================
Since changes to Neutron can affect Nova instances you may optionally turn on
notifications for Neutron. If you do not, networking changes will only be
picked up by Searchlight when notifications are received from Nova.
neutron.conf
------------
Notifications must be configured properly for searchlight to process
incremental updates. Enable notifications using the following::
[oslo_messaging_notifications]
driver = messagingv2
.. note::
Restart the Neutron api service (q-svc) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
The settings above may be automatically configured by ``stack.sh``
by adding them to the following post config section in devstack.
Just place the following in local.conf and copy the above settings
underneath it.::
[[post-config|$NEUTRON_CONF]]
[DEFAULT]
Release Notes
=============
1.0.0.0 (Newton)
----------------
In order to reduce the impact on the nova API, changes have been made to the
way notifications are processed. Currently searchlight has to retrieve nova
server information from nova because the notifications alone are missing
several pieces of information. Prior to Newton this meant up to 7 API requests
during a server boot. During Newton this was changed. There will now be one
initial nova request prior to the scheduler, one when the
``instance.create.start`` notification is received, one when networking is
established and one after the instance has booted and run any init scripts.
Other notifications during boot will update only the server status.
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
The following fields are exposed to administrators only for nova instances:
* OS-EXT-SRV-ATTR:*
Additional properties can be similarly protected with the `admin_only_fields`
under each plugin's configuration section. Glob-like patterns are supported.
For instance::
[resource_plugin:os_nova_server]
admin_only_fields=OS-EXT-STS:vm_state
See: ADMIN_ONLY_FIELDS in:
* searchlight/elasticsearch/plugins/nova/servers.py
0.1.0.0 (Liberty)
-----------------
All OS-EXT-SRV-ATTR:.* properties are filtered out from search results
for non-admin users. This is not a configuration option in this release.
To change this or filter out additional properties, you must change the
plugin code to add additional properties.
See: ADMIN_ONLY_PROPERTIES in searchlight/elasticsearch/plugins/nova/servers.py

223
doc/source/admin/plugins/swift.rst
View File

@ -1,223 +0,0 @@
..
(c) Copyright 2016 Hewlett-Packard Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
******************
Swift Plugin Guide
******************
WARNING: The Swift plugin is currently EXPERIMENTAL as incremental indexing is
only provided via an experimental OSLO messaging middleware patch while
other indexing methodologies are explored for Swift.
See :ref:`proxy-server.conf` for additional information.
Integration is provided via a plugin. There are multiple configuration
settings required for proper indexing and incremental updates. Some of the
settings are specified in Searchlight configuration files. Others are
provided in Swift configuration files.
Swift Configuration
====================
reseller_admin_role
-------------------
Users with the Keystone role defined in `reseller_admin_role` (`ResellerAdmin`
by default) can operate on any account. The auth system sets the request
context variable `reseller_request` to True if a request is coming from a user
with this role.
Searchlight needs this role for its service user to access all of the swift
accounts during initial indexing. The `searchlight` user and `service` project
being referred to here are defined in the `service_credentials` section of
`searchlight.conf` file. If any of the Swift plugins are enabled, this
role must be added prior to running `searchlight-manage index sync`.
::
openstack role add --user searchlight --project service ResellerAdmin
.. _proxy-server.conf:
proxy-server.conf
-----------------
Incremental indexing for searchlight is typically provided via OSLO
messaging. The Swift service currently doesn't send notifications, but
work has been started to investigate more performant ways to achieve
indexing. In the meantime, experimental support for providing notifications
via middleware is provided in the following patch:
* https://review.opendev.org/#/c/249471
#. Apply the patch to the Swift proxy server
#. Make the below configuration changes to `proxy-server.conf`
#. Restart the Swift proxy server
::
# Add the following new section
[filter:oslomiddleware]
paste.filter_factory = swift.common.middleware.oslo_notifications:filter_factory
publisher_id = swift.localhost
#Replace <user>,<password>,<rabbitip> and <rabbitport> for your environment values
transport_url = rabbit://<user>:<password>@<rabbitip>:<rabbitport>/
notification_driver = messaging
notification_topics = notifications
# Add oslomiddleware to pipeline:main
# see example below.
[pipeline:main]
pipeline = catch_errors gatekeeper ...<other>... oslomiddleware proxy-logging proxy-server
.. note::
Restart the swift proxy API service (s-proxy) after making changes.
Starting in Newton, Searchlight can share the same notification topic as
other services, because it uses a messaging pool.
Searchlight Configuration
=========================
Searchlight resource configuration options are shown below with their
configuration file and default values.
See :ref:`searchlight-plugins` for common options with their default values,
general configuration information, and an example complete configuration.
.. note::
Unless you are changing to a non-default value, you do not need to
specify any of the following configuration options. After enabling or
disabling a plugin you do need to restart the searchlight services
(`searchlight-api` and `searchlight-listener`).
After enabling a Swift plugin, you will also need to run the sync job:
`searchlight-manage index sync --type OS::Swift::Account`
searchlight.conf
----------------
Plugin: OS::Swift::Account
^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_swift_account]
enabled = true
resource_group_name = searchlight
# Specify same value as in swift proxy-server.conf for reseller_prefix
reseller_prefix = AUTH_
.. note::
`os_swift_account` is disabled by default. You need to explicitly
set enabled = True as shown above.
Plugin: OS::Swift::Container
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_swift_container]
enabled = true
resource_group_name = searchlight
.. note::
`os_swift_container` is disabled by default. You need to explicitly
set enabled = True as shown above.
Plugin: OS::Swift::Object
^^^^^^^^^^^^^^^^^^^^^^^^^
::
[resource_plugin:os_swift_object]
enabled = true
resource_group_name = searchlight
.. note::
`os_swift_object` is disabled by default. You need to explicitly
set enabled = True as shown above.
local.conf (devstack)
---------------------
At this time we recommend that you manually enable the Searchlight plugins
and middleware for Swift after devstack has completed stacking. Please
follow the instructions above.
Release Notes
=============
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
Large scale swift cluster support is targeted at a future release, but
we encourage trial deployments to help us address issues as soon as possible.
Swift did not generate notifications for account/container/object CRUD
during the Mitaka release. This means that search results will not include
incremental updates after the initial indexing. However, there is a patch
available to enable notifications via oslo messaging for the Mitaka release.
* https://review.opendev.org/#/c/249471
For devstack, the easiest way to test is::
cd /opt/stack/swift
git review -x 249471
<restart swift api>
Searchlight developers/installers should apply the above patch in Swift when
using Searchlight with the Swift Mitaka release. We are working with the
Swift team to create a supported incremental indexing methodology for future
releases.
Alternatively, you may set up a cron job to re-index swift
account/container/objects periodically to get updated information. The
recommendation is to use the notifications, because a full re-indexing will
not be performant in large installations.
::
searchlight-manage index sync --type OS::Swift::Account
The Searchlight Swift plugin resource types follow the hierarchy similar to
Swift concepts
::
OS::Swift:Account(Parent)
-> OS:Swift::Container(Child)
-> OS::Swift::Object(Grand Child)
which means indexing is initiated by specifying only the top parent
(OS::Swift::Account) and that will in-turn index all the child
plugins(Container and Object)
Searchlight is adding indexing isolation in the Newton release via a concept
called resource group isolation. This will better support re-indexing
scalability.
Additional properties can be similarly protected with the `admin_only_fields`
under each plugin's configuration section. Glob-like patterns are supported.
For instance::
[resource_plugin:os_swift_object]
admin_only_fields=x-meta-admin*

7
doc/source/cli/index.rst
View File

@ -1,7 +0,0 @@
CLI Reference
=============
.. toctree::
:maxdepth: 1
searchlight-status

78
doc/source/cli/searchlight-status.rst
View File

@ -1,78 +0,0 @@
==================
searchlight-status
==================
Synopsis
========
::
searchlight-status <category> <command> [<args>]
Description
===========
:program:`searchlight-status` is a tool that provides routines for checking the
status of a Searchlight deployment.
Options
=======
The standard pattern for executing a :program:`searchlight-status` command is::
searchlight-status <category> <command> [<args>]
Run without arguments to see a list of available command categories::
searchlight-status
Categories are:
* ``upgrade``
Detailed descriptions are below.
You can also run with a category argument such as ``upgrade`` to see a list of
all commands in that category::
searchlight-status upgrade
These sections describe the available categories and arguments for
:program:`searchlight-status`.
Upgrade
~~~~~~~
.. _searchlight-status-checks:
``searchlight-status upgrade check``
Performs a release-specific readiness check before restarting services with
new code. This command expects to have complete configuration and access
to databases and services.
**Return Codes**
.. list-table::
:widths: 20 80
:header-rows: 1
* - Return code
- Description
* - 0
- All upgrade readiness checks passed successfully and there is nothing
to do.
* - 1
- At least one check encountered an issue and requires further
investigation. This is considered a warning but the upgrade may be OK.
* - 2
- There was an upgrade status check failure that needs to be
investigated. This should be considered something that stops an
upgrade.
* - 255
- An unexpected error occurred.
**History of Checks**
**5.0.0 (Stein)**
* Placeholder to be filled in with checks as they are added in Stein.

253
doc/source/conf.py
View File

@ -1,253 +0,0 @@
# Copyright (c) 2010 OpenStack Foundation.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Searchlight documentation build configuration file, created by
# sphinx-quickstart on Tue May 26 13:50:15 2015.
#
# This file is execfile()'d with the current directory set to its containing
# dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import os
import sys
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path = [
os.path.abspath('../..'),
os.path.abspath('../../bin')
] + sys.path
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.coverage',
'sphinx.ext.ifconfig',
'sphinx.ext.graphviz',
'openstackdocstheme',
'oslo_policy.sphinxext',
'oslo_policy.sphinxpolicygen',
'sphinxcontrib.rsvgconverter',
]
# Add any paths that contain templates here, relative to this directory.
# templates_path = []
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8'
# The master toctree document.
master_doc = 'index'
# General information about the project.
copyright = u'2010-2015, OpenStack Foundation.'
policy_generator_config_file = \
'../../etc/oslo-policy-generator/searchlight.conf'
sample_policy_basename = '_static/searchlight'
# openstackdocstheme options
openstackdocs_repo_name = 'openstack/searchlight'
openstackdocs_pdf_link = True
openstackdocs_bug_project = 'searchlight'
openstackdocs_bug_tag = ''
# The short X.Y version.
from searchlight.version import version_info as searchlight_version
# The full version, including alpha/beta/rc tags.
release = searchlight_version.version_string_with_vcs()
# The short X.Y version.
version = searchlight_version.canonical_version_string()
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of documents that shouldn't be included in the build.
#unused_docs = []
# List of directories, relative to source directory, that shouldn't be searched
# for source files.
exclude_trees = ['api']
# The reST default role (for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
show_authors = True
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'native'
# A list of ignored prefixes for module index sorting.
modindex_common_prefix = ['searchlight.']
# -- Options for man page output --------------------------------------------
# Grouping the document tree for man pages.
# List of tuples 'sourcefile', 'target', u'title', u'Authors name', 'manual'
#man_pages = [
# ('man/searchlightapi', 'searchlight-api', u'Searchlight API Server',
# [u'OpenStack'], 1),
# ('man/searchlightlistener', 'searchlight-listener', u'Searchlight indexing agent',
# [u'OpenStack'], 1),
#]
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
#html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
html_last_updated_fmt = '%Y-%m-%d %H:%M'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
html_use_modindex = False
# If false, no index is generated.
html_use_index = False
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = ''
# Output file base name for HTML help builder.
htmlhelp_basename = 'searchlightdoc'
# -- Options for LaTeX output ------------------------------------------------
# The paper size ('letter' or 'a4').
#latex_paper_size = 'letter'
# The font size ('10pt', '11pt' or '12pt').
#latex_font_size = '10pt'
latex_use_xindy = False
latex_domain_indices = False
latex_elements = {
'makeindex': '',
'printindex': '',
'preamble': r'\setcounter{tocdepth}{3}',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author,
# documentclass [howto/manual]).
latex_documents = [
('index',
'doc-Searchlight.tex',
u'Searchlight Documentation',
u'Searchlight Team', 'manual', True),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# Additional stuff for the LaTeX preamble.
#latex_preamble = ''
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_use_modindex = True

250
doc/source/configuration/authentication.rst
View File

@ -1,250 +0,0 @@
..
Copyright 2010 OpenStack Foundation
All Rights Reserved.
c) Copyright 2015 Hewlett-Packard Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Keystone Authentication
=======================
Searchlight should be integrated with keystone. Setting this up is
relatively straightforward, as the keystone distribution includes the
necessary middleware. Once you have installed keystone and edited your
configuration files, users will need to have an authenticated keystone token
in all API requests. The keystone integration will allow both active denial
of requests from unauthenticated users and will also allow proper search
result filtering.
.. DANGER::
If the API is not configured with keystone, all data indexed by
searchlight is at risk of being accessed by unauthorized users.
Configuring the searchlight services to use keystone
----------------------------------------------------
Keystone is integrated with searchlight through the use of middleware.
The default configuration files for the Searchlight API use a single piece of
middleware called ``unauthenticated-context``, which generates a request
context containing blank authentication information. In order to configure
Searchlight to use Keystone, the ``authtoken`` and ``context`` middleware
must be deployed in place of the ``unauthenticated-context`` middleware.
The ``authtoken`` middleware performs the authentication token validation
and retrieves actual user authentication information. It can be found in
the keystone distribution. For more information, please refer to the Keystone
documentation on the ``auth_token`` middleware:
https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html
api-paste.ini
`````````````
First, ensure that declarations for the middleware exist in the
``api-paste.ini`` file. Here is an example for ``authtoken``::
[pipeline:searchlight-keystone]
pipeline = authtoken context rootapp
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
delay_auth_decision = true
searchlight.conf
````````````````
You must then update the main ``searchlight.conf`` configuration file
to enable the keystone application pipeline.
Set ``flavor`` to ``keystone`` in the ``paste_deploy`` group::
[paste_deploy]
flavor = keystone
Set ``keystone_authtoken`` options. The following sets the searchlight
service user as the user for performing policy API authentication checks.
The actual options and values in this section will need to be set according
to your environment::
[keystone_authtoken]
auth_url = http://127.0.0.1:5000
auth_type = password
project_domain_id = default
project_name = service
user_domain_id = default
password = <SERVICE_PASSWORD>
username = searchlight
.. note::
For development and unit testing, it is recommended to also set
``revocation_cache_timeout = 10`` under the ``keystone_authtoken`` group.
Set ``service_credentials`` options. Searchlight plugins may make API calls
to other services to index their data. Prior to doing this, it will get a
valid token based on the integration account credentials::
[service_credentials]
# These are needed to make API calls to other services when indexing
auth_type = password
username = searchlight
password = <SERVICE_PASSWORD>
user_domain_id = default
project_domain_id = default
project_name = service
auth_url = http://127.0.0.1:5000
# If resource_plugin.include_region_name is set, this value will be
# the default value for the 'region_name' field on all documents
# os_region_name =
For keystone v2 development::
[service_credentials]
auth_type = v2password
username = searchlight
tenant_name = service
password = <SERVICE_PASSWORD>
auth_url = http://127.0.0.1:35357/v2.0
# If resource_plugin.include_region_name is set, this value will be
# the default value for the 'region_name' field on all documents
# os_region_name =
Service integration account
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some of the above configuration implicitly uses a ``searchlight`` service user.
If you intend to use this user, it must have been created and registered with
keystone. Typically, this is done with the following commands (v3 keystone)::
$ openstack project create --or-show service --property domain=default
$ openstack user create searchlight --password <SERVICE_PASSWORD> --project service
$ openstack role add admin --project service --user searchlight
For more information on keystone service accounts, see:
https://docs.openstack.org/keystone/latest/admin/manage-services.html#create-service-users
Policy restriction
==================
Searchlight uses the oslo policy library to allow control over the level of
access a user has based on their authenticated roles. Policy rules are defined
in a configuration file (by default, `etc/policy.json`). By default, all
operations are allowed.
https://docs.openstack.org/oslo.policy/latest/reference/index.html
rule formatting.
During the last few cycles concerns were raised about the scope of the
``admin`` role within OpenStack. Many services consider any token scoped with
the ``admin`` role to have access to resources within any project. With the
introduction of keystone v3 it is possible to create users with the admin role
on a particular project, but not with the intention of them seeing resources in
other projects.
Keystone added two configuration options called ``admin_project_name`` and
``admin_project_domain_name`` to attempt to address this. If a request is
authenticated against a the project whose name is ``admin_project_name``
in the ``admin_project_domain_name`` domain, a flag is set on the
authentication response headers indicating that the user is authenticated
against the administrative project. This can then be supported by the policy
rule (in Searchlight's ``policy.json``)::
"is_admin_context": "role:admin and is_admin_project:True"
Since devstack configures keystone to support those options, this is the
default in Searchlight. To maintain backwards compatibility, if your keystone
is *not* configured to set these options, any token with the ``admin`` role
will be assumed to have administrative powers (this approach has been taken
by other OpenStack services).
For more history see https://bugs.launchpad.net/keystone/+bug/968696.
Access to operations
--------------------
It is possible to restrict access to functionality by setting rules for
``query``, ``facets`` or ``plugins_info``. For instance, to restrict facet
listing to administrators and disable plugin information for all users::
"facets": "role:admin",
"plugins_info": "!"
Where a request is disallowed on this basis, the user will receive a
403 Forbidden response.
Note that policy rules are applied on the fly; no server restart is required.
Policy rules denying access to operations take precedence over the per-resource
access described below.
Access to resources
-------------------
It is possible to disable access to individual plugins. For instance, the
following restricts access to Nova servers to admins, and disables access
entirely to Glance images::
"resource:OS::Nova::Server": "role:admin",
"resource:OS::Glance::Image": "!",
.. note::
At current plugins still apply RBAC separately from policy rules. We
aim to bring the two closer together in a later patch.
When resources are restricted in this way resources will be excluded
from the search (which may result in empty search results). No Forbidden
response will be returned.
.. _service-policy-controls:
Service policy controls
-----------------------
If configured, Searchlight can consult service policy files (e.g. that used
to configure the nova API). Each resource is configured with a policy target
it will check if possible. Policy file paths can either be absolute or relative
to `service_policy_path` (which itself can be relative to the current working
directory or left blank). The actual filepath used will be determined by
oslo.config using the same `logic`_ as for other config files (for logging,
searchlight's policy file etc). With the following configuration
stanza::
[service_policies]
service_policy_files=compute:nova-policy.json
service_policy_path=/etc/searchlight/
And with the following contents in nova-policy.json (which might be a symlink
to an existing nova policy file, a copy or a separate file)::
{
"is_admin": "role: admin",
"os_compute_api:servers:index": "rule:is_admin"
}
Only requests with the admin role assigned will be allowed to search or facet
Nova servers.
Policy files are configured per *service*, not per resource type. If files
are in different directories absolute paths should be used, and
``service_policy_path`` left unset.
.. note::
Policy rules are always *more* restrictive. If a rule in Searchlight's
``policy.json`` would allow access but a service policy file would disallow
it (or vice versa), the more restrictive rule will be used.
.. _logic: https://docs.openstack.org/oslo.config/latest/reference/configopts.html

11
doc/source/configuration/index.rst
View File

@ -1,11 +0,0 @@
=====================
Configuration guide
=====================
.. toctree::
:maxdepth: 2
authentication
plugins
policy
sample-policy-yaml

384
doc/source/configuration/plugins.rst
View File

@ -1,384 +0,0 @@
..
Copyright 2015 Hewlett-Packard Development Company, L.P.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _searchlight-plugins:
Searchlight Plugin Documentation
================================
The search service determines the types of information that is searchable
via a plugin mechanism.
.. _installing-plugins:
Installing Plugins
------------------
Plugins must be registered in ``setup.cfg``.
Within ``setup.cfg`` the setting within ``[entry_points]`` named
``searchlight.index_backend`` should list the plugin for each available
indexable type. After making a change, it's necessary to re-install the
python package (for instance with ``pip install -e .``).
Each plugin registered in ``setup.cfg`` is enabled by default. Typically it
should only be necessary to modify ``setup.cfg`` if you are installing a new
plugin. It is not necessary to modify ``[entry_points]`` to temporarily
enable or disable installed plugins. Once they are installed, they can be
disabled, enabled and configured in the ``searchlight.conf`` file.
Configuring Plugins
-------------------
After installation, plugins are configured in ``searchlight.conf``.
.. note::
After making changes to ``searchlight.conf`` you must perform the
actions indicated in the tables below.
1. ``Restart services``: Restart all running ``searchlight-api`` *and*
``searchlight-listener`` processes.
2. ``Re-index affected types``: You will need to re-index any resource
types affected by the change. (See :doc:`../admin/indexingservice`).
.. note::
Unless you are changing to a non-default value, you do not need to
specify any of the following configuration options.
.. _end-to-end-plugin-configuration-example:
End to End Configuration Example
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following shows a sampling of various configuration options in
``searchlight.conf``. These are **NOT** necessarily recommended
or default configuration values. They are intended for exemplary purposes only.
Please read the rest of the guide for detailed information.::
[listener]
notifications_pool = searchlight
[resource_plugin]
resource_group_name = searchlight
# include_region_name = True
[service_credentials:nova]
compute_api_version = 2.1
[resource_plugin:os_nova_server]
enabled = True
admin_only_fields = OS-EXT-SRV*,OS-EXT-STS:vm_state
[resource_plugin:os_nova_hypervisor]
enabled = True
[resource_plugin:os_nova_flavor]
enabled = True
[resource_plugin:os_nova_servergroup]
enabled = False
[resource_plugin:os_glance_image]
enabled = True
# override_region_name = Region1,Region2
[resource_plugin:os_glance_metadef]
enabled = True
[resource_plugin:os_cinder_volume]
enabled = True
[resource_plugin:os_cinder_snapshot]
enabled = True
[resource_plugin:os_neutron_net]
enabled = True
admin_only_fields=admin_state_up,status
[resource_plugin:os_neutron_port]
enabled = True
[resource_plugin:os_neutron_security_group]
enabled = True
[resource_plugin:os_designate_zone]
enabled = False
[resource_plugin:os_designate_recordset]
enabled = False
[resource_plugin:os_swift_account]
enabled = False
[resource_plugin:os_swift_container]
enabled = False
[resource_plugin:os_swift_object]
enabled = False
Common Plugin Configuration Options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are common configuration options that all plugins honor. They are split
between *global*, *inheritable* and *non-inheritable* options.
**Global** plugin configuration options apply to all plugins and cannot be
overridden by an individual plugin.
**Inheritable** common configuration options may be specified in a default
configuration group of ``[resource_plugin]`` in ``searchlight.conf`` and
optionally overridden in a specific plugin's configuration. For example::
[resource_plugin]
resource_group_name = searchlight
[resource_plugin:os_nova_server]
resource_group_name = searchlight-nova-servers
**Non-Inheritable** common configuration options are honored by all plugins,
but must be specified directly in that plugin's configuration group. They
are not inherited from the ``[resource_plugin]`` configuration group. For
example::
[resource_plugin:os_glance_image]
enabled = false
.. _plugin_notifications:
Notifications
.............
There are two ways to configure services to send notifications that
Searchlight can receive. The recommended method is to configure
Searchlight to use the notification topic that each service is already
configured to use and then to allow Searchlight to consume messages from
that topic using a pool, touched on in the `messaging documentation`_.
Searchlight uses this configuration by default.
.. _`messaging documentation`: https://docs.openstack.org/oslo.messaging/latest/configuration/opts.html
**Topics**
Searchlight defaults to using the oslo notification topic of
``notifications``. This is the oslo default topic which most services also
use to broadcast their notifications. You will need to change the topic in both
``searchlight.conf`` and the various service configuration files if you want
to modify the topic used by Searchlight. Each plugin can use a different topic.
Notification topics are a special case. It is possible to override
the notification ``topic`` as a shared setting; it is also possible to
override ``<topic>,<exchange>`` pairs per-plugin in the case where some
services are using different topics. For instance, in a setup where (for
example) neutron is using a separate notification topic::
[resource_plugin]
notifications_topic = searchlight_indexer
[resource_plugin:os_nova_server]
notifications_topics_exchanges = searchlight_indexer,nova
notifications_topics_exchanges = another-topic,neutron
If you override one service topic, you must provide topic,exchange pairs
for all service notifications a plugin supports.
**Pools**
In addition, Searchlight uses a notification pool. This allows Searchlight
to listen on the same topic to which other services are listening while
ensuring that Searchlight still gets its own copy of each notification. The
default notification pool is set to ``searchlight``. This is set using the
``notifications_pool`` setting in the ``[listener]`` configuration group.
Example::
[listener]
notification_pools = searchlight
See :ref:`individual-plugin-configuration` for more information and examples
on individual plugin configuration.
Publishers
..........
Searchlight supports configuration of publishers which push enriched
notification info to other external systems. To use this feature, you must first
register publishers in ``setup.cfg``. This is similar to
:ref:`installing-plugins` action. Within ``setup.cfg`` the setting within
``[entry_points]`` named ``searchlight.publishers`` should list publishers for
plugins to use. Don't forget to re-install the python package(`pip install -e`)
if you have made changes to publisher entry points.
Example::
[entry_points]
searchlight.publisher =
log_publisher = searchlight.publisher.log.LogPublisher
Publishers can be specified in a default configuration group of `[resource_plugin]`
in `searchlight.conf` or overridden in a specific plugin's configuration.
Example::
[resource_plugin]
publishers = log_publisher
Currently publishers only work for incremental updates. Bulk api updates are not
supported.
Global Configuration Options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+---------------------+---------------+-------------------------------------+---------------------------+
| Option | Default value | Description | Action(s) Required |
+=====================+===============+=====================================+===========================+
| resource_group_name | searchlight | Determines the ElasticSearch index | |
| | | and alias where documents will be | | Restart services |
| | | stored. Index names will be | | Re-index all types |
| | | suffixed with a timestamp. Group | |
| | | name must consist of only lowercase | |
| | | alphanumeric characters and | |
| | | underscores. The first character | |
| | | cannot be an underscore. | |
+---------------------+---------------+-------------------------------------+---------------------------+
| include_region_name | | Defined for all plugins. Controls | | Restart services |
| | | whether or not to include | | Reindex all |
| | | region_name as a mapping field and | |
| | | in each document. Defaults to off. | |
+---------------------+---------------+-------------------------------------+---------------------------+
.. note::
Sorting on fields across resource types(plugins), with each plugin specifying a different resource group
name will cause errors if sort-by fields are not defined in each resource type.
See :ref:`using-resource-groups` for more information on how to sort across different resource groups
Inheritable Common Configuration Options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+---------------------+---------------+-------------------------------------+---------------------------+
| Option | Default value | Description | Action(s) Required |
+=====================+===============+=====================================+===========================+
| mapping_use\_ | | Use doc_values to store documents | |
| doc_values | true | rather than fieldata. doc_values | | Full re-index |
| | | has some advantages, particularly | |
| | | around memory usage. | |
+---------------------+---------------+-------------------------------------+---------------------------+
| notifications_topic | notifications | The oslo.messaging topic on which | Restart listener |
| | | services send notifications. Each | |
| | | plugin defines a list of exchanges | |
| | | to which it will subscribe. | |
+---------------------+---------------+-------------------------------------+---------------------------+
| publishers | | Plugin can have multiple publishers | Restart listener |
| | | separated by commas. Each publisher | |
| | | will receive enriched notifications | |
| | | once plugin subscribed events come. | |
+---------------------+---------------+-------------------------------------+---------------------------+
Non-Inheritable Common Configuration Options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+---------------------+---------------+-------------------------------------+---------------------------+
| Option | Default value | Description | Action(s) Required |
+=====================+===============+=====================================+===========================+
| enabled | true | An installed plugin may be enabled | | Restart services |
| | | (true) or disabled (false). When | | Re-index affected types |
| | | disabled, it will not be available | |
| | | for bulk indexing, notification | |
| | | listening, or searching. | |
+---------------------+---------------+-------------------------------------+---------------------------+
| admin_only_fields | <none> | A comma separated list of fields | | Restart services |
| | | (wildcards allowed) that are only | | Re-index affected types |
| | | visible to administrators, and only | |
| | | searchable by administrators. Non- | |
| | | administrative users will not be | |
| | | able to see or search on these | |
| | | fields. | |
| | | These fields are typically | |
| | | specified for search performance, | |
| | | search accuracy, or security | |
| | | reasons. | |
| | | or security reasons. | |
| | | If a plugin has a hard-coded | |
| | | mapping for a specific field, it | |
| | | will take precedence over this | |
| | | configuration option. | |
+---------------------+---------------+-------------------------------------+---------------------------+
| notifications\_ | <none> | Override topic,exchange pairs (see | | Restart services |
| topics_exchanges | | note above). Use when services | |
| | | output notifications on dissimilar | |
| | | topics. | |
+---------------------+---------------+-------------------------------------+---------------------------+
| override_region_name| <none> | Specifies a region_name to be used | | Restart services |
| | | instead of one configured in | |
| | | service_credentials.os_region_name. | |
| | | Useful for multi-region deployments | |
| | | where a service is shared between | |
| | | regions. E.g. RegionOne,RegionTwo | |
+---------------------+---------------+-------------------------------------+---------------------------+
.. _individual-plugin-configuration:
Individual Plugin Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Individual plugins may also be configured in ``searchlight.conf``.
.. note::
Plugin configurations are typically named based on their resource type.
The configuration name uses the following naming pattern:
* The resource type name changed to all lower case
* All ``::`` (colons) converted into ``_`` (underscores).
For example: OS::Glance::Image --> [resource_plugin:os_glance_image]
To override a default configuration option on a specific plugin, you must
specify a configuration group for that plugin with the option(s) that you
want to override. For example, if you wanted to **just** disable the Glance
image plugin, you would add the following configuration group::
[resource_plugin:os_glance_image]
enabled = false
Each plugin may have additional configuration options specific to it.
Information about those configuration options will be found in documentation
for that plugin.
Finally, each integrated service (Glance, Nova, etc) may require
additional configuration settings. For example, typically, you will need
to add the ``searchlight_indexer`` notification topic to each service's
configuration in order for Searchlight to receive incremental updates from
that service.
.. note:: In Newton, notification messaging pools will become the default
recommended configuration, which does not require changing any
service configurations beyond enabling notifications.
To enable the use of notification pools instead of a separate
topic, add the ``notifications_pool`` option in the ``listener``
section of ``searchlight.conf``. There is no need in this case
to add an additional topic. Messages will begin to be delivered
to the pool after ``searchlight-listener`` has started.
Please review each plugin's documentation for more information:
.. toctree::
:maxdepth: 2
:glob:
../admin/plugins/*

12
doc/source/configuration/policy.rst
View File

@ -1,12 +0,0 @@
================================
Searchlight Policy Configuration
================================
Configuration
~~~~~~~~~~~~~
The following is an overview of all available policies in Searchlight.
For a sample configuration file, refer to :doc:`sample-policy-yaml`.
.. show-policy::
:config-file: ../../etc/oslo-policy-generator/searchlight.conf

8
doc/source/configuration/sample-policy-yaml.rst
View File

@ -1,8 +0,0 @@
===========
policy.yaml
===========
Use the ``policy.yaml`` file to define additional access controls that will be
applied to Searchlight:
.. literalinclude:: ../_static/searchlight.policy.yaml.sample

539
doc/source/contributor/authoring-plugins.rst
View File

@ -1,539 +0,0 @@
..
Copyright 2016 Hewlett-Packard Enterprise Development Company, L.P.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _searchlight-plugin-authoring:
Authoring Searchlight Plugins
=============================
At a bare minimum, a plugin must consist of an elasticsearch mapping, and a
method by which it can provide data to be indexed. Many plugins also require a
way to receive updates in order to keep the index up to date. For Openstack
resources, typically the service API is used for initial indexing and
notifications are received via oslo.messaging.
This documentation will use as an example the Neutron network plugin as a
reasonably complete and complex example.
Getting some data
-----------------
The very first thing you should do is figure out exactly what you're trying to
index. When I've developed plugins I've found it helpful to generate test data
both for initial indexing and for notifications.
Initial indexing
^^^^^^^^^^^^^^^^
In the case of neutron networks, the initial data will come from
``neutronclient``. Some browsing of the API documentation reveals that the
call I want is ``list_networks``::
import os
from oslo_serialization import jsonutils
from keystoneclient.auth.identity import v3
from keystoneclient import session
from neutronclient.v2_0 import client as nc_20
def get_session():
username = os.environ['OS_USERNAME']
password = os.environ['OS_PASSWORD']
auth_url = os.environ['OS_AUTH_URL']
tenant_name = os.environ['OS_TENANT_NAME']
auth = v3.Password(**locals())
return session.Session(auth=auth)
nc = nc_20.Client(session=get_session())
networks = nc.list_networks()
print(jsonutils.dumps(networks, indent=4, sort_keys=True))
This outputs::
{
"networks": [
{
"admin_state_up": true,
"availability_zone_hints": [],
"availability_zones": [
"nova"
],
"created_at": "2016-04-08T16:44:17",
"description": "",
"id": "4d73d257-35d5-4f4e-bc71-f7f629f21904",
"ipv4_address_scope": null,
"ipv6_address_scope": null,
"is_default": true,
"mtu": 1450,
"name": "public",
"port_security_enabled": true,
"provider:network_type": "vxlan",
"provider:physical_network": null,
"provider:segmentation_id": 1053,
"router:external": true,
"shared": false,
"status": "ACTIVE",
"subnets": [
"abcc5896-4844-4870-a5d8-6ae4b8edd42e",
"ea47304e-bd54-4337-901a-1eb5196ea18e"
],
"tags": [],
"tenant_id": "fa1537e9bda9405891d004ef9c08d0d1",
"updated_at": "2016-04-08T16:44:17"
}
]
}
Since that's the output from neutron client, that's what should go in
``searchlight/tests/functional/data/load/networks.json``, though you might
also want more examples to test different things.
Notifications
^^^^^^^^^^^^^
Openstack documents some of the notifications_ sent by some services. It's
also possible to eavesdrop on notifications sent by running services. Taking
neutron as an example (though all services are slightly different), we can
make it output notifications by editing ``/etc/neutron/neutron.conf`` and
adding under the ``[oslo_messaging_notifications]`` section::
driver = messagingv2
There are then two ways to configure the service to send notifications that
Searchlight can receive. The recommended method is to use notification pools,
touched on in the `messaging documentation`_.
.. _`messaging documentation`:
https://docs.openstack.org/oslo.messaging/latest/reference/notification_listener.html
Notification pools
##################
A notification messaging pool allows additional listeners to receive
messages on an existing topic. By default, Openstack services send notification
messages to an oslo.messaging 'topic' named `notifications`. To view these
notifications while still allowing ``searchlight-listener`` or Ceilometer's
agent to continue to receive them, you may use the utility script in
``test-scripts/listener.py``::
. ~/devstack/openrc admin admin
# If your rabbitmq user/pass are not the same as for devstack, you
# can set RABBIT_PASSWORD and/or RABBIT_USER
./test-scripts/listener.py neutron test-notifications
Adding a separate topic
#######################
In the same config file (``/etc/neutron/neutron.conf``) the following line
(again, under the ``[DEFAULT]`` section) will cause neutron to output
notifications to a topic named ``searchlight_indexer``::
notification_topics = searchlight_indexer
.. note::
``searchlight-listener`` also listens on the ``searchlight_indexer``
topic, so if you have ``searchlight-listener`` running, it will receive
and process some or all of the notifications you're trying to look at.
Thus, you should either stop the ``searchlight-listener`` or add another
topic (comma-separated) for the specific notifications you want to see.
For example::
notification_topics = searchlight_indexer,my_test_topic
After restarting the ``q-svc`` service notifications will be output to the
message bus (rabbitmq by default). They can be viewed in any RMQ management
tool; there is also a utility script in ``test-scripts/listener.py`` that
will listen for notifications::
. ~/devstack/openrc admin admin
# If your rabbitmq user/pass are not the same as for devstack, you
# can set RABBIT_PASSWORD and/or RABBIT_USER
./test-scripts/listener.py neutron
.. note::
If you added a custom topic as described above, you'll need to edit
``listener.py`` to use your custom topic::
# Change this line
topic = 'searchlight_indexer'
# to
topic = 'my_test_topic'
Using the results
#################
Issuing various commands (``neutron net-create``, ``neutron net-update``,
``neutron net-delete``) will cause ``listener.py`` to receive notifications.
Usually the notifications with ``event_type`` ending ``.end`` are the ones of
most interest (many fields omitted for brevity)::
{"event_type": "network.update.end",
"payload": {
"network": {
"status": "ACTIVE",
"router:external": false,
"subnets": ["9b6094de-18cb-46e1-8d51-e303ff844c86",
"face0b47-40d3-45c0-9b62-5f05311710f5",
"7b7bdf5f-8f22-44a3-bec3-1daa78df83c5"],
"updated_at": "2016-05-03T19:05:38",
"tenant_id": "34518c16d95e40a19b1a95c1916d8335",
"id": "abf3a939-4daf-4d05-8395-3ec735aa89fc", "name": "private"}
},
"publisher_id": "network.devstack",
"ctxt": {
"read_only": false,
"domain": null,
"project_name": "demo",
"user_id": "c714917a458e428fa5dc9b1b8aa0d4d6"
},
"metadata": {
"timestamp": "2016-05-03 19:05:38.258273",
"message_id": "ec9ac6a1-aa17-4ee3-aa6e-ab48c1fb81a8"
}
}
The entire message can go into
``searchlight/tests/functional/data/events/network.json``. The ``payload``
(in addition to the API response) will inform the mapping that should be
applied for a given plugin.
.. _notifications: https://wiki.openstack.org/wiki/SystemUsageData
File structure
--------------
Plugins live in ``searchlight/elasticsearch/plugins``. We have tended to create
a subpackage named after the service (``neutron``) and within it a module named
after the resource type (``networks.py``). Notification handlers can be in a file
specific to each resource type but can also be in a single file together
(existing ones use ``notification_handlers.py``).
``networks.py`` contains a class named ``NetworkIndex`` that implements the base
class ``IndexBase`` found in ``searchlight.elasticsearch.plugins.base``.
.. note::
If there are plugins for multiple resources within the same Openstack
service (for example, Glance images and meta definitions) those plugins
can exist in the same subpackage ('glance') in different modules, each
implementing an IndexBase.
Enabling plugins
----------------
Searchlight plugins are loaded by Stevedore_. In order for a plugin to be
enabled for indexing and searching, it's necessary to add an entry to the
``entry_points`` list in Searchlight's configuration in ``setup.cfg``. The
name should be the plugin resource name (typically the name used to represent
it in Heat_)::
[entry_points]
searchlight.index_backend =
os_neutron_net = searchlight.elasticsearch.plugins.neutron.networks:NetworkIndex
.. note::
After modifying entrypoints, you'll need to reinstall the searchlight
package to register them (you may need to activate your virtual environment;
see :ref:`Installation Instructions`)::
python setup.py develop
.. _Stevedore: https://docs.openstack.org/stevedore/latest/
.. _Heat: https://docs.openstack.org/heat/latest/template_guide/openstack.html
Writing some code
-----------------
At this point you're probably about ready to start filling in the code. My
usual approach is to create the unit test file first, and copy some of the
more boilerplate functionality from one of the other plugins.
You can run an individual test file with::
tox -epy34 searchlight.tests.unit.<your test module>
This has the advantage of running just your tests and executing them very
quickly. It can be easier to start from a full set of failing unit tests
and build up the actual code from there. Functional tests I've tended to add
later. Again, you can run an individual functional test file:
tox -epy34 searchlight.tests.functional.<your test module>
Required plugin functions
-------------------------
This section describes some of the functionality from ``IndexBase`` you will
need to override.
Document type
^^^^^^^^^^^^^
As a convention, plugins define their document type (which will map to an
ElasticSearch document type) as the `resource name`_ Heat uses to identify it::
@classmethod
def get_document_type(self):
return "OS::Neutron::Net"
.. _`resource name`: https://docs.openstack.org/heat/latest/template_guide/openstack.html
Retrieving object for initial indexing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Plugins must implement ``get_objects`` which in many cases will go to the
API of the service it's indexing. It should return an iterable that will be
passed to a function (also required) named ``serialize``, which in turn must
return a dictionary suitable for Elasticsearch to index. In the example for
Neutron networks, this would be a call to ``list_networks`` on an instance of
``neutronclient``::
def get_objects(self):
"""Generator that lists all networks owned by all tenants."""
# Neutronclient handles pagination itself; list_networks is a generator
neutron_client = openstack_clients.get_neutronclient()
for network in neutron_client.list_networks()['networks']:
yield network
Mapping
^^^^^^^
``get_mapping`` is also required. It must return a dictionary that tells
Elasticsearch how to map documents for the plugin (see the documentation for
mapping_).
At a minimum a plugin should define an ``id`` field and an ``updated_at`` field
because consumers will generally rely on those being present; a ``name`` field
is highly advisable. If the resource doesn"t contain these values your
``serialize`` function can map to them. In particular, if your resource does
not have a native ``id`` value, you must override ``get_document_id_field``
so that the indexing code can retrieve the correct value when indexing.
It is worth understanding how Elasticsearch indexes various field types,
particularly strings. String fields are typically broken down into tokens to
allow searching::
"The quick brown fox" -> ["The", "quick", "brown", "fox"]
This works well for full-text type documents but less well, for example,
for UUIDS::
"aaab-bbbb-55555555" -> ["aaab", "bbbb", "55555555"]
In the second example, a search for the full UUID will not match. As a result,
we tend to mark these kinds of fields as ``not_analyzed`` as with the example
to follow.
Where field types are not specified, Elasticsearch will make a best guess from
the first document that's indexed.
Some notes (expressed below as comments starting with #)::
{
# This allows indexing of fields not specified in the mapping doc
"dynamic": true,
"properties": {
# not_analyzed is important for id fields; it prevents Elasticsearch
# tokenizing the field, allowing for exact matches
"id": {"type": "string", "index": "not_analyzed"},
# This allows name to be tokenized for searching, but Searchlight will
# attempt to use the 'raw' (untokenized) field for sorting which gives
# more consistent results
"name": {
"type": "string",
"fields": {
"raw": {"type": "string", "index": "not_analyzed"}
}
}
}
}
If you are mapping a field which is a reference id to other plugin type, you
should add a _meta mapping for that field. This will enable Searchlight(SL) to
provide more information to CLI/UI. The reference id and the plugin resource
type can be used by CLI/UI to issue a ``GET`` request to fetch more information
from SL. See below for an example on nova server plugin mapping::
def get_mapping(self):
return {
'dynamic': True,
'properties': {
'id': {'type': 'string', 'index': 'not_analyzed'},
'name': {
'type': 'string',
'fields': {
'raw': {'type': 'string', 'index': 'not_analyzed'}
}
}
'image': {
'type': 'nested',
'properties': {
'id': {'type': 'string', 'index': 'not_analyzed'}
}
}
},
"_meta": {
"image.id": {
"resource_type": resource_types.GLANCE_IMAGE
}
},
}
.. note:: Parent plugin id field(when available) is automatically linked to the
parent resource type.
Doc values
^^^^^^^^^^
For many field types Searchlight will alter the mapping to change the format in
which field data is stored. Prior to Elasticsearch 2.x field values by default
were stored in 'fielddata' format, which could result in high memory usage under
some sort and aggregation operations. An alternative format, called ``doc_values``
trades slightly increased disk usage for better memory efficiency. In Elasticsearch
2.x ``doc_values`` is the default, and Searchlight uses this option as the default
regardless of Elasticsearch version. For more information see the Elasticsearch
documentation_.
.. _documentation: https://www.elastic.co/guide/en/elasticsearch/reference/2.1/doc-values.html
Generally this default will be fine. However, there are several ways in which
the default can be overridden:
* Globally in plugin configuration; in ``searchlight.conf``::
[resource_plugin]
mapping_use_doc_values = false
* For an individual plugin in ``searchlight.conf``::
[resource_plugin:os_neutron_net]
mapping_use_doc_values = false
* For a plugin's entire mapping; in code, override the ``mapping_use_doc_values``
property (and thus ignoring any configuration property)::
@property
def mapping_use_doc_values(self):
return False
* For individual fields in a mapping, by setting ``doc_values`` to False::
{
"properties": {
"some_field": {"type": "date", "doc_values": False}
}
}
Access control
^^^^^^^^^^^^^^
Plugins must define how they are access controlled. Typically this is a
restriction matching the user's project/tenant::
def _get_rbac_field_filters(self, request_context):
return [
{'term': {'tenant_id': request_context.owner}}
]
Any filters listed will be applied to queries against the plugin's document
type. A document will match the RBAC filters if any of the clauses match.
Administrative users can specify ``all_projects`` in searches to bypass
these filters. This default behavior can be overridden for a plugin by setting
the ``allow_admin_ignore_rbac`` property to ``False`` on the plugin (currently
only in code). ``all_projects`` will be ignore for that plugin.
Policy
^^^^^^
Related to access control is policy. Most services control API access with
policy files that define rules enforced with `oslo.policy`_. Searchlight has
its own policy file that configures access to its own API and resources, but
it also supports reading other services' policy files. In the future this will
be expanded to define RBAC rules, but at present external policy files are
only used to determine whether a resource should be available to a user.
To support this in your plugin, you must define two properties. The first
is ``service_type`` which must correspond to the service 'type' as seen
in the keystone catalog (e.g. nova's service 'type' is 'compute'). The
second property is ``resource_allowed_policy_target`` which identifies the
rule name in the service's policy files. If either of these properties are
'None' no rule will be enforced.
For example::
@property
def resource_allowed_policy_target(self):
return 'os_compute_api:servers:index'
@property
def service_type(self):
return 'compute'
See :ref:`service-policy-controls` for configuration information.
.. _oslo.policy: https://docs.openstack.org/oslo.policy/latest/
Faceting
^^^^^^^^
Any fields defined in the mapping document are eligible to be identified as
facets, which allows a UI to let users search on specific fields. Many plugins
define ``facets_excluded`` which exclude specified fields. Many also define
``facets_with_options`` which should return fields with low cardinality where
it makes sense to return valid options for those fields.
Protected fields
^^^^^^^^^^^^^^^^
``admin_only_fields`` determines fields which only administrators should be
able to see or search. For instance, this will mark any fields beginning with
``provider:`` as well as any defined in the plugin configuration::
@property
def admin_only_fields(self):
from_conf = super(NetworkIndex, self).admin_only_fields
return ['provider:*'] + from_conf
These fields end up getting indexed in separate admin-only documents.
Parent/child relationships
--------------------------
In some cases there is a strong ownership implied between plugins. In these
cases the child plugin can define ``parent_plugin_type`` and
``get_parent_id_field`` (which determines a field on the child that refers
to its parent). See the Neutron ``Port`` plugin for an example.
Remember that Elasticsearch is not a relational database and it doesn't do
joins, per se, but this linkage does allow running queries referencing children
(or parents).
.. _mapping: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
Pipeline architecture
---------------------
Notification handlers can emit enriched resource data into pipeline, configured
publishers could use these data to notify external systems. To use this feature,
each event handler should return one or a sequence of pipeline items. These items
will be passed to subscribed publshers::
def create_or_update(self, event_type, payload, timestamp):
network_id = payload['network']['id']
LOG.debug("Updating network information for %s", network_id)
network = serialize_network(payload['network'])
version = self.get_version(network, timestamp)
self.index_helper.save_document(network, version=version)
return pipeline.IndexItem(self.index_helper.plugin,
event_type,
payload,

88
doc/source/contributor/contributing.rst
View File

@ -1,88 +0,0 @@
============================
So You Want to Contribute...
============================
For general information on contributing to OpenStack, please check out the
`contributor guide <https://docs.openstack.org/contributors/>`_ to get started.
It covers all the basics that are common to all OpenStack projects: the accounts
you need, the basics of interacting with our Gerrit review system, how we
communicate as a community, etc.
Below will cover the more project specific information you need to get started
with Searchlight.
Communication
~~~~~~~~~~~~~~
.. This would be a good place to put the channel you chat in as a project; when/
where your meeting is, the tags you prepend to your ML threads, etc.
- IRC channel: #openstack-searchlight
- Mailing list's prefix: [searchlight]
- Meeting Times: http://eavesdrop.openstack.org/#Searchlight_Team_Meeting
- Meeting Agenda: https://etherpad.openstack.org/p/search-team-meeting-agenda
- Meeting Logs: http://eavesdrop.openstack.org/meetings/openstack_search
Contacting the Core Team
~~~~~~~~~~~~~~~~~~~~~~~~~
.. This section should list the core team, their irc nicks, emails, timezones etc. If
all this info is maintained elsewhere (i.e. a wiki), you can link to that instead of
enumerating everyone here.
- Trinh Nguyen <dangtrinhnt@gmail.com> (PTL) - dangtrinhnt - GMT+9
- Thuy Dang <thuydang.de@gmail.com> - thuydang - GMT+1
New Feature Planning
~~~~~~~~~~~~~~~~~~~~
.. This section is for talking about the process to get a new feature in. Some
projects use blueprints, some want specs, some want both! Some projects
stick to a strict schedule when selecting what new features will be reviewed
for a release.
1. Talk to the team via IRC (meeting) or ML (with [searchlight] prefix) about
the feature you want to add. We will discuss and figure out what is the best
way forward considering our plan for the development cycle.
2. After the team has discussed and everyone agreed to have your feature
landed in the current cycle, you can propose a blueprint document describing your
feature in details (e.g., architecture, components, usage, etc.)
3. The team will review your blueprint and make comments until it is good enough
to start implementing the feature.
4. You implement the feature based on the design in the blueprint.
5. Searchlight team will again review your code and approve it.
Task Tracking
~~~~~~~~~~~~~~
.. This section is about where you track tasks- launchpad? storyboard? is there more
than one launchpad project? what's the name of the project group in storyboard?
We track our tasks in `Storyboard <https://storyboard.openstack.org/#!/project/openstack/searchlight>`_
If you're looking for some smaller, easier work item to pick up and get started
on, search for the 'low-hanging-fruit' tag.
.. NOTE: If your tag is not 'low-hanging-fruit' please change the text above.
Reporting a Bug
~~~~~~~~~~~~~~~
.. Pretty self explanatory section, link directly to where people should report bugs for
your project.
You found an issue and want to make sure we are aware of it? You can do so
`HERE <https://storyboard.openstack.org/#!/project/openstack/searchlight>`_.
Getting Your Patch Merged
~~~~~~~~~~~~~~~~~~~~~~~~~
.. This section should have info about what it takes to get something merged. Do
you require one or two +2's before +W? Do some of your repos require unit test
changes with all patches? etc.
Due to the small number of core reviewers of the Searchlight project, we only need
one +2 before +W (merge). All patches excepting for documentation or typos fixes
must have unit test.
Project Team Lead Duties
------------------------
.. this section is where you can put PTL specific duties not already listed in
the common PTL guide (linked below) or if you already have them written
up elsewhere, you can link to that doc here.
All common PTL duties are enumerated here in the `PTL guide <https://docs.openstack.org/project-team-guide/ptl.html>`_.

186
doc/source/contributor/feature-requests-bugs.rst
View File

@ -1,186 +0,0 @@
..
c) Copyright 2015 Hewlett-Packard Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
================================
Feature Requests and Bug Reports
================================
Searchlight is an open project and we encourage contribution from everybody.
We support both developers and non-developers who want to provide input,
requests for features, and bug fixes. We want to be able to move quickly
without getting too bogged down in process, but still provide a rich mechanism
for feature reviews as needed.
Workflow
========
Our process is meant to allow users, developers, and operators to express
their desires for new features using Storyboard blueprints. A review of
blueprints is done regularly. These may turn directly into features, or
for complex requests, additional specifications ("specs") may be needed.
The workflow is very simple:
* If something is clearly broken, submit a `bug report`_ in Storyboard.
* If you want to change or add a feature, submit a `blueprint`_ in Storyboard.
* Searchlight drivers may request that you submit a `specification`_ to gerrit to elaborate on the feature request
* Significant features require `Release Notes`_ to be included when the code is merged
.. note::
If you already have code to submit, go ahead and submit a gerrit review!
We encourage early code sharing. Just be aware that it needs to be cross
linked with a bug or blueprint and may not be accepted without approval
of the blueprint or bug. We also use the blueprint and bug priorities
to guide our code review priorities.
We will review incoming bugs and blueprints on an ongoing basis. It is
always okay to ask for feedback on a bug or blueprint that has been submitted.
We always welcome you in the IRC channel, the mailing list, and the weekly
meeting. We just ask that you understand that there may be many reviews
happening at any point in time and it may take a little time for reviews to be
completed.
.. _bug report:
Bug Reports
-----------
Current Bugs are found here:
* https://storyboard.openstack.org/#!/project_group/93
A bug may be filed by adding a story.
Please provide information on what the problem is, how to replicate it,
any suggestions for fixing it, and a recommendation of the priority.
Security Bugs
~~~~~~~~~~~~~
Reporting bugs referencing security related vulnerabilities in released
versions of Searchlight requires additional consideration. A bug with security
implications should be filed as a private security bug. This prevents public
disclosure of potential security issues before they can be addressed.
To mark a bug as a private security bug, set the value for the field "This bug
contains information that is:" from "Public" to "Private Security". If you have
questions, please contact either of the following groups:
* `Searchlight Core Security Team <https://launchpad.net/~searchlight-coresec>`_
* `OpenStack Vulnerability Management Process <https://security.openstack.org/vmt-process.html>`_
In the event that a bug filed as a private security bug is determined not to
have security implications, the bug will be moved to a public bug report.
.. _blueprint:
Blueprints
----------
Current blueprints are found here:
* https://storyboard.openstack.org/#!/project_group/93
A blueprint may be filed by adding a story.
The initial blueprint primarily needs to express the intent of the idea with
enough details that the idea can be evaluated for compatibility with the
Searchlight mission and whether or not the change requires a
`specification`_ for change tracked reviews. It is *not*
expected to contain all of the implementation details. If the feature
is very simple and well understood by the team, then describe it simply.
Searchlight team members will request more information as needed.
If the blueprint starts to seem non-trivial, or seem like it will benefit
from a better tool for comments and change tracking, then you should
submit a `specification`_ to gerrit proactively and simply
link to it from your blueprint to accommodate better reviews.
.. _specification:
Specifications
--------------
We use the `searchlight-specs
<https://opendev.org/openstack/searchlight-specs>`_ repository for
specification reviews. Specifications:
* Provide a review tool for collaborating on feedback and reviews for complex features
* Serve as the basis for documenting the feature once implemented
* Ensure that the overall impact on the system is considered
The Searchlight team does not enforce deadlines for specs. These can be submitted
throughout the release cycle. The drivers team will review this on a regular
basis throughout the release, and based on the load for the milestones, will
assign these into milestones or move them to the backlog for selection into
a future release.
Please note that we use a `template
<https://opendev.org/openstack/searchlight-specs/src/branch/master/specs/template.rst>`_
for spec submissions. It is not required to fill out all sections in the
template. Review of the spec may require filling in information left out by
the submitter.
The review system will run a few tests to check the basic format and
syntax of your spec. You will just need to run ``tox`` locally from within
the checked out spec repository to replicate the review tests.
Release Notes
=============
The release notes for a patch should be included in the patch. If not, the
release notes should be in a follow-on review.
If any of the following applies to the patch, a release note is required:
* The deployer needs to take an action when upgrading
* A new feature is implemented
* Plugin API function was removed or changed
* Current behavior is changed
* A new config option is added that the deployer should consider changing from
the default
* A security bug is fixed
* Change may break previous versions of the client library(ies)
* Requirement changes are introduced for important libraries like oslo, six
requests, etc.
* Deprecation period starts or code is purged
A release note is suggested if a long-standing or important bug is fixed.
Otherwise, a release note is not required.
Searchlight uses `reno <https://docs.openstack.org/reno/latest/user/usage.html>`_ to
generate release notes. Please read the docs for details. In summary, use
the following:
.. code-block:: bash
$ tox -e venv -- reno new <bug-,bp-,whatever>
Then edit the sample file that was created and push it with your change.
To see the results:
.. code-block:: bash
$ git commit # Commit the change because reno scans git log.
$ tox -e releasenotes
Then look at the generated release notes files in releasenotes/build/html in
your favorite browser.

12
doc/source/contributor/index.rst
View File

@ -1,12 +0,0 @@
==================
Contribution guide
==================
.. toctree::
:maxdepth: 2
contributing
searchlight-vision
vision-reflection
authoring-plugins
feature-requests-bugs

52
doc/source/contributor/searchlight-vision.rst
View File

@ -1,52 +0,0 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Searchlight Team Vision
=======================
Searchlight was originally developed and released in the Kilo release of
Glance as the Catalog Index Service [#]_. At the Liberty Summit, we decided
to broaden the scope to provide advanced and scalable search across
multi-tenant cloud resources. And in the Stein cycle, we have developed a
grand vision for Searchlight to help sustain the project in alignment with
the OpenStack Cloud Vision [#]_.
Our Vision
----------
With the modular architecture of Searchlight and based on the discussions of
the Searchlight team, we envisioned making Searchlight a universal search
interface not only for OpenStack but also other cloud platforms such as
Microsoft Azure [#]_, AWS [#]_, or even Kubernetes [#]_, etc.. The final
product of this vision could be building a multi-cloud management application
and a unified API for multi-cloud resource discovery, which serves as a
cloud information base for automation application, e.g., VIM management,
NFV MANO. This requires new designs of additional data models, APIs,
communication, and features which are analyzed further in specific use
case analysis & design documents. References will be provided when available.
.. figure:: ../../../images/SeaaS.png
:width: 100%
:alt: Search as a Service
References
----------
.. [#] http://specs.openstack.org/openstack/glance-specs/specs/kilo/catalog-index-service.html
.. [#] https://governance.openstack.org/tc/reference/technical-vision.html
.. [#] https://storyboard.openstack.org/#!/story/2004718
.. [#] https://storyboard.openstack.org/#!/story/2004719
.. [#] https://storyboard.openstack.org/#!/story/2004382

77
doc/source/contributor/vision-reflection.rst
View File

@ -1,77 +0,0 @@
===========================================================
Searchlight reflects on the 2018 OpenStack Technical Vision
===========================================================
In late-2018, the OpenStack Technical Committee composed a technical vision
[#]_ of what OpenStack clouds should look like. This document compares the
state of Searchlight relative to that vision to provide some guidance on
broad-stroke ways in which Searchlight may need to change to match the vision.
Note that there is also a Searchlight Vision document [#]_.
The TC vision document is divided into three sections, which this document
mirrors. This should be a living document which evolves as Searchlight itself
evolves.
The Pillars of Cloud
====================
While the Searchlight API performs read-only accesses of Elasticsearch for
user-facing features (i.e., search), the Searchlight Amin CLI provides
read-write access to Elasticsearch for administrative tasks (e.g., re-index
data, etc.), so basically, anything can talk to it, enabling the self-service
and application control features that define a cloud.
OpenStack-specific Considerations
=================================
Interoperability
----------------
Because Searchlight only has Keystone as a hard-dependency, Searchlight
deployment can be ported from one OpenStack cloud to another easily with
minimal modification of the settings depends on the services Searchlight want
to index.
Bidirectional Compatibility
---------------------------
Searchlight APIs is versioned and designed to allow client introspection of
the available versions and features.
Cross-Project Dependencies
--------------------------
Searchlight depends on Keystone to have universal access to other OpenStack
services and user authentication. Other services that Searchlight supports
(e.g., Nova, Cinder, Glance, etc.) are soft dependencies.
Partitioning
------------
At the current state, Searchlight only supports indexing resource information
of one single OpenStack instance where Searchlight deployed. In the Stein
cycle, we have designed a vision that is to make Searchlight a multi-cloud
solution. It means that Searchlight will be able to search resource
information across different cloud platforms and tenants. Moreover, by going
in that direction, Searchlight would offer features such as tagging and or
partitioning resource information arbitrarily.
Design Goals
============
Searchlight already maps well to most of the design goals in the TC vision
document such as scalability, reliability, customization, and flexible
utilization models. Searchlight also offers a graphical user interface to
query the indexed resources. We should strive to keep this. Details of how we
plan to do so should be maintained in the Searchlight Vision document.
References
==========
.. [#] https://governance.openstack.org/tc/reference/technical-vision.html
.. [#] https://docs.openstack.org/searchlight/latest/contributor/searchlight-vision.html

113
doc/source/index.rst
View File

@ -1,113 +0,0 @@
..
Copyright 2010 OpenStack Foundation
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Welcome to Searchlight's documentation!
=======================================
The Searchlight project provides indexing and search capabilities across
OpenStack resources. Its goal is to achieve high performance and flexible
querying combined with near real-time indexing. It uses
`Elasticsearch <http://http://www.elasticsearch.org/>`_, a real-time
distributed indexing and search engine built on Apache Lucene, but adds
OpenStack authentication and Role Based Access Control to provide appropriate
protection of data.
Searchlight, as with all OpenStack projects, is written with the following design
guidelines in mind:
* **Microservice based architecture**: Quickly add new behaviors
* **Highly available**: Scale to very serious workloads
* **Fault tolerant**: Isolated processes avoid cascading failures
* **Recoverable**: Failures should be easy to diagnose, debug, and rectify
* **Open standards**: Be a reference implementation for a community-driven api
This documentation is generated by the Sphinx toolkit using `tox -e docs` and
lives in the source tree.
Additional documentation on Searchlight can be found on the `OpenStack wiki`_.
.. _`OpenStack wiki`: https://wiki.openstack.org/wiki/Searchlight
Indexes
-------
.. toctree::
:maxdepth: 2
user/index
admin/index
install/index
configuration/index
contributor/index
cli/index
Contributing
------------
.. toctree::
:maxdepth: 2
contributor/feature-requests-bugs
contributor/authoring-plugins
Overview
--------
.. toctree::
:maxdepth: 2
admin/architecture
user/searchlightapi
admin/indexingservice
configuration/authentication
Client and UI
^^^^^^^^^^^^^
* `Horizon UI Plugin`_
* `OpenStack Client Plugin`_
.. _`Horizon UI Plugin`: https://opendev.org/openstack/searchlight-ui
.. _`OpenStack Client Plugin`: https://opendev.org/openstack/python-searchlightclient
Search plugins
--------------
The search service determines the type of information that is indexed and
searchable via a plugin mechanism.
.. toctree::
:maxdepth: 2
configuration/plugins
contributor/authoring-plugins
See each plugin below for detailed information about specific plugins:
.. toctree::
:maxdepth: 2
:glob:
admin/plugins/*
Installation and Setup
----------------------
.. toctree::
:maxdepth: 2
install/dev-environment
install/elasticsearch
install/uwsgi

394
doc/source/install/dev-environment.rst
View File

@ -1,394 +0,0 @@
..
c) Copyright 2015 Hewlett-Packard Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
***********************
Development Environment
***********************
This guide will walk you through setting up a typical development
environment. You may set it up using devstack or manually.
.. _Devstack Development Environment:
Devstack Development Environment
++++++++++++++++++++++++++++++++
Please see: https://opendev.org/openstack/searchlight/src/branch/master/devstack
.. _Manual Development Environment:
Manual Development Environment
++++++++++++++++++++++++++++++
Prerequisites
=============
Install prerequisites::
# Ubuntu/Debian:
sudo apt-get update
sudo apt-get install python-dev libssl-dev python-pip libxml2-dev \
libxslt-dev git git-review libffi-dev gettext \
python-tox
# Fedora/RHEL:
sudo yum install python-devel openssl-devel python-pip libxml2-devel \
libxslt-devel git git-review libffi-devel gettext
# openSUSE/SLE 12:
sudo zypper install git git-review libffi-devel \
libopenssl-devel libxml2-devel libxslt-devel \
python-devel python-flake8 \
python-nose python-pip python-setuptools-git \
python-testrepository python-tox python-virtualenv \
gettext-runtime
sudo easy_install nose
sudo pip install virtualenv setuptools-git flake8 tox testrepository
If using RHEL and yum reports "No package python-pip available" and "No
package git-review available", use the EPEL software repository. Instructions
can be found at `<http://fedoraproject.org/wiki/EPEL/FAQ#howtouse>`_.
You may need to explicitly upgrade virtualenv if you've installed the one
from your OS distribution and it is too old (tox will complain). You can
upgrade it individually, if you need to::
sudo pip install -U virtualenv
This guide assumes that you also have the following services
minimally available:
* Elasticsearch (See :doc:`elasticsearch` for install information).
* Keystone
.. note::
Some Elasticsearch installation methods may call for installing a
headless JRE. If you are working on a desktop based OS (such as Ubuntu
14.04), this may cause tools like pycharms to no longer launch. You can
switch between JREs and back: to a headed JRE version using:
"sudo update-alternatives --config java".
Additional services required to be installed will depend on the plugins
activated in searchlight.
.. _Installation Instructions:
Installing Searchlight
======================
.. index::
double: install; searchlight
1. Clone the Searchlight repo from opendev.org
::
$ mkdir openstack
$ cd openstack
$ git clone https://opendev.org/openstack/searchlight.git
$ cd searchlight
2. Setup a virtualenv
.. note::
This is an optional step, but will allow Searchlight's dependencies
to be installed in a contained environment that can be easily deleted
if you choose to start over or uninstall Searchlight.
::
$ tox -evenv --notest
Activate the virtual environment whenever you want to work in it.
All further commands in this section should be run with the venv active:
::
$ . .tox/venv/bin/activate
.. note::
When ALL steps are complete, deactivate the virtualenv: $ deactivate
4. Install Searchlight and its dependencies
::
(venv) $ python setup.py develop
5. Generate sample config.
::
(venv) $ oslo-config-generator --config-file
etc/oslo-config-generator/searchlight.conf
6. Create Searchlight's config files by copying the sample config files
::
$ cd etc/
$ ls *.sample | while read f; do cp $f $(echo $f | sed "s/.sample$//g"); done
7. Make the directory for Searchlight's log files
::
$ mkdir -p ../log
8. Make the directory for Searchlight's state files
::
$ mkdir -p ../state
Configuring Searchlight
=======================
.. index::
double: configure; searchlight
Searchlight has several configuration files. The following are the basic
configuration suggestions.
Keystone integration
--------------------
Keystone integration should be set up for proper authentication and service
integration
.. toctree::
:maxdepth: 2
../configuration/authentication
Other development environment configuration
-------------------------------------------
Additional development environment configuration items are specified below.
searchlight.conf
````````````````
::
[DEFAULT]
debug = true
log_file = log/searchlight.log
Plugin Configuration
--------------------
The search service is driven using a plugin mechanism for integrating to other
services. Each integrated service may require additional configuration
settings. For example, you typically will need to enable the
``oslo_messaging_notifications`` messaging driver and may need to add
the ``notifications`` topic to each service's configuration.
Searchlight uses notification messaging pools. This usually does not
require changing any service configurations beyond enabling the notifications
driver. Searchlight uses sensible defaults for most deployments, but if you
want to customize the settings see :ref:`searchlight-plugins` for plugin
installation and general configuration information.
See each plugin below for detailed information about specific plugins:
.. toctree::
:maxdepth: 2
:glob:
../admin/plugins/*
Initialize the Elasticsearch Index
==================================
.. index::
double: install; sync
Perform initial sync of the resources to Elasticsearch. All plugins for
searchlight must have their services installed and available at an endpoint
registered in keystone (e.g. glance).
::
$ cd <install dir. eg: openstack/searchlight>
# Make sure your virtualenv is sourced
$ . .tox/venv/bin/activate
# Run the index operation.
(venv) $ searchlight-manage index sync
# Alternatively, you can directly invoke the command using the following.
(venv) $ python searchlight/cmd/manage.py --config-file etc/searchlight.conf index sync
This command may be re-run at any time to perform a full re-index.
.. IMPORTANT:: You must perform initial sync to create a new index in Elasticsearch,
even if there are no resources to sync in your environment.
The V2 Elasticsearch Client
---------------------------
.. IMPORTANT:: Please read if you receive a warning about using the
elasticsearch v2 client when running `index sync`
The v2 elasticsearch client removed functionality Searchlight uses to
clear existing data. If the v2 client is installed, you will receive a
warning when attempting to index data. A workaround is to run the
`index sync` command below with the `--no-delete` flag. If you have
existing data indexed, at present it is necessary to delete the entire
index and reindex. You can do this by running
::
# Assume elasticsearch is running on localhost, and the default
# 'searchlight' index is in use
curl -X DELETE localhost:9200/searchlight
(venv) $ searchlight-manage --config-file etc/searchlight.conf index sync --no-delete
Alternatively, install the Elasticsearch 1.9.0 client by editing
`requirements.txt` and making the following change
::
# Change THIS
elasticsearch>=1.3.0
# To THIS
elasticsearch>=1.3.0,<2.0.0
Then re-install requirements and index
::
(venv) $ pip install -r requirements.txt
(venv) $ searchlight-manage --config-file etc/searchlight.conf index sync
Note that if you are running a version 2 elasticsearch *server*, the 1.x
*client* will not work and you must follow the workaround above.
Start Index Update Monitoring
=============================
The index is updated continually based on updates to the source resource
data. Start this service to start update monitoring. Note, depending on the
resource type, this will typically require that you have configured
notifications properly for the service which owns the resource (e.g. Glance
images).
::
$ cd <install dir. eg: openstack/searchlight>
# Make sure your virtualenv is sourced
$ . .tox/venv/bin/activate
# Start the index update monitoring.
(venv) $ searchlight-listener --config-file etc/searchlight.conf
# Alternatively, you can directly invoke the command using the following.
(venv) $ python searchlight/cmd/listener.py --config-file
etc/searchlight.conf
Initialize & Start the API Service
==================================
.. index::
double: install; api
Open up a new ssh window and log in to your server (or however you're
communicating with your server).
::
$ cd <install dir. eg: openstack/searchlight>
# Make sure your virtualenv is sourced
$ . .tox/venv/bin/activate
# Start the API Service.
(venv) $ searchlight-api --config-file etc/searchlight.conf
# Alternatively, you can directly invoke the command using the following.
(venv) $ python searchlight/cmd/api.py --config-file etc/searchlight.conf
You should now see the log from the API service.
Exercising the API
==================
.. note:: If you have a firewall enabled, make sure to open port 9393.
Using a web browser, curl statement, or a REST client, calls can be made to the
Searchlight API using the following format where "api_version" is v1
and "command" is any of the commands listed under the :doc:`../user/searchlightapi`
::
http://IP.Address:9393/api_version/command
Example: List plugins::
$ curl http://localhost:9393/v1/search/plugins
Example: List all data::
# Prerequisite Setup:
$ token=<insert token>
$ touch search.json
# Paste content
{
"query": {
"match_all": {}
},
"limit": 1000
}
# Execute query
$ curl -X POST http://localhost:9393/v1/search -H "X-Auth-Token:$token" \
-d @search.json -H "Content-Type: application/json" | python -mjson.tool
You can find the IP Address of your server by running
::
curl -s checkip.dyndns.org | sed -e 's/.*Current IP Address: //' -e 's/<.*$//'
Troubleshooting
===============
Elasticsearch:
You can directly connect to the Elasticsearch server and examine
its contents. We recommend using the Sense extension for google chrome.
Notifications:
Use rabbitmqctl to examine unconsumed notifications::
sudo rabbitmqctl list_queues | grep info
There are also a number of management tools available to help with
troubleshooting.
Please see: https://www.rabbitmq.com/management.html
If you have searchlight setup to share a notification topic, but do
not have a notification pool configured, then notifications may be consumed
by other listeners and searchlight will not receive the notifications.

246
doc/source/install/elasticsearch.rst
View File

@ -1,246 +0,0 @@
..
Copyright (c) 2015 Hewlett-Packard Development Company, L.P.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Installing and Configuring Elasticsearch
========================================
The Searchlight indexing service is responsible for indexing data in
Elasticsearch [1]_; Elasticsearch has very good documentation on
installation but some pointers are provided here.
.. IMPORTANT:: We *strongly* recommend using ElasticSearch 5.x and the
accompanying python client version [2]_. ElasticSearch 2.x is still
supported and will be removed after Stein cycle.
Installation
~~~~~~~~~~~~
Elasticsearch requires a Java Runtime Environment (or Java Development Kit). OpenJDK
and Oracle's Java are supported. Information on the current recommended version can
be found in the installation instructions [3]_.
Installing from a download package
##################################
Links to various formats and also older versions of Elasticsearch can be found
on the download page [4]_. Once downloaded and extracted, you can start
Elasticsearch with:
.. code-block:: console
bin/elasticsearch
For more details see the installation instructions [5]_
Quick command line example with 5.6.11:
.. note::
Do the following commands as "root" or via sudo <command>
Download the ES package:
.. code-block:: console
cd ~
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.11.deb
sudo dpkg -i elasticsearch-5.6.11.deb
sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start
.. note::
If you install ElasticSearch 5.x, You should install
python-elasticsearch>=5.0.0. global-requirements currently constrains it to v2.x
Configuration
~~~~~~~~~~~~~
Elasticsearch comes with very a very sensible default configuration that
allows for clustering and high performance out of the box. There are some
settings, both general and specific to Searchlight's indexing service, that might
be of interest depending on your scenario.
Elasticsearch's configuration is in $ES_HOME/config/elasticsearch.yaml, where
ES_HOME is the directory in which Elasticsearch is installed. It uses YAML,
a superset of JSON.
Indices
#######
Elasticsearch (and Lucene) store information in indices. Within an index can
be one or more document types. Searchlight's indexing service uses an index
per service that has a plugin available, and each plugin generally will have
its own document type. For instance, the glance plugin has *glance.image* and
*glance.metadef*. Since the volume of data is lower than a typical use case for
Elasticsearch it may make sense to change the default sharing and replication
mechanism. We also recommend disabling implicit index creation, though if you
are sharing an Elasticsearch installation this may be inadvisable. The
following options control indexing behavior:
.. code-block:: yaml
# Number of shards for each index (performance)
index.number_of_shards: 5
# Number of replicas per shard (redundancy and recovering)
index.number_of_replicas: 1
# Disable automatic index creation so that index creation
# is an explicit action
action.auto_create_index: false
Index settings
**************
In addition to server-wide index settings it's possible to configure
Searchlight to apply settings to indices it creates with
``searchlight-manage``. Index settings can be specified as follows in
``searchlight.conf``:
.. code-block:: yaml
[elasticsearch]
index_settings = refresh_interval:2s,number_of_replicas:1
The ``index.`` prefix for settings is optional; Searchlight will prepend it if
it's not given (e.g. ``index.refresh_interval`` is also acceptable).
Index settings are applied at creation time and so are not limited to the
'dynamic' index settings. They are applied to all indices at the time they
are created. If you wish to update settings for an existing index, you
should use the Elasticsearch API to do so or reindex.
See also [9]_, [10]_.
Scripts
#######
The scripting module allows to use scripts in order to evaluate custom
expressions. Scripting is turned off by default in elasticsearch latest
versions. Searchlight doesn't allow scripts in the search api but requires
scripts to sync Index updates from notifications. For security purpose index
updates are allowed only for admin role.
.. code-block:: yaml
script.engine.groovy.inline.update: on
See also [8]_.
Development
###########
For development, Elasticsearch's default configuration is overkill. It's
possible to run Elasticsearch with a much lower memory footprint than by
default, and you may wish to disable clustering behavior.
.. code-block:: yaml
# Configures elasticsearch as a single node (no discovery)
node.local: true
# Disable sharding and replication
index.number_of_shards: 1
index.number_of_replicas: 0
JVM settings
************
Setting the ES_HEAP_SIZE environment variable will restrict how much memory
Elasticsearch uses, equivalent to setting -Xmx and -Xms to the same value for
the Java runtime. For development you can set it as low as a few tens of MB:
.. code-block:: console
export ES_HEAP_SIZE=40m
Memory usage will be somewhat higher than that figure, because Java itself
requires memory on top of that.
Production
##########
Some settings you may wish to change for production:
.. code-block:: yaml
# Cluster name is used by cluster discovery; it's important to ensure
# this is set across all nodes you wish to be in the cluster
cluster.name: searchlight
# By default elasticsearch picks a random name from a list of Marvel
# comic characters. If you specify this, make sure it's different on
# each node in the cluster
node.name: This Node Name
# Bind to a non-standard address
network.host: 0.0.0.0
# Bind to a non-standard port
http.port: 9200
# Configure the default data and log directories. By default, these
# directories will be created in $ES_HOME.
path:
logs: /var/log/elasticsearch
data: /var/data/elasticsearch
# This setting locks the Elasticsearch process address space into RAM
# (preventing locking). If you set this, ensure that you've configured
# ES_HEAP_SIZE appropriately (see below). Linux only.
bootstrap.mlockall: true
For more details see Elasticsearch's configuration information [6]_.
Specifying nodes in a cluster
*****************************
Elasticsearch's default discovery relies on multicast requests. If this isn't
a good fit, you can use unicast discovery:
.. code-block:: yaml
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ['w.x.y.z', 'w.x.y.z']
See [7]_ for more details.
JVM settings
************
For production, Elasticsearch recommends setting the ES_HEAP_SIZE environment
variable to a value around 60% of a dedicated machine's memory:
.. code-block:: yaml
export ES_HEAP_SIZE=2g
References
~~~~~~~~~~
.. [1] https://www.elastic.co/
.. [2] https://elasticsearch-py.readthedocs.io/en/master/#compatibility
.. [3] https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html
.. [4] https://www.elastic.co/downloads/elasticsearch
.. [5] https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html
.. [6] https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html
.. [7] https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html
.. [8] https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html#modules-scripting
.. [9] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index\_.html
.. [10] https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html

10
doc/source/install/index.rst
View File

@ -1,10 +0,0 @@
====================
Installation guide
====================
.. toctree::
:maxdepth: 2
dev-environment
elasticsearch
uwsgi

71
doc/source/install/uwsgi.rst
View File

@ -1,71 +0,0 @@
..
Copyright (c) 2017 Intel Corporation
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Running Searchlight API using uWSGI
===================================
The recommended way to deploy Searchlight is have a web server such as Apache
or nginx to handle http requests and proxy these requests to Searchlight WSGI
app running in uWSGI. Searchlight comes with some configuration templates on
how to deploy the api service with Apache and uWSGI.
app.wsgi
********
The ``searchlight/api/app.wsgi`` file contains a WSGI application of
Searchlight API service. This file is installed with Searchlight application
code.
apache-searchlight.template
***************************
The ``searchlight/etc/apache-searchlight.template`` file contains a copy
of Apache configuration file for Searchlight API used by devstack.
searchlight-uwsgi.ini.sample
****************************
The ``searchlight/etc/searchlight-uwsgi.ini.sample`` file is a sample
configuration file for uWSGI server. Update the file to match your
system configuration.
Steps to use these sample configuration files:
1. Enable mod_proxy_uwsgi module
* On Ubuntu install required uwsgi package
``sudo apt-get install libapache2-mod-proxy-uwsgi``; enable using
``sudo a2enmod proxy``, ``sudo a2enmod proxy_uwsgi``.
* On Fedora the required package is mod_proxy_uwsgi; enable by creating a file
``/etc/httpd/conf.modules.d/11-proxy_uwsgi.conf`` containing
``LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so``
2. On deb-based systems copy or symlink the file to
``/etc/apache2/sites-available/searchlight.conf``. For rpm-based systems the file should go into
``/etc/httpd/conf.d/searchlight.conf``.
3. Enable Searchlight site. On deb-based systems::
$ a2ensite searchlight
$ service apache2 reload
On rpm-based systems::
$ service httpd reload
4. Copy searchlight/etc/searchlight-uwsgi.ini.sample to /etc/searchlight/searchlight-uwsgi.ini.
5. Start Searchlight api using uWSGI::
$ sudo pip install uwsgi
$ uwsgi /etc/searchlight/searchlight-uwsgi.ini

9
doc/source/user/index.rst
View File

@ -1,9 +0,0 @@
============
User guide
============
.. toctree::
:maxdepth: 2
usecases
searchlightapi

827
doc/source/user/searchlightapi.rst
View File

@ -1,827 +0,0 @@
..
Copyright (c) 2015 Hewlett-Packard Development Company, L.P.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Searchlight API
===============
Searchlight's API adds authentication and Role Based Access Control in front
of Elasticsearch's query API.
Authentication
--------------
Searchlight, like other OpenStack APIs, depends on Keystone and the
OpenStack Identity API to handle authentication. You must obtain an
authentication token from Keystone and pass it to Searchlight in API requests
with the ``X-Auth-Token`` header.
See :doc:`../configuration/authentication` for more information on integrating with Keystone.
Using v1
--------
For the purposes of examples, assume a Searchlight server is running
at the URL ``http://searchlight.example.com`` on HTTP port 80. All
queries are assumed to include an ``X-Auth-Token`` header. Where request
bodies are present, it is assumed that an appropriate ``Content-Type``
header is present (usually ``application/json``).
Searches use Elasticsearch's
`query DSL <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html>`_.
Elasticsearch stores each 'document' in an 'index', which has one or more
'types'. Searchlight's indexing service stores all resource
types in their own document type, grouped by service into indices. For
instance, the ``OS::Glance::Image`` and ``OS::Glance::Metadef`` types both
reside in the ``searchlight`` index. ``type`` is unique to a resource type.
Document access is defined by each document type, for instance for glance
images:
* If the current user is the resource owner OR
* If the resource is marked public
Some resources may have additional rules. Administrators have access to all resources,
though by default searches are restricted to the current tenant unless ``all_projects``
is set in the search request body.
Querying available plugins
~~~~~~~~~~~~~~~~~~~~~~~~~~
Searchlight indexes OpenStack resources as defined by installed plugins. In
general, a plugin maps directly to an OpenStack resource type. For instance, a
plugin might index nova instances, or glance images. There may be multiple
plugins related to a given OpenStack project (an example being glance images
and metadefs).
A given deployment may not necessarily expose all available plugins.
Searchlight provides a REST endpoint to request a list of installed plugins.
A ``GET`` request to ``http://searchlight.example.com/v1/search/plugins``
might yield::
{
"plugins": [
{
"type": "OS::Glance::Image",
"alias-searching": "searchlight-search"
"alias-indexing": "searchlight-listener"
},
{
"type": "OS::Glance::Metadef",
"alias-searching": "searchlight-search"
"alias-indexing": "searchlight-listener"
}
]
}
This response shows the plugin information associated with the Glance image
and metadef resources.
* **type**: the resource group, which is used as the document type in
Elasticsearch.
* **alias-searching**: the Elasticsearch alias used for querying.
* **alias-indexing**: the Elasticsearch alias used for indexing.
If desired, all indexed Glance images can be queried directly from
Elasticsearch, rather than using Searchlight. Assuming an Elasticsearch
server running on localhost, the following request can be made::
curl http://localhost:9200/searchlight-search/OS::Glance::Image/_search
Running a search
~~~~~~~~~~~~~~~~
The simplest query is to ask for everything we have access to. We issue a
``POST`` request to ``http://searchlight.example.com/v1/search`` with the
following body::
{
"query": {
"match_all": {}
}
}
The data is returned as a JSON-encoded mapping from Elasticsearch::
{
"_shards": {
"failed": 0,
"successful": 2,
"total": 2
},
"hits": {
"hits": [
{
"_id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"_index": "searchlight",
"_type": "OS::Glance::Image"
"_score": 1.0,
"_source": {
"id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"members": [],
"name": "cirros-0.3.2-x86_64-uec",
"owner": "d95b27da6e9f4acc9a8031918e443e04",
"visibility": "public",
...
}
},
{
"_id": "OS::Software::DBMS",
"_index": "searchlight",
"_type": "metadef",
"_score": 1.0,
"_source": {
"description": "A database is an ...",
"display_name": "Database Software",
"namespace": "OS::Software::DBMS",
"objects": [
{
"description": "PostgreSQL, often simply 'Postgres' ...",
"name": "PostgreSQL",
"properties": [
{
"default": "5432",
"description": "Specifies the TCP/IP port...",
"property": "sw_database_postgresql_listen_port",
...
},
...
]
}
],
"tags": [
{
"name": "Database"
},
]
}
},
...
],
"max_score": 1.0,
"total": 8
},
"timed_out": false,
"took": 1
}
Each ``hit`` is a document in Elasticsearch, representing an OpenStack
resource. the fields in the root of each hit are:
* ``_id``
Uniquely identifies the resource within its OpenStack context (for
instance, Glance images use their GUID).
* ``_index``
The service to which the resource belongs (e.g. ``searchlight``).
* ``_type``
The document type within the service (e.g. ``image``, ``metadef``)
* ``_score``
Where applicable the relevancy of a given ``hit``. By default,
the field upon which results are sorted.
* ``_source``
The document originally indexed. The ``_source`` is a map, where each key
is a ``field`` whose value may be a scalar value, a list, a nested object
or a list of nested objects.
More example searches
~~~~~~~~~~~~~~~~~~~~~
Results are shown here only where it would help illustrate the example. The
``query`` parameter supports anything that Elasticsearch exposes via its
`query DSL`_. There are normally multiple ways to represent the same query,
often with some subtle differences, but some common examples are shown here.
.. _`query DLS`: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-queries.html
Administrators - search all resources
*************************************
By default, all users see search results restricted by access control; in
practice, this is a combination of resources belonging to the user's current
tenant/project, and any fields that are restricted to administrators.
Administrators also have the option to view all resources, by passing
``all_projects`` in the search request body. For instance, a ``POST`` to
``http://searchlight.example.com/searchlight/v1/search``::
{
"query": {
"match_all": {}
},
"all_projects": true
}
Restricting document index or type
**********************************
To restrict a query to Glance image and metadef information only (both
``index`` and ``type`` can be arrays or a single string)::
{
"query": {
"match_all": {}
},
"type": ["OS::Glance::Image", "OS::Glance::Metadef"]
}
If ``index`` or ``type`` are not provided they will default to covering as
wide a range of results as possible. Be aware that it is possible to specify
combinations of ``index`` and ``type`` that can return no results. In general
``type`` is preferred since ``type`` is unique to a resource.
Retrieving an item by id
************************
To retrieve a resource by its OpenStack ID (e.g. a glance image), we can use
Elasticsearch's `term query <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-term-query.html>`_::
{
"index": "searchlight",
"query": {
"term": {
"id": "79fa243d-e05d-4848-8a9e-27a01e83ceba"
}
}
}
Limiting and paging results
***************************
Elasticsearch (and Searchlight) support paging_ through the
``size`` and ``from`` parameters (Searchlight also accepts
``limit`` and ``offset`` respectively as synonyms). ``from`` is
zero-indexed. If ``size`` is zero, no results will be returned. This
can be useful for retrieving the total number of hits for a query without
being interested in the results themselves, or for `aggregations`_::
{
"query": {"match_all": {}},
"size": 0
}
Gives::
{
"hits": {
"hits": [],
"max_score": 0.0,
"total": 40
}
}
.. _paging: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html
Limiting the fields returned
****************************
To restrict the ``source`` to include only certain fields using Elasticsearch's
`source filtering <https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-source-filtering.html>`_::
{
"type": "OS::Glance::Image",
"_source": ["name", "size"]
}
Gives::
{
"_shards": {
"failed": 0,
"successful": 1,
"total": 1
},
"hits": {
"hits": [
{
"_id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"_index": "searchlight",
"_score": 1.0,
"_source": {
"name": "cirros-0.3.2-x86_64-uec",
"size": 3723817
},
"_type": "OS::Glance::Image"
},
...
],
"max_score": 1.0,
"total": 4
},
"timed_out": false,
"took": 1
}
Versioning
**********
Internally an always-incrementing value is stored with search results to
ensure that out of order notifications don't lead to inconsistencies with
search results. Normally this value is not exposed in search results, but
including a search parameter ``version: true`` in requests will result in
a field named ``_version`` (note the underscore) being present in each result::
{
"index": "searchlight",
"query": {"match_all": {}},
"version": true
}
{
"hits": {
"hits": [
{
"_id": "76580e9d-f83d-49d8-b428-1fb90c5d8e95",
"_index": "searchlight",
"_version": 462198730000000000,
....
},
....
]
},
...
}
Sorting
*******
Elasticsearch allows sorting by single or multiple fields. See Elasticsearch's
`sort <https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html>`_
documentation for details of the allowed syntax. Sort fields can be included as a top
level field in the request body. For instance::
{
"query": {"match_all": {}},
"sort": {"name": "desc"}
}
You will see in the search results a ``sort`` field for each result::
...
{
"_id": "7741fbcc-3fa9-4ace-adff-593304b6e629",
"_index": "glance",
"_score": null,
"_source": {
"name": "cirros-0.3.4-x86_64-uec",
"size": 25165824
},
"_type": "image",
"sort": [
"cirros-0.3.4-x86_64-uec",
25165824
]
},
...
Wildcards
*********
Elasticsearch supports regular expression searches but often wildcards within
``query_string`` elements are sufficient, using ``*`` to represent one or more
characters or ``?`` to represent a single character. Note that *starting* a
search term with a wildcard can lead to *extremely* slow queries::
{
"query": {
"query_string": {
"query": "name: ubun?u AND mysql_version: 5.*"
}
}
}
Highlighting
************
A common requirement is to highlight search terms in results::
{
"type": "OS::Glance::Metadef",
"query": {
"query_string": {
"query": "database"
}
},
"_source": ["namespace", "description"],
"highlight": {
"fields": {
"namespace": {},
"description": {}
}
}
}
Results::
{
"hits": {
"hits": [
{
"_id": "OS::Software::DBMS",
"_index": "searchlight",
"_type": "OS::Glance::Metadef",
"_score": 0.56079304,
"_source": {
"description": "A database is an organized collection of data. The data is typically organized to model aspects of reality in a way that supports processes requiring information. Database management systems are computer software applications that interact with the user, other applications, and the database itself to capture and analyze data. (http://en.wikipedia.org/wiki/Database)"
},
"highlight": {
"description": [
"A <em>database</em> is an organized collection of data. The data is typically organized to model aspects of",
" reality in a way that supports processes requiring information. <em>Database</em> management systems are",
" computer software applications that interact with the user, other applications, and the <em>database</em> itself",
" to capture and analyze data. (http://en.wikipedia.org/wiki/<em>Database</em>)"
],
"display_name": [
"<em>Database</em> Software"
]
}
}
],
"max_score": 0.56079304,
"total": 1
},
"timed_out": false,
"took": 3
}
Faceting
********
Searchlight can provide a list of field names and values present for those
fields for each registered resource type. Exactly which fields are returned
and whether values are listed is up to each plugin. Some fields or values may
only be listed for administrative users. For some string fields, 'facet_field'
may be included in the result and can be used to do an exact term
match against facet options.
To list supported facets, issue a ``GET`` to
``http://searchlight.example.com/v1/search/facets``::
{
"OS::Glance::Image": [
{
"name": "status",
"type": "string"
},
{
"name": "created_at",
"type": "date"
},
{
"name": "virtual_size",
"type": "long"
},
{
"name": "name",
"type": "string",
"facet_field": "name.raw"
},
...
],
"OS::Glance::Metadef": [
{
"name": "objects.description",
"type": "string"
},
{
"name": "objects.properties.description",
"type": "string",
"nested": true
},
...
],
"OS::Nova::Server": [
{
"name": "status",
"options": [
{
"doc_count": 1,
"key": "ACTIVE"
}
],
"type": "string"
},
{
"name": "OS-EXT-SRV-ATTR:host",
"type": "string"
},
{
"name": "name",
"type": "string",
"facet_field": "name.raw"
},
{
"name": "image.id",
"type": "string",
"nested": false
},
{
"name": "OS-EXT-AZ:availability_zone",
"options": [
{
"doc_count": 1,
"key": "nova"
}
],
"type": "string"
}
...
]
}
Facet fields containing the 'nested' (boolean) attribute indicate that the
field mapping type is either 'nested' or 'object'. This can influence how a
field should be queried. In general 'object' types are queried as any other
field; 'nested' types require some `additional complexity`_.
It's also possible to request facets for a particular type by adding a
``type`` query parameter. For instance, a ``GET`` to
``http://searchlight.example.com/v1/search/facets?type=OS::Nova::Server``::
{
"OS::Nova::Server": [
{
"name": "status",
"options": [
{
"doc_count": 1,
"key": "ACTIVE"
}
],
"type": "string"
},
...
]
}
As with searches, administrators are able to request facet terms for all
projects/tenants. By default, facet terms are limited to the currently scoped
project; adding ``all_projects=true`` as a query parameter removes the
restriction.
It is possible to limit the number of ``options`` returned for fields that
support facet terms. ``limit_terms`` restricts the number of terms (sorted
in order of descending frequency). A value of 0 indicates no limit, and is the
default.
It is possible to not return any options for facets. By default all options
are returned for fields that support facet terms. Adding
``exclude_options=true`` as a query parameter will return only the facet
field and not any of the options. Using this option will avoid an aggregation
query being performed on Elasticsearch, providing a performance boost.
.. _`additional complexity`: https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html
Aggregations
************
`Faceting`_ (above) is a more general form of `Elasticsearch aggregation`_.
Faceting is an example of 'bucketing'; 'metrics' includes functions like min,
max, percentiles.
Aggregations will be based on the ``query`` provided as well as restrictions
on resource type and any RBAC filters.
To include aggregations in a query, include ``aggs`` or ``aggregations`` in
a search request body. ``"size": 0`` prevents Elasticsearch
returning any results, just the aggregation, though it is valid to retrieve
both search results and aggregations from a single query. For example::
{
"query": {"match_all": {}},
"size": 0,
"aggregations": {
"names": {
"terms": {"field": "name"}
},
"earliest": {
"min": {"field": "created_at"}
}
}
}
Response::
{
"hits": {"total": 2, "max_score": 0.0, "hits": []},
"aggregations": {
"names": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{"key": "for_instance1", "doc_count": 2},
{"key": "instance1", "doc_count": 1}
]
},
"earliest": {
"value": 1459946898000.0,
"value_as_string": "2016-04-06T12:48:18.000Z"
}
}
}
Note that for some aggregations ``value_as_string`` may be more useful than
``value`` - for example, the ``earliest`` aggregation in the example operates
on a date field whose internal representation is a timestamp.
The `global aggregation`_ type is not allowed because unlike other aggregation
types it operates outside the search query scope.
.. _`Elasticsearch aggregation`: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html
.. _`global aggregation`: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-global-aggregation.html
Freeform queries
****************
Elasticsearch has a flexible query parser that can be used for many kinds of
search terms: the `query_string <https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html>`_
operator.
Some things to bear in mind about using ``query_string`` (see the documentation
for full options):
* A query term may be prefixed with a ``field`` name (as seen below). If it
is not, by default the entire document will be searched for the term.
* The default operator between terms is ``OR``
* By default, query terms are case insensitive
For instance, the following will look for images with a
restriction on name and a range query on size::
{
"query": {
"query_string": {
"query": "name: (Ubuntu OR Fedora) AND size: [3000000 TO 5000000]"
}
}
}
Within the query string query, you may perform a number of interesting
queries. Below are some examples.
Phrases
"""""""
::
\"i love openstack\"
By default, each word you type will be searched for
individually. You may also try to search an exact phrase by
using quotes ("my phrase") to surround a phrase. The search
service may allow a certain amount of phrase slop - meaning that
if you have some words out of order in the phrase it may still
match.
Wildcards
"""""""""
::
python3.?
10.0.0.*
172.*.4.*
By default, each word you type will match full words
only. You may also use wildcards to match parts of words. Wildcard
searches can be run on individual terms, using ? to replace a
single character, and * to replace zero or more character. 'demo'
will match the full word 'demo' only. However, 'de*'
will match anything that starts with 'de', such as 'demo_1'.
'de*1' will match anything that starts with 'de' and ends with '1'.
.. note:: Wildcard queries place a heavy burden on the search service and
may perform poorly.
Term Operators
""""""""""""""
::
+apache
-apache
web +(apache OR python)
Add a '+' or a '-' to indicate terms that must or must
not appear. For example '+python -apache web' would find
everything that has 'python' does NOT have 'apache' and should have
'web'. This may also be used with grouping. For example,
'web -(apache AND python)' would find anything with 'web', but does
not have either 'apache' or 'python'.
Boolean Operators
"""""""""""""""""
::
python AND apache
nginx OR apache
web && !apache
You can separate search terms and groups with
AND, OR and NOT (also written &&, || and !). For example,
'python OR javascript' will find anything with either term
(OR is used by default, so does not need to be specified).
However, 'python AND javascript' will find things that only have
both terms. You can do this with as many terms as you'd like (e.g.
'django AND javascript AND !unholy'). It is important to use all
caps or the alternate syntax (&&, ||), because 'and' will be
treated as another search term, but 'AND' will be treated as a
logical operator.
Grouping
""""""""
::
python AND (2.7 OR 3.4)
web && (apache !python)
Use parenthesis to group different aspects of your
query to form sub-queries. For example, 'web OR (python AND
apache)' will return anything that either has 'web' OR has both
'python' AND 'apache'.
Facets
""""""
::
name:cirros
name:cirros && protected:false
You may decide to only look in a certain field for a
search term by setting a specific facet. This is accomplished by
either selecting a facet from the drop down or by typing the facet
manually. For example, if you are looking for an image, you
may choose to only look at the name field by adding 'name:foo'.
You may group facets and use logical operators.
Range Queries
"""""""""""""
::
size:[1 TO 1000]
size:[1 TO *]
size:>=1
size:<1000
Date, numeric or string fields can use range queries.
Use square brackets [min TO max] for inclusive ranges and curly
brackets {min TO max} for exclusive ranges.
IP Addresses
""""""""""""
::
172.24.4.0/16
[10.0.0.1 TO 10.0.0.4]
IPv4 addresses may be searched based on ranges and with CIDR notation.
Boosting
""""""""
::
web javascript^2 python^0.1
You can increase or decrease the relevance of a search
term by boosting different terms, phrases, or groups. Boost one of
these by adding ^n to the term, phrase, or group where n is a
number greater than 1 to increase relevance and between 0 and 1 to
decrease relevance. For example 'web^4 python^0.1' would find
anything with both web and python, but would increase the
relevance for anything with 'web' in the result and decrease the
relevance for anything with 'python' in the result.
Reserved Characters
"""""""""""""""""""
::
python \(3.4\)
The following characters are reserved and must be
escaped with a leading \ (backslash)::
+ - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \
Advanced Features
-----------------
CORS - Accessing Searchlight from the browser
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Searchlight can be configured to permit access directly from the browser. For
details on this configuration, please refer to the
`OpenStack Cloud Admin Guide`_.
.. _`OpenStack Cloud Admin Guide`: http://docs.openstack.org/admin-guide-cloud/cross_project_cors.html

47
doc/source/user/usecases.rst
View File

@ -1,47 +0,0 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Searchlight Use Cases
=====================
Below are the use cases of Searchlight:
* **Cloud resources lookup:** whenever the user needs to query some
information of the desired OpenStack services (e.g. Nova, Neutron, etc.),
instead of using the service's APIs individually, she only needs to use
one single Searchlight web interface to perform all the queries. Currently,
Searchlight supports indexing OpenStack services such as Cinder, Designate,
Glance, Ironic, Neutron, Nova, and Swift. Searchlight also provides a
context menu for quick operations on resource search results, e.g., creating
an instance from an image.
* **Cloud resource repository:** because Searchlight keeps records of
real-time notification data from other OpenStack's components, it is
aware of the committed, available cloud resources, and other operational
events. This capacity of Searchlight can be leveraged by virtual infrastructure
management (VIM) applications, e.g., self-healing, self-configuration, RCA, etc.
This is a work-in-progress [#]_.
* **Multi-cloud discovery:** Searchlight resource repositories can be connected
forming a distributed resource discovery infrastructure across multi-cloud
deployment, i.e., cross OpenStack domain, OpenStack-Azure, AWS, or Containers
on OpenStack VMs. This new role of Searchlight can take advantages of the
existing user-friendly interface for multi-cloud resource administrators [#]_
while also providing a unified API for the automation of the resource
management operation.
References
----------
.. [#] https://storyboard.openstack.org/#!/story/2004721
.. [#] https://www.slideshare.net/vietstack/building-a-universal-search-interface-for-the-cloud

2
elasticsearch5.txt
View File

@ -1,2 +0,0 @@
# Elasticsearch 5.x
elasticsearch>=5.0.0,<6.0.0 # Apache-2.0

2
elasticsearch6.txt
View File

@ -1,2 +0,0 @@
# Elasticsearch 6.x
elasticsearch>=6.0.0,<7.0.0 # Apache-2.0

1
etc/apache-searchlight.template
View File

@ -1 +0,0 @@
ProxyPass "/search" "unix:/var/run/uwsgi/searchlight-api-wsgi.socket" retry=0

38
etc/api-paste.ini
View File

@ -1,38 +0,0 @@
# Use this pipeline for no auth - DEFAULT
[pipeline:searchlight]
pipeline = cors http_proxy_to_wsgi versionnegotiation unauthenticated-context rootapp
[pipeline:searchlight-keystone]
pipeline = cors versionnegotiation authtoken context rootapp
[composite:rootapp]
paste.composite_factory = searchlight.api:root_app_factory
/: apiversions
/v1: apiv1app
[app:apiversions]
paste.app_factory = searchlight.api.versions:create_resource
[app:apiv1app]
paste.app_factory = searchlight.api.v1.router:API.factory
[filter:versionnegotiation]
paste.filter_factory = searchlight.api.middleware.version_negotiation:VersionNegotiationFilter.factory
[filter:unauthenticated-context]
paste.filter_factory = searchlight.api.middleware.context:UnauthenticatedContextMiddleware.factory
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
delay_auth_decision = true
[filter:context]
paste.filter_factory = searchlight.api.middleware.context:ContextMiddleware.factory
[filter:cors]
paste.filter_factory = oslo_middleware.cors:filter_factory
oslo_config_project = searchlight
[filter:http_proxy_to_wsgi]
paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory
oslo_config_project = searchlight

9
etc/oslo-config-generator/searchlight.conf
View File

@ -1,9 +0,0 @@
[DEFAULT]
output_file = etc/searchlight.conf.sample
namespace = searchlight
namespace = oslo.policy
namespace = keystonemiddleware.auth_token
namespace = oslo.log
namespace = oslo.service.service
namespace = oslo.middleware.cors
namespace = oslo.messaging

3
etc/oslo-policy-generator/searchlight.conf
View File

@ -1,3 +0,0 @@
[DEFAULT]
output_file = etc/policy.yaml.sample
namespace = searchlight

34
etc/property-protections-policies.conf.sample
View File

@ -1,34 +0,0 @@
# property-protections-policies.conf.sample
#
# This file is an example config file for when
# property_protection_rule_format=policies is enabled.
#
# Specify regular expression for which properties will be protected in []
# For each section, specify CRUD permissions. You may refer to policies defined
# in policy.json or policy.yaml.
# The property rules will be applied in the order specified. Once
# a match is found the remaining property rules will not be applied.
#
# WARNING:
# * If the reg ex specified below does not compile, then
# the glance-api service fails to start. (Guide for reg ex python compiler
# used:
# http://docs.python.org/2/library/re.html#regular-expression-syntax)
# * If an operation(create, read, update, delete) is not specified or misspelt
# then the glance-api service fails to start.
# So, remember, with GREAT POWER comes GREAT RESPONSIBILITY!
#
# NOTE: Only one policy can be specified per action. If multiple policies are
# specified, then the glance-api service fails to start.
[^x_.*]
create = default
read = default
update = default
delete = default
[.*]
create = context_is_admin
read = context_is_admin
update = context_is_admin
delete = context_is_admin

32
etc/property-protections-roles.conf.sample
View File

@ -1,32 +0,0 @@
# property-protections-roles.conf.sample
#
# This file is an example config file for when
# property_protection_rule_format=roles is enabled.
#
# Specify regular expression for which properties will be protected in []
# For each section, specify CRUD permissions.
# The property rules will be applied in the order specified. Once
# a match is found the remaining property rules will not be applied.
#
# WARNING:
# * If the reg ex specified below does not compile, then
# glance-api service will not start. (Guide for reg ex python compiler used:
# http://docs.python.org/2/library/re.html#regular-expression-syntax)
# * If an operation(create, read, update, delete) is not specified or misspelt
# then the glance-api service will not start.
# So, remember, with GREAT POWER comes GREAT RESPONSIBILITY!
#
# NOTE: Multiple roles can be specified for a given operation. These roles must
# be comma separated.
[^x_.*]
create = admin,member,_member_
read = admin,member,_member_
update = admin,member,_member_
delete = admin,member,_member_
[.*]
create = admin
read = admin
update = admin
delete = admin

25
etc/searchlight-uwsgi.ini.sample
View File

@ -1,25 +0,0 @@
[uwsgi]
wsgi-file = /usr/local/bin/searchlight-api-wsgi
chmod-socket = 666
socket = /var/run/uwsgi/searchlight-api-wsgi.socket
# Override the default size for headers from the 4k default.
buffer-size = 65535
# This is running standalone
master = true
enable-threads = true
# Tune this to your environment.
processes = 4
# uwsgi recommends this to prevent thundering herd on accept.
thunder-lock = true
plugins = python
# This ensures that file descriptors aren't shared between Searchlight processes.
lazy-apps = true

BIN
images/Figure1.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 38 KiB

BIN
images/Figure2.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 180 KiB

BIN
images/Figure3.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 155 KiB

BIN
images/Figure4.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 234 KiB

BIN
images/SeaaS.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 87 KiB

44
lower-constraints.txt
View File

@ -1,44 +0,0 @@
coverage==4.5.2
elasticsearch==2.0.0
eventlet==0.18.2
fixtures==3.0.0
greenlet==0.4.10
keystonemiddleware==4.17.0
os-api-ref==1.4.0
oslo.concurrency==3.26.0
oslo.config==5.2.0
oslo.context==2.19.2
oslo.i18n==3.15.3
oslo.log==3.36.0
oslo.messaging==5.29.0
oslo.middleware==3.31.0
oslo.policy==1.30.0
oslo.reports==1.18.0
oslo.serialization==2.18.0
oslo.service==1.33.0
oslo.upgradecheck==0.1.0
oslo.utils==3.33.0
oslotest==3.2.0
osprofiler==1.4.0
Paste==2.0.2
PasteDeploy==1.5.0
pbr==2.0.0
psutil==3.2.2
pyOpenSSL==17.1.0
python-cinderclient==3.3.0
python-designateclient==2.7.0
python-glanceclient==2.8.0
python-ironicclient==2.3.0
python-keystoneclient==3.8.0
python-neutronclient==6.11.0
python-novaclient==9.1.0
python-swiftclient==3.2.0
requests==2.21.0
Routes==2.3.1
simplejson==3.5.1
sphinx==1.6.2
stestr==2.0.0
stevedore==1.20.0
testtools==2.2.0
WSME==0.8.0
WebOb==1.8.4

27
pylintrc
View File

@ -1,27 +0,0 @@
[Messages Control]
# W0511: TODOs in code comments are fine.
# W0142: *args and **kwargs are fine.
# W0622: Redefining id is fine.
disable-msg=W0511,W0142,W0622
[Basic]
# Variable names can be 1 to 31 characters long, with lowercase and underscores
variable-rgx=[a-z_][a-z0-9_]{0,30}$
# Argument names can be 2 to 31 characters long, with lowercase and underscores
argument-rgx=[a-z_][a-z0-9_]{1,30}$
# Method names should be at least 3 characters long
# and be lowercased with underscores
method-rgx=[a-z_][a-z0-9_]{2,50}$
# Module names matching nova-* are ok (files in bin/)
module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+)|(nova-[a-z0-9_-]+))$
# Don't require docstrings on tests.
no-docstring-rgx=((__.*__)|([tT]est.*)|setUp|tearDown)$
[Design]
max-public-methods=100
min-public-methods=0
max-args=6

0
releasenotes/notes/.placeholder
View File

62
releasenotes/notes/add-0_2_1-notes-e1f8910ad27d88bb.yaml
View File

@ -1,62 +0,0 @@
---
prelude: |
The initial mitaka release (0.2) did not support
ElasticSearch 2.x. ElasticSearch made several changes
to internal index data mappings that were not
backwards compatible with ElasticSearch 1.x. This
release has all the changes required for Searchlight
to support both ElasticSearch 1.x and 2.x.
Searchlight now supports oslo pools instead of requiring
a different topic for each listener. This simplifies deployment
configuration. Previously, services would have to be specifically
configured to send notifications to the Searchlight topic. With
this change, Searchlight can share topics with other services
such as Ceilometer.
upgrade:
- To support ElasticSearch 1.x and 2.x, several internal data
mappings have been updated. If you previously deployed Searchlight,
you will need to reindex your resource data using
``searchlight-manage index sync``. You may re-index everything
or limit it to ``OS::Nova::Server``, ``OS::Glance::Image``,
``OS::Glance::Metadef``, ``OS::Cinder:Volume``,
and ``OS::Cinder::Snapshot``.
- |
To start using a pool with a shared topic instead of separate topics:
* Update the respective service configuration files for searchlight
enabled plugins to only publish to a single topic
(e.g. set ``notification_topics = notifications``). If you have
Ceilometer enabled, this must be the same topic which Ceilometer uses.
You also typically must restart each service for the change to
take effect. For example, in ``nova.conf``, ``glance-api.conf``,
``cinder.conf``, ``neutron.conf``, and ``designate.conf``:
::
notification_topics = notifications
* Update ``searchlight.conf``. In the ``[resource_plugin]`` section, set
``notifications_topic`` to match the shared topic that you set in the
service configurations. In the ``[listener]`` sections, set
``notifications_pool`` to your desired pool name. The default name
starting in Newton is ``searchlight``. The ``notifications_pool`` does
not have to match anything from other services configuration files.
For example:
::
[resource_plugin]
notifications_topic = notifications
[listener]
notifications_pool = searchlight
fixes:
- Bug 1570213 Apply query to highlight query
- Bug 1532010 Ensure consistency in mapping field types
- Bug 1570674 Fix unicode error when booting instance from volume
- Bug 1570199 Fix inconsistent mapping in image plugin
- Bug 1568709 Remove port.create.end handler from nova
- Bug 1565015 Add volume.retype event
- Bug 1583215 Correct Cinder exchange value
- Bug 1583215 Enable notification messaging pools

4
releasenotes/notes/add-devstack-elasticsearch-5x-48934b931a31c823.yaml
View File

@ -1,4 +0,0 @@
---
upgrade:
- |
Add support for ElasticSearch 5.x

6
releasenotes/notes/add-nova-microversion-support-8b40da4458678f20.yaml
View File

@ -1,6 +0,0 @@
---
features:
- Added microversion support for Nova plugins, the default
API version for Nova plugins is 2.1. User can change the
API microversion for Nova using compute_api_version config
option added in section service_credentials:nova.

3
releasenotes/notes/add-server-group-notification-handler-d10aed78bede2c12.yaml
View File

@ -1,3 +0,0 @@
---
features:
- Adds notification handler for nova server group.

Some files were not shown because too many files have changed in this diff Show More