Commit Graph

252 Commits

Author SHA1 Message Date
Scott Hussey 0f39a55942 Refactor build to include Go
Adding the baclient code to Drydock requires a refactor
of the build automation to support multiple languages
and multiple artifacts included in a single Docker image

NOTE: the go source here is a placeholder 'hello world' sample

Change-Id: I1b4883f018b33b3d4fcd7cbcb6cba660fcdc93de
2018-09-04 13:13:21 -05:00
Scott Hussey 75009880ff [Fix] Improve idempotence of creating MAAS domains
- Need to refresh the local copy of existing domains
  during a deployment so that the same domain is not attempted
  to be created multiple times.

Change-Id: Idb792561690d2014aa2fceb54bf85c8e27cd91db
2018-08-28 08:37:22 -05:00
Scott Hussey adbce25373 YAPF formatting
Cleanup some formatting.

Change-Id: Icbe0ab0b380817169959e6b527e83320fbd702fa
2018-08-22 09:30:07 -05:00
Scott Hussey daf94bdeac Add DNS domains during network config
- Node domains must be added to MAAS before assigning
  them to any domain. These domains can be added as the
  network they are assigned to is configured.

Change-Id: I4a10cc7779b74dd79b8ad4602ae0f9050081d6d4
2018-08-22 09:30:07 -05:00
Scott Hussey e044575e05 Node naming validations
- Update the hostname validator account for the FQDN

Change-Id: If59eb5d5d351e2251cf881492b46e109111c91ec
2018-08-22 09:29:20 -05:00
Scott Hussey 89ce941202 [443514] Set domain for nodes during deployment
- Resolve the FQDN of node based on the DNS domain of the primary
  network
- When identifying the node in MAAS, set the hostname and the domain
  for the node

Change-Id: I1fc5d89e07784ae8168461bcf4ffe400956616f9
2018-08-21 14:09:36 -05:00
Scott Hussey e2b3e8ee8e Database tunability
Add additional tunables to how SQLalchemy creates connection pools

Also start including static doc assets to they can be published

Change-Id: I268dc265a6b6cf1a200b235a5f99e65e89a95637
2018-08-21 14:09:36 -05:00
Scott Hussey 5027b385ec (fix) API client used wrong parameter type
The API client for the nodefilter API endpoint
documented the wrong required parameter type.

Change-Id: I61373d92ed3e019eb3c332f578ac5e5bec6e1222
2018-08-21 14:09:34 -05:00
Scott Hussey 4cbd641e90 (fix) fix missing kwarg
A positional arg should have been a keyword arg

Change-Id: I13c6d3aa6a5d901d0fd14eb3edda99b0b9049964
2018-08-21 14:08:53 -05:00
Scott Hussey 541b2d413c Make device alias resolution optional
When compiling an effective site design, make
the step for resolving device aliases optional
as only a few tasks need this information but it
takes a large amount of the time.

Change-Id: I9f1090de0ec26887217be80567367547306b3905
2018-08-21 14:08:16 -05:00
Zuul a67efe071e Merge "Added code to support Destroy Node" 2018-08-14 16:35:45 +00:00
Ahmad Mahmoudi 471bc6147e Added code to support Destroy Node
Aded API capability to erase the storage and destroy one
or more nodes from any node state.
After this operation, the destroyed nodes are deleted from maas
database, and will need to be re-enlisted and be added to maas
in New state. if they need to be re-deployed.

Change-Id: I986278902965d93514f8198602f971669e1e2db3
2018-08-13 20:37:34 -05:00
Scott Hussey 7b6af1bdc9 (fix) Fix builddata API issue
The builddata API was passing a kwarg
as a parameter causing a failure accessing
database.

Also update the log extraction to improve
description in the builddata

Change-Id: I01b4cbb14eea5c513a164818160fb10a693c4696
2018-07-31 15:08:02 -05:00
Scott Hussey 246775da42 Add build data access to Drydock client
- Add CLI actions to output the build data for a node or
  a task
- Add API methods to access the Drydock API to retrieve node
  or task build data

Change-Id: I0ee01bd4b165b93c2bc0e3050554514ba40f152a
2018-07-30 10:08:22 -05:00
Scott Hussey f57301fae9 Add build data for MAAS logs
- Collect logs from MAAS when failures happen
  during deployment
- Save the logs to build data so it is available via
  API
- Add postgres integration test

Change-Id: Ied2d8539fe02a75f1f175a421b897b4f8ce07c8d
2018-07-30 10:08:22 -05:00
Scott Hussey eb996c27f4 (format): yapf reformat
- Run yapf to correct formatting
- Update one variable typo

Change-Id: Ic756ce4f9811cdf2726907c54299f4740fca9ce5
2018-07-27 20:16:25 -05:00
Scott Hussey b53a3ff0ea Add support of MAAS node-results API
- Support extraction of log data from MAAS

Change-Id: I55fcbad5e06378f148c9d5de476e1eef8d898ed2
2018-07-27 20:16:25 -05:00
Scott Hussey cff7420cff Support links for task status
- Some status changes in a task may have additional information that
  is referenced by a URI link. Support describing these links and
  returning them via API.
- Refactor alembic stuff to better handle table schema updates
- Add unit tests

Change-Id: Iae63a9716f2522578be0244925fc274a4338eac4
2018-07-27 20:16:24 -05:00
Scott Hussey 4423ee985e Add additional repo options
- Some repos are built without components that MAAS
  assumes are there. This allows the configuration to
  describe that so MAAS can be updated to disregard
  those components

Change-Id: I4eafe9fb8278b944bcb993f10379966a01e94bf6
2018-07-14 14:23:34 -05:00
Scott Hussey 06d6747b50 Make client HTTP connections resilient
- Make Keystone session use a timeout to prevent hangs
- Support retries
- Make the above configurable

Change-Id: I7123bd2fdcd329eae5b8b40f09168a1d599fa0f7
2018-06-26 14:41:24 -05:00
Scott Hussey 7e7739ce2c (fix) Guard against DB connection leaks
- Use connection contexts to ensure database connections
  are released back to the pool.
- Make database connection pool size configurable, defaults to 15

Change-Id: Id8011fbf45a1b3c87835ff5f47ebfa9334488319
2018-06-26 14:38:40 -05:00
Zuul 6af445ac4a Merge "Improvements for handling existing nodes" 2018-06-22 15:56:04 +00:00
Scott Hussey da60e2d127 Improvements for handling existing nodes
- Additional improvements in how Drydock handles nodes
  that are already known to MAAS.

Change-Id: I893348e0fd8a1907ebe35e33edf64e622dd79feb
2018-06-21 17:31:03 -05:00
Zuul b93cf21fa2 Merge "(fix) Fix invalid kwarg" 2018-06-18 23:54:36 +00:00
Scott Hussey a5cfdd25ca (fix) Fix invalid kwarg
Incorrect kwarg used.

Change-Id: Id65e37d1051d833d9a7c3a95efc73e8bfbbeab80
2018-06-18 16:30:33 -05:00
Zuul f738bd4a1c Merge "Fix name of error CollectSubtaskTimeout" 2018-06-18 16:19:45 +00:00
Scott Hussey 396f1da5e0 (fix) Make noops considered a success
When a task for prepare_nodes encounters a node that
is already managed by the node provisioner, it skips
additional interactions to prevent disturbing existing
nodes. Add these nodes to the success list for the
task so upstream workflows can validate that a described
node was found and explicitly considered.

Change-Id: Ia387ab4d6fc0d9ad3838b9df54f46d6921d12a42
2018-06-18 09:20:57 -05:00
Bryan Strassner 2fc3e22dc2 Fix name of error CollectSubtaskTimeout
Was misnamed in the error.py as CollectSubaskTimeout (missing the 't' in
task)

Change-Id: Id57130955f7f68b535fda1616c8aac6b118ba1f3
2018-06-17 15:23:31 -05:00
Zuul b4a31a79de Merge "[411429] maasdriver support of packages" 2018-06-14 14:56:47 +00:00
Mark Burnett 9dd1722368 Order hugepagesz and hugepages kernel parameters
These kernel parameters must be specified in order.

* Moves kernel parameter string from the maas driver to the model
* Forces hugepagesz to occur immediately before hugepages

Change-Id: Id0cc18b73bde17f9bfac980b0eefe37699e86eff
2018-06-13 12:41:08 -05:00
Scott Hussey fda50be35e [411429] maasdriver support of packages
- Support pkg_list bootactions in the MAAS driver by using
  cloud-init user_data on deployment
- Add site definition caching to ease load on Deckhand

Change-Id: I2c8c7dfdd23992fae42fa32edab308f801d05867
2018-06-12 09:58:02 -05:00
Scott Hussey e35712a573 [411430] Validate bootaction pkg_list
- Add a validator for bootactions to warn if a node doesn't
  have at least one
- Add a validator for bootactions to error if a package
  version specifier is invalid
- Unit tests for the validation

Change-Id: I61d8aa3831791af0484498e6fe9f7c1c83dbf540
2018-06-12 09:58:02 -05:00
Scott Hussey 1b0797440b [411428] Bootaction pkg_list support
- Support a list of debian packages as a bootaction asset
- Add unit testing for parsing the additional bootaction information
- Add __eq__ and __hash__ for DocumentReference to allow checking
  equality and list presence

Change-Id: I0ca42baf7aae6dc2e52efd5b311d0632e069dd79
2018-06-09 19:53:43 +00:00
Mark Burnett 714528a549 Update Formatting with YAPF
Change-Id: I351f55b5ffbbae8c15d529777bfac058054c43c9
2018-06-08 13:24:26 +00:00
Scott Hussey 6dad448ca6 [411390] Configure repo in MAAS
- Add a new action for ConfigureNodeProvisioner to configure a node
  provisioner with site-wide configuration
- Add a maasdriver action to configure repositories

Change-Id: I8e216a269b300159b7cc26c3a4542e8b61496dc7
2018-06-08 02:45:14 +00:00
Scott Hussey cc77125953 [411387] Schema update to support repo
- Support one or more repo specifications for a site
- Add object model for repository
- Add testing for repository parsing

Update freeze job with make target

- Update the requirements freeze job
  to have a mark target that rebuilds the tox
  virtualenv each run
- Update Dockerfile to create a valid /etc/protocols file

Change-Id: I9d09b7dd7226827995e23756ff968b36eaa4d16c
2018-06-07 23:08:20 +00:00
Mark Burnett 786271af5c Fix: use only node's HW profile for nic logical names
Change-Id: Ib0d4c2e341ffb8c7a367d1a48cc7e99d60f70ef6
2018-06-05 12:03:38 -05:00
Scott Hussey 20873ad4f9 Update docs for developer overview
- New diagrams and documents for developer overview
- Update conf.py for docs to work w/ readthedocs.io
- Add policy and config gen to `make docs`
- Update zuul-linter to support checked in images
- Last fix to document publishing

Change-Id: I4faa1b87032ae5b0e786aa0fd998f809124b7987
2018-06-01 14:50:05 -05:00
Scott Hussey 53f6ccbfb7 Clean up gates
- Remove OSH-based gate as it is not relevant at this point
- Add entries for unit testing, linting and security scans
- Some trivial changes to trigger all gates

Change-Id: Icb2731573b45f3824cde8cb44ef64e9d04c017bc
2018-05-25 16:36:45 -05:00
Scott Hussey d107e65a98 Libvirt support in maasdriver
- Add validations that OOB configs for nodes are valid for the
  oob type defined
- Add documentation for using Drydock/MAAS to deploy libvirt VMs
- Add logic to update the MAAS node power parameters to allow power
  control of libvirt VMs

Change-Id: Ia7d5fbd1659636d46cf1790fe3fc66ca6c6fee89
2018-05-25 11:13:53 -05:00
Scott Hussey dbad75775b Libvirt OOB driver
- Create a driver to support OOB actions via libvirt API
- Update Makefile with external dependency target
- Update Makefile and tooling to support new chart pipeline
   - Add 'drydock' make target for chart building
   - Add step to install helm binary

Change-Id: I8a3984d8fd70f99a82a954b7a869eab8e30145b4
2018-05-25 11:13:53 -05:00
Scott Hussey cd9770a979 Zuul: Initial Airship-Drydock checks
This PS adds the skeleton for a set of zuul checks and gates for
Airship, using the framework from OpenStack-Helm.

[Fix] Zuul lint fix

- Fix whitespace linting errors
- Update docuemtation errors
- Remove dead tests

Change-Id: Id1ed3aa2321d518ed15fcbb3b698b49709bed287
Signed-off-by: Pete Birley <pete@port.direct>
2018-05-24 16:59:12 -05:00
Aaron Sheffield 506e06623a Added ValidateDesign to API Client
- Added ValidateDesign to API client.
- Added ValidteDesign to CI client.
- Added a couple unit tests.

Change-Id: I1def14b3703eba12b10a063bf09a3120dcf54b8d
2018-05-14 10:54:26 -05:00
Aaron Sheffield 8274d56c71 Handle no baremetal_nodes in process_node_filter
- Added a try when getting baremetal_nodes from site_design.
   If baremetal_nodes isn't there it will return an empty set.

Change-Id: I413892274d938ffd23102c73a569b83a59025206
2018-05-10 09:45:16 -05:00
Aaron Sheffield 8af92eaf29 Change POST nodefilter Input
- Changed the expected input from site_design to design_ref.
- Converts the design_ref json/dict into a SiteDesign object.

Change-Id: Ie1f6738fec64749cde9d2843d9373175006794c0
2018-05-08 10:32:26 -05:00
Aaron Sheffield d052664f74 Add Drydock API Client Options
- Added builddata, subtaskerrors, and layers as options to get_task.
- Added get_nodes_for_filter to call POST /nodefilter.
- Updated endpoint for POST /nodes to POST /nodefilter.
- Made node_filter optional in POST /nodefilter.

Change-Id: I456a6e9991d03af3d375c448f5cbf07a21e91f1d
2018-05-08 14:11:15 +00:00
Scott Hussey 3b41868802 [373577] Fix hostnames with underscores
Hostnames with underscores caused a deployment failure.

Update to use a double underscore as a delimiter and provide
a design validator to check that hostnames do not contain it.

Closes #78

Change-Id: Ib148aed5cffe7fd8bc08441eaef8a45af6601bdd
2018-05-03 15:42:33 +00:00
Aaron Sheffield b138b3c179 New Drydock APIs for nodes and tasks
- Added a new query param 'layers=x' to /tasks/{task_id}
  where x is a number.
  This param allows the API to drill down to all the subtasks.
  If x is -1 it will reeturn all subtasks, otherwise it will return
  x layers of subtasks.
- Added a new query param 'subtaskerrors=true' to /tasks/{task_id}.
  If true then any errors from subtasks will be included in the
  subtask_errors element in the response.
- Added a POST to /nodes that requires node_filter and site_design
  in the body.  Then uses those parameters to return a list of nodes.

Change-Id: I9b4c06dd1c9de2b0500092aa4b4bfacac08eac54
2018-05-02 14:42:41 +00:00
Scott Hussey 2d2bb08d86 [391121] Fix reported bootaction defect
- Created unit test scenario to match bug report and unable
  to reproduce error.

Change-Id: I06bdd6d2e89257c71d00e03c599a062f10f9e198
2018-05-01 16:09:49 +00:00
Scott Hussey 815aa333ec [Fix] Support deploying nodes w/o tags
- The apply_platform task was not considered successful if a node had
  no kernel parameters or tags, so the deploy step would not start

Change-Id: I5e59e35ec666b3ff2741aae823366d0a39bf4490
2018-04-23 19:19:19 +00:00