Commit Graph

103 Commits

Author SHA1 Message Date
Scott Hussey 0f39a55942 Refactor build to include Go
Adding the baclient code to Drydock requires a refactor
of the build automation to support multiple languages
and multiple artifacts included in a single Docker image

NOTE: the go source here is a placeholder 'hello world' sample

Change-Id: I1b4883f018b33b3d4fcd7cbcb6cba660fcdc93de
2018-09-04 13:13:21 -05:00
Zuul f8c821e721 Merge "Remove the duplicated word" 2018-08-29 12:55:26 +00:00
melissaml 364b34c6a4 Remove the duplicated word
Change-Id: I96a7bc066a073d1176efd286f6b70bf3e7c21e03
2018-08-22 23:55:26 +08:00
Scott Hussey adbce25373 YAPF formatting
Cleanup some formatting.

Change-Id: Icbe0ab0b380817169959e6b527e83320fbd702fa
2018-08-22 09:30:07 -05:00
Scott Hussey daf94bdeac Add DNS domains during network config
- Node domains must be added to MAAS before assigning
  them to any domain. These domains can be added as the
  network they are assigned to is configured.

Change-Id: I4a10cc7779b74dd79b8ad4602ae0f9050081d6d4
2018-08-22 09:30:07 -05:00
Scott Hussey e044575e05 Node naming validations
- Update the hostname validator account for the FQDN

Change-Id: If59eb5d5d351e2251cf881492b46e109111c91ec
2018-08-22 09:29:20 -05:00
Scott Hussey 89ce941202 [443514] Set domain for nodes during deployment
- Resolve the FQDN of node based on the DNS domain of the primary
  network
- When identifying the node in MAAS, set the hostname and the domain
  for the node

Change-Id: I1fc5d89e07784ae8168461bcf4ffe400956616f9
2018-08-21 14:09:36 -05:00
Scott Hussey 541b2d413c Make device alias resolution optional
When compiling an effective site design, make
the step for resolving device aliases optional
as only a few tasks need this information but it
takes a large amount of the time.

Change-Id: I9f1090de0ec26887217be80567367547306b3905
2018-08-21 14:08:16 -05:00
Scott Hussey 246775da42 Add build data access to Drydock client
- Add CLI actions to output the build data for a node or
  a task
- Add API methods to access the Drydock API to retrieve node
  or task build data

Change-Id: I0ee01bd4b165b93c2bc0e3050554514ba40f152a
2018-07-30 10:08:22 -05:00
Scott Hussey f57301fae9 Add build data for MAAS logs
- Collect logs from MAAS when failures happen
  during deployment
- Save the logs to build data so it is available via
  API
- Add postgres integration test

Change-Id: Ied2d8539fe02a75f1f175a421b897b4f8ce07c8d
2018-07-30 10:08:22 -05:00
Scott Hussey eb996c27f4 (format): yapf reformat
- Run yapf to correct formatting
- Update one variable typo

Change-Id: Ic756ce4f9811cdf2726907c54299f4740fca9ce5
2018-07-27 20:16:25 -05:00
Scott Hussey b53a3ff0ea Add support of MAAS node-results API
- Support extraction of log data from MAAS

Change-Id: I55fcbad5e06378f148c9d5de476e1eef8d898ed2
2018-07-27 20:16:25 -05:00
Scott Hussey cff7420cff Support links for task status
- Some status changes in a task may have additional information that
  is referenced by a URI link. Support describing these links and
  returning them via API.
- Refactor alembic stuff to better handle table schema updates
- Add unit tests

Change-Id: Iae63a9716f2522578be0244925fc274a4338eac4
2018-07-27 20:16:24 -05:00
Roman Gorshunov fa07ea4f4d Cleanup yaml files
Cleanup yaml files.

Change-Id: Iff474e33a12632ad74cfaaccc9152bb5066b90b1
2018-06-22 18:49:59 +02:00
Mark Burnett 9dd1722368 Order hugepagesz and hugepages kernel parameters
These kernel parameters must be specified in order.

* Moves kernel parameter string from the maas driver to the model
* Forces hugepagesz to occur immediately before hugepages

Change-Id: Id0cc18b73bde17f9bfac980b0eefe37699e86eff
2018-06-13 12:41:08 -05:00
Scott Hussey e35712a573 [411430] Validate bootaction pkg_list
- Add a validator for bootactions to warn if a node doesn't
  have at least one
- Add a validator for bootactions to error if a package
  version specifier is invalid
- Unit tests for the validation

Change-Id: I61d8aa3831791af0484498e6fe9f7c1c83dbf540
2018-06-12 09:58:02 -05:00
Scott Hussey 1b0797440b [411428] Bootaction pkg_list support
- Support a list of debian packages as a bootaction asset
- Add unit testing for parsing the additional bootaction information
- Add __eq__ and __hash__ for DocumentReference to allow checking
  equality and list presence

Change-Id: I0ca42baf7aae6dc2e52efd5b311d0632e069dd79
2018-06-09 19:53:43 +00:00
Mark Burnett 714528a549 Update Formatting with YAPF
Change-Id: I351f55b5ffbbae8c15d529777bfac058054c43c9
2018-06-08 13:24:26 +00:00
Scott Hussey 6dad448ca6 [411390] Configure repo in MAAS
- Add a new action for ConfigureNodeProvisioner to configure a node
  provisioner with site-wide configuration
- Add a maasdriver action to configure repositories

Change-Id: I8e216a269b300159b7cc26c3a4542e8b61496dc7
2018-06-08 02:45:14 +00:00
Scott Hussey cc77125953 [411387] Schema update to support repo
- Support one or more repo specifications for a site
- Add object model for repository
- Add testing for repository parsing

Update freeze job with make target

- Update the requirements freeze job
  to have a mark target that rebuilds the tox
  virtualenv each run
- Update Dockerfile to create a valid /etc/protocols file

Change-Id: I9d09b7dd7226827995e23756ff968b36eaa4d16c
2018-06-07 23:08:20 +00:00
Mark Burnett 786271af5c Fix: use only node's HW profile for nic logical names
Change-Id: Ib0d4c2e341ffb8c7a367d1a48cc7e99d60f70ef6
2018-06-05 12:03:38 -05:00
Scott Hussey dbad75775b Libvirt OOB driver
- Create a driver to support OOB actions via libvirt API
- Update Makefile with external dependency target
- Update Makefile and tooling to support new chart pipeline
   - Add 'drydock' make target for chart building
   - Add step to install helm binary

Change-Id: I8a3984d8fd70f99a82a954b7a869eab8e30145b4
2018-05-25 11:13:53 -05:00
Scott Hussey cd9770a979 Zuul: Initial Airship-Drydock checks
This PS adds the skeleton for a set of zuul checks and gates for
Airship, using the framework from OpenStack-Helm.

[Fix] Zuul lint fix

- Fix whitespace linting errors
- Update docuemtation errors
- Remove dead tests

Change-Id: Id1ed3aa2321d518ed15fcbb3b698b49709bed287
Signed-off-by: Pete Birley <pete@port.direct>
2018-05-24 16:59:12 -05:00
Aaron Sheffield 506e06623a Added ValidateDesign to API Client
- Added ValidateDesign to API client.
- Added ValidteDesign to CI client.
- Added a couple unit tests.

Change-Id: I1def14b3703eba12b10a063bf09a3120dcf54b8d
2018-05-14 10:54:26 -05:00
Aaron Sheffield 8274d56c71 Handle no baremetal_nodes in process_node_filter
- Added a try when getting baremetal_nodes from site_design.
   If baremetal_nodes isn't there it will return an empty set.

Change-Id: I413892274d938ffd23102c73a569b83a59025206
2018-05-10 09:45:16 -05:00
Aaron Sheffield 8af92eaf29 Change POST nodefilter Input
- Changed the expected input from site_design to design_ref.
- Converts the design_ref json/dict into a SiteDesign object.

Change-Id: Ie1f6738fec64749cde9d2843d9373175006794c0
2018-05-08 10:32:26 -05:00
Aaron Sheffield d052664f74 Add Drydock API Client Options
- Added builddata, subtaskerrors, and layers as options to get_task.
- Added get_nodes_for_filter to call POST /nodefilter.
- Updated endpoint for POST /nodes to POST /nodefilter.
- Made node_filter optional in POST /nodefilter.

Change-Id: I456a6e9991d03af3d375c448f5cbf07a21e91f1d
2018-05-08 14:11:15 +00:00
Scott Hussey 3b41868802 [373577] Fix hostnames with underscores
Hostnames with underscores caused a deployment failure.

Update to use a double underscore as a delimiter and provide
a design validator to check that hostnames do not contain it.

Closes #78

Change-Id: Ib148aed5cffe7fd8bc08441eaef8a45af6601bdd
2018-05-03 15:42:33 +00:00
Aaron Sheffield b138b3c179 New Drydock APIs for nodes and tasks
- Added a new query param 'layers=x' to /tasks/{task_id}
  where x is a number.
  This param allows the API to drill down to all the subtasks.
  If x is -1 it will reeturn all subtasks, otherwise it will return
  x layers of subtasks.
- Added a new query param 'subtaskerrors=true' to /tasks/{task_id}.
  If true then any errors from subtasks will be included in the
  subtask_errors element in the response.
- Added a POST to /nodes that requires node_filter and site_design
  in the body.  Then uses those parameters to return a list of nodes.

Change-Id: I9b4c06dd1c9de2b0500092aa4b4bfacac08eac54
2018-05-02 14:42:41 +00:00
Scott Hussey 2d2bb08d86 [391121] Fix reported bootaction defect
- Created unit test scenario to match bug report and unable
  to reproduce error.

Change-Id: I06bdd6d2e89257c71d00e03c599a062f10f9e198
2018-05-01 16:09:49 +00:00
Scott Hussey 94d351b364 [Fix] Migrate to pymongo BSON library
* bson is incompatible w/ pip 10, migrate to pymongo bson
* Update tox to run tests w/ locked requirements file as this
  is what production images are built with

Change-Id: Iacf3301ddc224f3525102b0204f349b733608138
2018-04-19 09:30:18 -05:00
Scott Hussey 29f416aefe [Fix] whole-disk sizing fails
- When using the '>' operator to create a partition
  using the full available storage left on a physical
  device fails with 'not enough storage available'. Nothing
  in MaaS documents what would cause this, but it is
  possible that the available_size field on the block
  device does not account for storage consumed for writing
  the partition table.

Change-Id: Iac777e9d18df1df92e25779f9297a3e2dea12a12
2018-04-11 11:07:10 -04:00
Scott Hussey 984eab8168 [Fix] Support validation with no nodes
- Drydock should successfully validate a design with
  no baremetal nodes
- Unit test for this scenario

Change-Id: Icebb5387d716e56534a1163603c8454a13ac6760
2018-03-24 02:12:32 -04:00
Scott Hussey a71c4be911 [Fix] Adhere to UCP spec for validation status
- The spec calls it out capitalized strings
- Remove pip cache from Dockerfile build

Change-Id: I4c8907120ba0213936060d9f0b463fc5e0bcf7db
2018-03-23 10:31:27 -05:00
Scott Hussey b628a1bfce [390773] Support SRIOV fields in definition profiles
- Add hugepages and cpu_sets stanzas to HardwareProfile as the
  size and count of hugepages and the exact CPUs to pin for SRIOV
  are dependent on hardware.
- Add sriov stanza to a node interface to specify vf_count and
  trustedmode. These will be passthrough values as Drydock doesn't
  configure SRIOV.
- Add sriov information to the bootaction context so it can be written
  to disk on a deployed node if needed
- Allow an interface configuration to be skipped when an interface has
  no defined network_link for things like SR-IOV interfaces.
- Add kernel parameter reference support to access hardware profile
  information
- Add unit tests
- Update topology documentation for usage of HardwareProfile
  and kernel parameter references

Change-Id: Iefd326f5c6fad19dbd21300ee249019a3dfd4848
2018-03-14 14:12:51 -05:00
Scott Hussey c27c8e6c9b Use new validator model for validation
- Update all validator rules to publish messages using
  new format
- Update unit tests to follow new validation framework
- Add validation for no tagged VLANs on non-trunked links
- Add validation for node interfaces attaching to networks
  not allowed on the interface link
- Unit test change to work around an issue where coverage testing
  and unit testing show different results

- Update tox to skip E126 to match YAPF formatting

Change-Id: Ifef21112896b88c2bd2b361630e041806ebb6663
2018-03-14 14:12:51 -05:00
Aaron Sheffield 6c83117f2c Refactoring validations into individual classes
- Updated validations and tests for new classes.

Change-Id: I66cfd8cc5857a63d6764804b71c00384fd3d1bcf
2018-03-13 10:11:49 -05:00
Scott Hussey 47a27981ec Enhance validation message flow
- Update validation message format to latest UCP standard
- Support message levels of ERROR, WARN, INFO
- Update ingestion and schema validation to use new validation
  message format

Change-Id: Ic463badaea8025c5dece88b3999075ead3419bec
2018-03-13 10:11:49 -05:00
Aaron Sheffield cff99e4d1c Enhanced Drydock Health Check
- Added a database connection check to the health check.
- Added a MaaS connection check to the health check.

Change-Id: I6c771a9ed6a278eb75dbc0f2e503088ff9046149
2018-03-12 15:36:15 -05:00
Bryan Strassner 0dacd50c3a [390136] Drydock client timeout options
Adds options to drydock client to allow for override
of both connect and read timeouts.
Adds logging to the drydock client for get and post
requests being made
Bring yapf changes current.

Change-Id: I5ff007315059e2087612e2209966815291433893
2018-03-07 16:59:15 -06:00
Aaron Sheffield 2b26d5204c Remove BuildDataError from Logicalnames
- Removed raising a BuildDataError exception when applying logicalnames.

Change-Id: I15dcab186954284f60fad440659a8558cd8be278
2018-03-05 10:37:21 -06:00
Scott Hussey 2749f98125 Fix: CLI default filter wrong
- If no filter items are specified on the command
  line, the CLI created a default node filter that
  evaluated to no nodes instead of all nodes
- Add unit test to test the null node filter condition

Closes #85

Change-Id: I948059028207a86ff52479b244dbbc449c8741bc
2018-02-28 10:42:37 -05:00
Scott Hussey 83f57669fb Fix: boot actions signaling
- When signaling is false, boot action should be
  added to database with new result 'Unreported' indicating
  the boot action result is indeterminate.

Closes #87

Change-Id: I505cb61d6ec4aeb752b6c2c08109b59e1f9347f8
2018-02-26 13:00:22 -05:00
Aaron Sheffield 8939447669 Gets a dictionary of device aliases to logicalnames for the site_design.
Change-Id: I9c5ec6d4dd3ac0114fc6d866a7dd8033b7807b15
2018-02-26 10:09:12 -05:00
Stacey Fletcher e7f630651f Update Makefile
This ps updates the build commands to allow the same
format to be used in att-comdev projects

Change-Id: Ibc5be98636b2d0f29654a66261df27367ded49a6
2018-02-25 20:08:04 -05:00
Scott Hussey 992359fa21 Address issues found in lab deploys
- Don't include route to source subnet for routedomains

Closes #76

- Wrap adding network address ranges in MaaS driver in a try
  block and log failures without fatally exiting function

Closes #82

- Wrap action.start() in a try block and mark initiating
  task failed on exception (with detail status msg)

Closes #81

Change-Id: I29aea0e91b8fb5619b3349e99526afbc737d540e
2018-02-23 15:19:32 -05:00
Jason Beard 14f4cfddc5 Fix Token Timeout Defect
Defect for Shipyard-45

- Change session authentication to a user
  provided function for generating request
  headers.
- On a 401 response from Drydock, the API client
  will refresh the authentication headers once and
  retry the call.

Change-Id: I0f0db8885f1e5f15b5ab6dda4549b9e81fb4a184
2018-02-13 14:13:10 -05:00
Scott Hussey 4a1367a39a Support image and kernel selection
- Image selection based of Ubuntu series name (e.g. 'xenial')
- Kernel selection based on MAAS kernel names of 'ga-16.04'
  or 'hwe-16.04'
- Validation that defined images and kernels are valid in the
  current node driver
- Unit test for the validation

Change-Id: I6256a2e00a4594af38a2d862c738e03efb7ddb29
2018-02-08 15:22:16 -06:00
Scott Hussey 9f1c5eba82 Build data collection
Add steps in the ConfigureHardware step of the maasdriver
to collect `lshw` and `lldp` results from the MaaS API
and persist them as node build data

- Add collection logic to the ConfigureHardware action
- Add integration test for validating collection

Change-Id: Id85175e23b94d9b6d9df7066edf0dcfe79d9c9ef
2018-01-08 16:21:16 -06:00
Scott Hussey 74ce4aaef0 Add build data support to the API
- Add list of build data to detailed task response when builddata=true
  specifed in query string
- Add new endpoint for /nodes/nodename/builddata to retrieve
  build data for a particular node
- Update docs for new API capabilities
- Testing all around

Change-Id: If0fcd2962d4389789af45ad1fbe61d226ac6a403
2018-01-08 16:21:16 -06:00