In Change I523a4a85867f82d234ba1f3e6fad8b8cd2291182, the pep8 test was
accidentally dropped.
Fix up code so that pep8 passes.
In addition to that following change has been added here to unbreak CI:
Add WebTest as an indirect test dependency
Pecan has made webtest an optional dependency for testing only [1].
Since it is still used for testing we need to add it to our
test-requirements.txt.
[1]: https://github.com/pecan/pecan/pull/140
Change-Id: I2f85adb4ef29a43389897c201e6152fd4c7be9d6
We depend on the Kubernetes Python client for several things such as
health checks & metrics polling. Those are both run inside periodic
jobs which spawn in greenthreads.
The Kubernetes API uses it's own thread pools which seem to use
native pools and cause several different deadlocks when it comes to
logging. Since we don't make extensive use of the Kubernetes API
and we want something that doesn't use any threadpools, we can
simply use a simple wrapper using Requests.
This patch takes care of dropping the dependency and refactoring
all the code to use this simple mechansim instead, which should
reduce the overall dependency list as well as avoid any deadlock
issues which are present in the upstream client.
Change-Id: If0b7c96cb77bba0c79a678c9885622f1fe0f7ebc
Adding the master_lb_enabled option when creating a cluster,
which will benefit both the cloud provider side and the end
user side. For cloud prodiver, they don't have to maintain
separate cluster templates with or w/o master_lb_enabled enabled.
For end user, they can easily use one single template to create
different clusters with different configs.
Task: 39680
Story: 2007634
Change-Id: I0b586f05168ece84fd340ef7493a56688191053d
The original design of k8s cluster health status is allowing
the health status being updated by Magnum control plane. However,
it doesn't work when the cluster is private. This patch supports
updating the k8s cluster health status via the Magnum cluster
update API by a 3rd party service so that a controller (e.g.
magnum-auto-healer) running inside the k8s cluster can call
the Magnum update API to update the cluster health status.
Task: 38583
Story: 2007242
Change-Id: Ie7189d328c4038403576b0324e7b0e8a9b305a5e
At present, if floating_ip_enabled is true and master_lb_enabled is also
true but master_lb_fip_enabled is false, a cluster still resolves to
being accessible therefore resolves to an UNHEALTHY status when it
should be UNKNOWN. This patch fixes this edge case and also adds a unit
test to capture this issue.
Story: 2007242
Task: 39140
Change-Id: I74f7455e3caa920032080747a315470878ba5500
If nodes in a cluster were deleted, period health checks for the cluster
will timeout. This will result in logs of logs like the following:
ERROR oslo.service.loopingcall ValueError: Field
`health_status_reason[api]' cannot be None
The timeout is successfully caught by exception handling in this part of
the code. However, it is thrown as a type MaxRetryError exception, which
does not have body or message attrs. E.g.
MaxRetryError("HTTPSConnectionPool(host='115.146.81.72', port=6443): Max
retries exceeded with url: /healthz (Caused by
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object
at 0x7f6b9b915a20>: Failed to establish a new connection: [Errno 110]
ETIMEDOUT',))",)
This means health_status_reason will be a dict like `{'api': None}`.
Saving this using oslo_versionedobjects will throw an ValueError,
because althought the dict itself is corced as a `Dict(default=<class
'oslo_versionedobjects.fields.UnspecifiedDefault'>,nullable=True)`, the
None value will be coerced as a `String(default=<class
'oslo_versionedobjects.fields.UnspecifiedDefault'>,nullable=False)` and
that is not nullable.
Task: 38316
Change-Id: I8fd8d363284b06cf0bfba45d5845ba8687a2c783
When the k8s cluster is created in a private network without
FIP and without a LB FIP, the cluster is not reachable by Magnum
control plane. For this case, the cluster health status is unknown
for Magnum and Magnum should not connect the cluster to avoid
unnecessary logs.
Task: 36187
Story: 2006383
Change-Id: Ibb589ec51f91f05a334c907560ef5fe57cbfbffb
Calling Kubernetes native API to update the cluster health status
so that it can used for cluster auto healing.
Task: 24593
Story: 2002742
Change-Id: Ia76eeeb2f1734dff38d9660c804d7d2d0f65b9fb
For a really long time, we generated and maintained our very own python
client generated from kubernetes swagger json files. Now in Kubernetes
Community there is a concerted effort to organize an official python
client (also generated from swagger) for everyone to use. So let us
please switch over from our python-k8sclient and use the community
driven python client. I have ported all of our end-to-end tests and got
them working in kubernetes client-python project upstream so we should
be protected from regressions.
Implements: blueprint replace-k8sclient-with-upstream-kubernetes-client
Depends-On: I72359f2b811392008eb5267812bf343797b1553a
Change-Id: Ib81a69cfdc25198e259e3b3d4081c92c01fd1bc5
Following changes for cluster-drivers, move coe specific monitors
at driver level. This change is needed to add the driver field
properly.
Change-Id: Id4658b8f7400bf3c86c8ff81756fb33d1211a0b3
Implements: blueprint bay-drivers
Jake Yip