This commit adds the configuration options related to resource limits
in the Heat project. The `max_software_configs_per_tenant`,
`max_software_deployments_per_tenant`, and `max_snapshots_per_stack`
options have been added to control the maximum limits for software
configs, software deployments, stack snapshots.
Story: 2011006
Task: 49401
Change-Id: If33a1c6f3eb9e93f586931bc5c05104439c92bf9
has_calls is not the correct method for asserting a list of calls
has occured during testing. Use assert_has_calls instead.
This also revealed an issue in the expected signature for one
of the mock calls in the test_software_config module.
Change-Id: Ib01ee79f49525f6d0ba7915108ef92c338a8f393
This also gets us away from autocommit, i.e. the last outstanding item
in our preparation for SQLAlchemy 2.x.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I95c5e5d6b33e73a1df2bf036df44aa2e06cc9640
Software deployments have FK constraints on software configs.
This change ensures the DBReferenceError caused by the constraints is
properly caught. With this change now heat returns 400 Bad Response,
instead of 500 Internal Server Error when a user tries to create
a software deployment with a non-existing software config.
Also, the stack_user_project_id field is defined as 64-chars-long
string in DB model, so we should ensure that the input value is shorter
than 65 chars. Otherwise it also results in DB error.
Story: 2010001
Task: 45098
Change-Id: I03274dc0cffa226140eb720458cce81e8b5ce187
We don't have another ORM to content with here. Simplify
'heat.db.sqlalchemy' to 'heat.db'.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Id1db6c0ff126859f436c6c9b1187c250f38ebb62
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I8f764e9ba46a4e2055be61eb0fe97d155ab1c70e
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 18 of a serie of 28 patches
Change-Id: I433b5e06ad288fd284b3055cbe2fa4bbcc3e43cd
All of the existing unit tests for the software config service assume the
'config' part is a string; add a test where the config is a dict, as in
OS::Heat::StructuredConfig, to ensure that this doesn't break.
Change-Id: I2abcaae952890e206857363d34f976e04937f324
While we are careful to ensure there are no race conditions in updating
the software deployment metadata in the Heat database, there was no such
protection in place for the actual update of the metadata provided to
the server via Swift. In legacy Heat, this race could only occur when
software deployments were created from the REST API or from separate
stacks deploying software to the same server, because resources in the
same stack were handled by a single thread making synchronous calls to
create the software deployments. However, with convergence, resources in
the same stack not only can be processed concurrently but will tend to
be synchronised if they have the same dependencies (since they'll be
triggered by consecutive asynchronous RPC messages).
This handles the race by retrying if a concurrent update is detected by
the time the data has been provided to Swift.
Note that this is only guaranteed to result in the latest data being
available on the assumption that a later write to Swift cannot overwrite
an earlier one that has been accepted. This is probably *not* guaranteed
by Swift in the presence of clock skew. Nonetheless, this should
drastically reduce the failure rate.
Similar caveats would probably apply to Zaqar, although Zaqar nominally
provides FIFO queues (in practice, it likely depends on the backend
used). However, with Zaqar clients receive all updates, not just the
latest one, and os-collect-config stores the deployments contained in
every message. Thus, Zaqar is not affected even assuming out-of-order
arrival.
Change-Id: Ic9a447f27e8c51f91f47f93b0fd1e9710341ec38
Closes-Bug: #1731032
Eliminate the 3 slowest unit tests, which collectively block one of the
test processes for a minute and a half, by stubbing out the sleep() calls
in tenacity.
ServerInternalPortTest.test_prepare_ports_for_replace_detach_failed
75s -> 0.3s
SoftwareConfigServiceTest.test_push_metadata_software_deployments_retry
11s -> 0.2s
ServerInternalPortTest.test_restore_ports_after_rollback_attach_failed
4.6s -> 0.3s
Change-Id: I4ee0c7eb249c4e8aa6ab6da1e4f945e3a629451b
Before making a software deployments update in the Swift backend, check
the data content to operate as a transaction.
Change-Id: I58d3af3d1de481b3d314313c9f39255504792634
Closes-Bug: #1731032
We recently (I87edc0d8f1fba2fb61276c682a60c1b2241b5705) enforced default
type for software config input, whereas it was always a string before.
It causes issues with upgrades, as we used to store "" as a default for
all configs regarless of their type, and it failed when we started
validating.
This identifies this case and skip validation.
Change-Id: I8a4f4e834b0862ecef4b97f208f4b03be3572e86
Closes-Bug: #1697627
This was most likely meant as a max 2s delay here, not a max 2ms
delay.
Also includes a related change: when retries for metadata updates are
attempted, make sure we do not have a stale value of the atomic_key
(otherwise we'll just inevitably hit the ConcurrentTransaction issue).
Co-Authored-By: Crag Wolfe <cwolfe@redhat.com>
Partial-Bug: #1651768
Change-Id: Ie56e0e4ff93633db1f4752859d2b2a9506922911
The default type for input was set to string, but it actually don't
match what's used for the type property. Set it to ANY to let the user
provide the proper value.
Change-Id: I87edc0d8f1fba2fb61276c682a60c1b2241b5705
Closes-Bug: #1687643
This change adds project information of software configs
if using admin context.
Change-Id: Ia26919aa1177a9366c65710becb2097b79e02445
Closes-Bug: #1646312
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.
Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
It's possible that we could end up with multiple resources with the same
physical resource ID, but that would be undetectable since we return only
one from the database layer. This change allows us to detect the problem an
return an error where the result is rendered ambiguous.
Change-Id: I2c5ddbe6731c33a09ec7c4a7b91dcfe414da4385
When the only way to define a Software Config was via a Heat resource, the
input and output configs were validated by the properties of the resource.
However, subsequently a REST API to create Software Configs directly was
added. That means that configs created in this way do not have the contents
of the inputs and outputs sections validated. This change adds validation
to ensure that the configs always follow the correct schema.
Change-Id: I8c66bb82484b75723524959be753a4cd20c0f84d
Curently you get events that look like this with the default
event formatter:
2016-05-11 17:49:12 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded
Including the ID in the reason string will help (particularly in the
case of failure), as you can then easily show the deployment and
thus the stderr associated with the failure.
Change-Id: I59a3e9bf527040c4d807adc4f822bfa2c1591585
Related-Bug: #1564627
This changes the way SoftwareDeployment updates its internal status.
Instead of querying the deployment status in a loop using RPC client, it
makes a single call, which returns once the status has been updated.
Closes-Bug: #1549219
Change-Id: I484b7c8cb4a4e71817be6bea764f23b68a39b2d4
This patch removes specific DeploymentConcurrentTransaction exception
and adds one common ConcurrentTransaction exception. New exception may
be re-sed for another cases with concurrent transactions with different
error message.
Change-Id: Iaa82732702ad6035a7784eb6689c2536535142f9
When a deployment is deleted the metadata needs to be pushed so that
stale deployment data is not present in the metadata. There is a risk
that stale data will mask other deployments.
Change-Id: I3df4d9a27be5e59609ac4374556d3c0c707b068b
Closes-Bug: #1523705
Zaqar queues were accessed and created with the user provided token, but
we need them to be in the stack project for instances to access them.
This changes to provide a stack user token instead.
Change-Id: If30ad4ad63b845b8dabe0b3ac0e4209057c3753c
Closes-Bug: #1532235
This patch splits the stack watch related unit tests into a separate
module. Some mox calls are replaced with mock calls where approriate.
It also moves the existing modules related to engine service tests
into a separate subdirectory. This is to make room for other unit
tests that are coming in.
Change-Id: I7ea61c22e017785c480d78d6c07132164ed0b889
lujiefsi