This commit adds the configuration options related to resource limits
in the Heat project. The `max_software_configs_per_tenant`,
`max_software_deployments_per_tenant`, and `max_snapshots_per_stack`
options have been added to control the maximum limits for software
configs, software deployments, stack snapshots.
Story: 2011006
Task: 49401
Change-Id: If33a1c6f3eb9e93f586931bc5c05104439c92bf9
Snapshot.get_all does not return all snapshots of the project but
returns all snapshots associated with a single stack, so its name
should contain _by_stack for consistency.
Change-Id: Ic6b93b7cfc84793077672b3f1052f03519e4c5a1
We were previously calling 'connect()' on the 'connectable' object in
'run_migrations_online', regardless of whether it was an 'Engine' or
'Connection' object. This worked because, as noted in an inline comment,
"when connectable is already a Connection object, calling 'connect()'
gives us a *branched connection*." This is no longer the case. From the
SQLAlchemy docs [1]:
The Connection object does not support "branching", which was a
pattern by which a sub "connection" would be used that refers to this
connection as a parent.
Update our code to reflect this change, using the newly updated example
from the SQLAlchemy cookbook doc [2] as inspiration.
[1] https://docs.sqlalchemy.org/en/14/core/future.html#sqlalchemy.future.Connection
[2] https://alembic.sqlalchemy.org/en/latest/cookbook.html#connection-sharing
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Id41ae6d3fa3f343efdb727aae9bdef8ad2a098fd
This also gets us away from autocommit, i.e. the last outstanding item
in our preparation for SQLAlchemy 2.x.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I95c5e5d6b33e73a1df2bf036df44aa2e06cc9640
Make sure we avoid nested sessions going forward.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ia171bb823f5302fb3a6b2f6f88f7295f97254759
There are only a few callers. Do it directly.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ib9cb53c0a9a6b309e625bc8ed4facc18c4fbe0ba
This is the final preparation step for our removal of autocommit. This
makes our life easier since we can insist on only opening transactions
inside the public method.
On a related point, we remove one commit call from a private method and
keep another.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I75cf29a18b2179949ee4713f85ca52256d170c84
Well, we actually remove it but keep the entry point to avoid breaking
scripts. This has been around since 2016 and anyone that hasn't upgraded
now ain't upgrading.
This is one less set of APIs to worry about when removing the use of
autocommit.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I43051b3ca6e292f1164d750b04d7a440012b2ef5
Another step to make removal of autocommit easier.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I664a927198df1be89dcd9a5304afaf452795dcc6
This will make our life easier as we attempt to remove autocommit by
allowing us to maintain a pattern of only opening transactions inside
public methods.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I74069645babc4200ffd4087503277282b4c12792
This turned out to be easier done that I expected.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ie50b4bd212c758ec41c82be145290024910099a0
Software deployments have FK constraints on software configs.
This change ensures the DBReferenceError caused by the constraints is
properly caught. With this change now heat returns 400 Bad Response,
instead of 500 Internal Server Error when a user tries to create
a software deployment with a non-existing software config.
Also, the stack_user_project_id field is defined as 64-chars-long
string in DB model, so we should ensure that the input value is shorter
than 65 chars. Otherwise it also results in DB error.
Story: 2010001
Task: 45098
Change-Id: I03274dc0cffa226140eb720458cce81e8b5ce187
This will make a future change easier to grok.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I0eefa245e827f951fc2333f64670f19100cbcc8a
Resolve the following RemovedIn20Warning warning:
The current statement is being autocommitted using implicit
autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
method of Engine or Connection in order to use an explicit transaction
for DML and DDL statements.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ibe7503002be9f7cdf13259379ebbf6e9c329a028
Resolve the following RemovedIn20Warning warnings:
Using strings to indicate column or relationship paths in loader
options is deprecated and will be removed in SQLAlchemy 2.0. Please
use the class-bound attribute directly.
Using strings to indicate relationship names in Query.join() is
deprecated and will be removed in SQLAlchemy 2.0. Please use the
class-bound attribute directly.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I81a506281a0cece77cad41c5290e41fa3d7ef246
Resolve the following RemovedIn20Warning warnings:
The MetaData.bind argument is deprecated and will be removed in
SQLAlchemy 2.0.
The ``bind`` argument for schema methods that invoke SQL against an
engine or connection will be required in SQLAlchemy 2.0.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Idcd26c539f6c46dac245e6a445e64a0bf1596379
Resolve the following sqlalchemy.exc.RemovedIn20Warning warning:
The legacy calling style of select() is deprecated and will be removed
in SQLAlchemy 2.0. Please use the new calling style described at
select().
For more information, refer to http://sqlalche.me/e/b8d9.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Id64999a7aa6081a5b1f2ac12674204bd02f2c012
Resolve the following RemovedIn20Warning warnings:
The autoload parameter is deprecated and will be removed in version
2.0. Please use the autoload_with parameter, passing an engine or
connection.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I2a626bd60ecc824d9328d08e582d78d5cb568f5a
Actually, not really. We just remove the warning. Apparently it's not
used here.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ie6c45772247850d40b0cf4b135e5a3028b1758af
Resolve the following RemovedIn20Warning warning:
The Engine.execute() method is considered legacy as of the 1.x
series of SQLAlchemy and will be removed in 2.0.
This one is pretty simple.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Iec56e6bff476ae9b68a4821be9cc93e3af84a325
We don't have another ORM to content with here. Simplify
'heat.db.sqlalchemy' to 'heat.db'.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Id1db6c0ff126859f436c6c9b1187c250f38ebb62
Unlike Cinder, Nova etc., we haven't had any sqlalchemy-migrate-based
migrations in Heat for some time. That means we don't need a transition
period to handle users that may not have the latest
sqlalchemy-migrate-based migration and we can remove these migrations
immediately. Doing so also means we can also remove sqlalchemy-migrate
from our dependencies. Hurrah!
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I825c58a42e9d430c9d660dae9bb70d1cb6f91079
This is significantly easier than Cinder, Nova etc. as Heat hasn't had
any database migrations in multiple cycles, meaning we don't need to
worry about having to apply any older sqlalchemy-migrate migrations
before switching to alembic. Instead, we simply need to determine we're
upgrading a deployment that was previously using sqlalchemy-migrate,
upgrading a deployment that has already migrated to alembic, or
deploying a new deployment, adjusting accordingly.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I808af9cb21ba21808209b1daddac7426f4cad310
Alembic's auto-generate functionality doesn't preserve index names or
column order. This causes comparisons between databases to fail. Update
the initial alembic migration to match the schema's generated by
sqlalchemy-migrate.
Change-Id: Ie8d98731c61434e85896062fe05137cbadf0eaa5
This was generated using a different method to the one I've used
previously [1][2]. Instead of manually converting sqlalchemy-migrate
calls to alembic equivalents, we generate it automatically based on our
models using alembic's auto-generate functionality:
alembic --config heat/db/sqlalchemy/alembic.ini \
revision -m 'Initial revision 2'
alembic --config heat/db/sqlalchemy/alembic.ini \
upgrade head
alembic --config heat/db/sqlalchemy/alembic.ini \
revision -m 'Initial revision 2' --autogenerate
The two files were then combined, formatting and imports adjusted, and
schema of this and the sqlalchemy-migrate migrations compared. The
latter step was done by comparing the schemas of a database created by
the sqlalchemy-migrate tool to the one created by alembic:
python heat/db/sqlalchemy/migrate_repo/manage.py version_control \
sqlite:///heat-old.db heat/db/sqlalchemy/migrate_repo/ 072
python heat/db/sqlalchemy/migrate_repo/manage.py upgrade \
sqlite:///heat-old.db heat/db/sqlalchemy/migrate_repo
With the two databases created, we can compare them using the
methodologies described in [3].
[1] https://review.opendev.org/c/openstack/nova/+/799527
[2] https://review.opendev.org/c/openstack/cinder/+/786931
[3] https://that.guru/blog/comparing-nova-db-migrations/
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ie32fbb42576760f2c4a52d8305823b99df2b41ba
No migrations yet: this is simply the output of 'alembic init' with some
minor tweaks.
A note about ordering of requirements is removed from requirements.txt:
it is no longer true with the new dependency resolver introduced in pip
20.3.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I7790222ab5eaf6c47dd386fe472275a3037a9898
These are unused since change Icdab0db3f3371cd4eb8a8fb11cbc2328c0a830e7
way back in 2016.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I2e337ff0fbbded0c0cd42060aa35b4e0787d0d5f
If the auth_encryption_key changes We can possibly
ignore these errors when deleting stacks.
Task: 42056
Change-Id: I326e415db194a5b9c67acd038d7d2d993293ecb3
This change fixes the one remaining usage without the flag which was
missed by the previous attempt[1].
[1] 14d2d859ef
Change-Id: Ic0b34b11d669851e47cae670f5299f934f396257
oslo.db 12.1.0 has changed the default value for the 'autocommit'
parameter of 'LegacyEngineFacade' from 'True' to 'False'. This is a
necessary step to ensure compatibility with SQLAlchemy 2.0. However, we
are currently relying on the autocommit behavior and need changes to
explicitly manage sessions. Until that happens, we need to override the
default.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I57b8cce28c2ad7899b831067c94b9d913f3f1334
Since sqlalchemy 1.4.14[1], the following warning is raised unless
the cache_ok flag is explicitly set in classes inheriting
TypeDecorator.
SAWarning: TypeDecorator Json() will not produce a cache key because
the ``cache_ok`` attribute is not set to True. This can have
significant performance implications including some performance
degradations in comparison to prior SQLAlchemy versions.
Set this attribute to True if this type object's state is safe to use
in a cache key, or False to disable this warning. (Background on this
error at: https://sqlalche.me/e/14/cprf)
Because the flag was set to True by default in older releases, this
adds explicit cache_ok = True, to restore the previous behavior in
Wallaby and older. (wallaby u-c has SQLAlchemy===1.3.23 while xena u-c
has SQLAlchemy===1.4.23)
[1] 6967b45020
Change-Id: I6b84a626994543468f49373a554de44025576a9a
OpenDev infra only keep around the latest two Fedora releases in their
mirrors. Probe for the image from the local test mirror, but if not
found, fallback to upstream. This will be much less reliable, but can
avoid gate breakage until new images can be used.
Also, use endpoint_type when creating keystoneclient
Keystone admin endpoint has been removed from devstack with[1].
This would use the public endpoint by default.
Change-Id: I96ab14871ee8c5d5b83cc0cd4abc840ef0218ca8
The change to SQLAlchemy upper constraints has
broken some of the tests in Heat. Syntax and usage
needs to be updated accordingly to ensure tests work
with the 1.4 version of SQLAlchemy.
Reference:
https://storyboard.openstack.org/#!/story/2008974
Story: 2008974
Task: 42616
Change-Id: Id4755c8c58401ebfd42a8c57f94d111de9f74ac1
With I04e7ad90944c2d03ce0e59ba16af9d60d6e01222 we allowed
update of previously-existing resource if replacement
creation has failed during last update. However, we force
update the status of the existing resource to 'COMPLETE'.
Therefore, in the next update if properties/type has not
changed for the resource it won't try to update/replace
the previously existing resource resulting in false positive
of stack updated successfully.
Task: 42194
Change-Id: Icc90a921ec67e49aec2c4acfad72235c57c78421
Allow following db configs when calling wrap_db_retry:
* database.db_max_retries
* database.db_retry_interval
* database.db_inc_retry_interval
* database.db_max_retry_interval
So database cofig can now control db retries.
Please reference [1] for what each config options can do.
[1] https://opendev.org/openstack/oslo.db/src/branch/master/oslo_db/options.py
Change-Id: I034625733c2d22f0f5635f58e9df3d5785e58cf5
With SQlAlchemy 1.3.23 constraint.copy() fails with error[0].
This seems like a regression after[1].
This patch fixes the issue by providing the ``target_table``
argument to c.copy().
[0] http://paste.openstack.org/show/802486
[1] 7dd3381edb
Change-Id: Ia25b8443bbe576f73fb7debe54f307deadf34e04
Signed-off-by: Kevin Carter <kecarter@redhat.com>
Remove E402 from flake8 list, issues are fixed or addressed with noqa.
Remove E305 as comment, the warning is enabled.
Enable E241 and fix all problems
Change-Id: Ifd2bbce98be4206908da769df686a370328c7ee1
A simple matter of having log lines fit
with all the others. It will now match the
ambiguous standard.
Also change comments inline to alias with log fixes.
Change-Id: I4a2ed6134f70c2965811f75ccb6ab8221fa0e960
Story: #2007319
Task: 38830
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 4 of a serie of 28 patches
Change-Id: I871c2dad10abc35790e730c7c4c5272f499b7623
We seem to be using sqlalchemy.Text for 'status_reason' column for
some tables in model and heat type LONGTEXT in migration scripts.
Change-Id: I8b792936722f6fed2dd4e579aa41f6bf6aa27f7c
Like DbDeadlock retry in DBConnectionError as oslo.db and SQLAlchemy
already automatically reconnect, if the database was disconnected
and a new transaction is beginning.
Change-Id: I479408fc47753e5971b347b379079b30e035cc08