Commit Graph

646 Commits

Author SHA1 Message Date
lujiefsi 7d86fc6d84 Allow more options to limit number of resources
This commit adds the configuration options related to resource limits
in the Heat project. The `max_software_configs_per_tenant`,
`max_software_deployments_per_tenant`, and `max_snapshots_per_stack`
options have been added to control the maximum limits for software
configs, software deployments, stack snapshots.

Story: 2011006
Task: 49401
Change-Id: If33a1c6f3eb9e93f586931bc5c05104439c92bf9
2024-01-26 22:22:11 +00:00
Takashi Kajinami 13ec108b0d Fix inconsistent naming in db api
Snapshot.get_all does not return all snapshots of the project but
returns all snapshots associated with a single stack, so its name
should contain _by_stack for consistency.

Change-Id: Ic6b93b7cfc84793077672b3f1052f03519e4c5a1
2024-01-27 07:21:46 +09:00
Stephen Finucane 9b754a019e db: Don't rely on branched connections
We were previously calling 'connect()' on the 'connectable' object in
'run_migrations_online', regardless of whether it was an 'Engine' or
'Connection' object. This worked because, as noted in an inline comment,
"when connectable is already a Connection object, calling 'connect()'
gives us a *branched connection*." This is no longer the case. From the
SQLAlchemy docs [1]:

  The Connection object does not support "branching", which was a
  pattern by which a sub "connection" would be used that refers to this
  connection as a parent.

Update our code to reflect this change, using the newly updated example
from the SQLAlchemy cookbook doc [2] as inspiration.

[1] https://docs.sqlalchemy.org/en/14/core/future.html#sqlalchemy.future.Connection
[2] https://alembic.sqlalchemy.org/en/latest/cookbook.html#connection-sharing

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Id41ae6d3fa3f343efdb727aae9bdef8ad2a098fd
2023-10-12 15:46:28 +00:00
Stephen Finucane bb6b5ed5c7 db: Migrate to enginefacade pattern
This also gets us away from autocommit, i.e. the last outstanding item
in our preparation for SQLAlchemy 2.x.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I95c5e5d6b33e73a1df2bf036df44aa2e06cc9640
2023-10-12 15:46:21 +00:00
Stephen Finucane 879bc37bb0 db: Remove use of 'transaction' helper
Make sure we avoid nested sessions going forward.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ia171bb823f5302fb3a6b2f6f88f7295f97254759
2023-10-12 15:46:13 +00:00
Stephen Finucane b2b1c52af7 db: Replace use of update_and_save
There are only a few callers. Do it directly.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ib9cb53c0a9a6b309e625bc8ed4facc18c4fbe0ba
2023-10-12 15:46:03 +00:00
Stephen Finucane 67070d4ed5 db: Only call private methods from public methods
This is the final preparation step for our removal of autocommit. This
makes our life easier since we can insist on only opening transactions
inside the public method.

On a related point, we remove one commit call from a private method and
keep another.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I75cf29a18b2179949ee4713f85ca52256d170c84
2023-10-12 15:45:52 +00:00
Stephen Finucane 9ba72bbc18 cmd: Deprecate heat-manage migrate_properties_data command
Well, we actually remove it but keep the entry point to avoid breaking
scripts. This has been around since 2016 and anyone that hasn't upgraded
now ain't upgrading.

This is one less set of APIs to worry about when removing the use of
autocommit.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I43051b3ca6e292f1164d750b04d7a440012b2ef5
2023-10-13 00:45:07 +09:00
Stephen Finucane d1124ac268 db: Group related DB APIs
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I10b57246e44d5b6d75a0514508761d8de7adea6e
2023-10-12 15:43:34 +00:00
Stephen Finucane 6b514e29d9 db: Make utility methods private
Another step to make removal of autocommit easier.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I664a927198df1be89dcd9a5304afaf452795dcc6
2023-10-12 15:43:23 +00:00
Stephen Finucane ea321003eb db: Remove unnecessary helper functions
This will make our life easier as we attempt to remove autocommit by
allowing us to maintain a pattern of only opening transactions inside
public methods.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I74069645babc4200ffd4087503277282b4c12792
2023-10-12 15:43:06 +00:00
Stephen Finucane a9634d3f45 db: Remove use of subtransactions
This turned out to be easier done that I expected.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ie50b4bd212c758ec41c82be145290024910099a0
2023-10-13 00:41:37 +09:00
Ekaterina Chernova 7860e7acbc Handle FK error when creating/updating software deployments
Software deployments have FK constraints on software configs.
This change ensures the DBReferenceError caused by the constraints is
properly caught. With this change now heat returns 400 Bad Response,
instead of 500 Internal Server Error when a user tries to create
a software deployment with a non-existing software config.

Also, the stack_user_project_id field is defined as 64-chars-long
string in DB model, so we should ensure that the input value is shorter
than 65 chars. Otherwise it also results in DB error.

Story: 2010001
Task: 45098
Change-Id: I03274dc0cffa226140eb720458cce81e8b5ce187
2023-09-13 15:26:11 +09:00
Stephen Finucane 04c8faba49 db: Removing aliasing of context.session
This will make a future change easier to grok.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I0eefa245e827f951fc2333f64670f19100cbcc8a
2023-06-19 11:19:34 +01:00
Stephen Finucane 1331c650b0 db: Don't rely on autocommit behavior
Resolve the following RemovedIn20Warning warning:

  The current statement is being autocommitted using implicit
  autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
  method of Engine or Connection in order to use an explicit transaction
  for DML and DDL statements.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ibe7503002be9f7cdf13259379ebbf6e9c329a028
2023-06-19 10:56:19 +01:00
Stephen Finucane 591d0ba1be db: Replace use of strings in join, defer operations
Resolve the following RemovedIn20Warning warnings:

  Using strings to indicate column or relationship paths in loader
  options is deprecated and will be removed in SQLAlchemy 2.0. Please
  use the class-bound attribute directly.

  Using strings to indicate relationship names in Query.join() is
  deprecated and will be removed in SQLAlchemy 2.0. Please use the
  class-bound attribute directly.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I81a506281a0cece77cad41c5290e41fa3d7ef246
2023-06-19 10:45:58 +01:00
Stephen Finucane 6f61fa05c5 db: Remove use of 'bind' arguments
Resolve the following RemovedIn20Warning warnings:

  The MetaData.bind argument is deprecated and will be removed in
  SQLAlchemy 2.0.

  The ``bind`` argument for schema methods that invoke SQL against an
  engine or connection will be required in SQLAlchemy 2.0.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Idcd26c539f6c46dac245e6a445e64a0bf1596379
2023-06-19 10:45:58 +01:00
Stephen Finucane 304a433b64 db: Update 'select()' calls
Resolve the following sqlalchemy.exc.RemovedIn20Warning warning:

  The legacy calling style of select() is deprecated and will be removed
  in SQLAlchemy 2.0.  Please use the new calling style described at
  select().

For more information, refer to http://sqlalche.me/e/b8d9.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Id64999a7aa6081a5b1f2ac12674204bd02f2c012
2023-06-19 10:45:58 +01:00
Stephen Finucane 60e86b721e db: Replace use of 'autoload' parameter
Resolve the following RemovedIn20Warning warnings:

  The autoload parameter is deprecated and will be removed in version
  2.0.  Please use the autoload_with parameter, passing an engine or
  connection.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I2a626bd60ecc824d9328d08e582d78d5cb568f5a
2023-06-19 10:45:58 +01:00
Stephen Finucane 15f5e3f08a db: Remove use of Query.get() method
Actually, not really. We just remove the warning. Apparently it's not
used here.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ie6c45772247850d40b0cf4b135e5a3028b1758af
2023-06-19 10:45:58 +01:00
Stephen Finucane 9d2cc71975 db: Replace use of Engine.execute() method
Resolve the following RemovedIn20Warning warning:

  The Engine.execute() method is considered legacy as of the 1.x
  series of SQLAlchemy and will be removed in 2.0.

This one is pretty simple.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Iec56e6bff476ae9b68a4821be9cc93e3af84a325
2023-06-19 10:45:58 +01:00
Stephen Finucane 43a5f3984e db: Remove layer of indirection
We don't have another ORM to content with here. Simplify
'heat.db.sqlalchemy' to 'heat.db'.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Id1db6c0ff126859f436c6c9b1187c250f38ebb62
2023-03-25 12:02:27 +09:00
Stephen Finucane 998d8df420 db: Remove legacy migrations
Unlike Cinder, Nova etc., we haven't had any sqlalchemy-migrate-based
migrations in Heat for some time. That means we don't need a transition
period to handle users that may not have the latest
sqlalchemy-migrate-based migration and we can remove these migrations
immediately. Doing so also means we can also remove sqlalchemy-migrate
from our dependencies. Hurrah!

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I825c58a42e9d430c9d660dae9bb70d1cb6f91079
2023-03-25 03:01:02 +00:00
Stephen Finucane 64621053c2 db: Migrate to alembic
This is significantly easier than Cinder, Nova etc. as Heat hasn't had
any database migrations in multiple cycles, meaning we don't need to
worry about having to apply any older sqlalchemy-migrate migrations
before switching to alembic. Instead, we simply need to determine we're
upgrading a deployment that was previously using sqlalchemy-migrate,
upgrading a deployment that has already migrated to alembic, or
deploying a new deployment, adjusting accordingly.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I808af9cb21ba21808209b1daddac7426f4cad310
2023-03-25 12:00:35 +09:00
Stephen Finucane 2cf28d5adf db: Sync alembic, sqlalchemy-migrate migrations
Alembic's auto-generate functionality doesn't preserve index names or
column order. This causes comparisons between databases to fail. Update
the initial alembic migration to match the schema's generated by
sqlalchemy-migrate.

Change-Id: Ie8d98731c61434e85896062fe05137cbadf0eaa5
2023-03-23 11:01:07 +00:00
Stephen Finucane 56bcf53bdd db: Add initial alembic migration
This was generated using a different method to the one I've used
previously [1][2]. Instead of manually converting sqlalchemy-migrate
calls to alembic equivalents, we generate it automatically based on our
models using alembic's auto-generate functionality:

  alembic --config heat/db/sqlalchemy/alembic.ini \
    revision -m 'Initial revision 2'
  alembic --config heat/db/sqlalchemy/alembic.ini \
    upgrade head
  alembic --config heat/db/sqlalchemy/alembic.ini \
    revision -m 'Initial revision 2' --autogenerate

The two files were then combined, formatting and imports adjusted, and
schema of this and the sqlalchemy-migrate migrations compared. The
latter step was done by comparing the schemas of a database created by
the sqlalchemy-migrate tool to the one created by alembic:

  python heat/db/sqlalchemy/migrate_repo/manage.py version_control \
    sqlite:///heat-old.db heat/db/sqlalchemy/migrate_repo/ 072
  python heat/db/sqlalchemy/migrate_repo/manage.py upgrade \
    sqlite:///heat-old.db heat/db/sqlalchemy/migrate_repo

With the two databases created, we can compare them using the
methodologies described in [3].

[1] https://review.opendev.org/c/openstack/nova/+/799527
[2] https://review.opendev.org/c/openstack/cinder/+/786931
[3] https://that.guru/blog/comparing-nova-db-migrations/

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ie32fbb42576760f2c4a52d8305823b99df2b41ba
2023-03-23 11:01:07 +00:00
Stephen Finucane ac25a28d1f db: Wire up for auto-generation
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ic1544f3c5af482c8931b77a7faea059cc3c497dc
2023-03-23 11:01:07 +00:00
Stephen Finucane ecad14368e db: Add initial alembic configuration
No migrations yet: this is simply the output of 'alembic init' with some
minor tweaks.

A note about ordering of requirements is removed from requirements.txt:
it is no longer true with the new dependency resolver introduced in pip
20.3.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I7790222ab5eaf6c47dd386fe472275a3037a9898
2023-03-23 11:01:07 +00:00
Stephen Finucane ebf6886b4a db: Remove unused util methods
These are unused since change Icdab0db3f3371cd4eb8a8fb11cbc2328c0a830e7
way back in 2016.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I2e337ff0fbbded0c0cd42060aa35b4e0787d0d5f
2023-03-23 10:33:46 +00:00
Stephen Finucane 25f1cc2e25 Integrate pre-commit
Makes my life easier wrt testing.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5a79494eb81622be5e56686a1995eb2f0bddb085
2023-03-22 17:16:30 +00:00
Zuul a7ac8f00a2 Merge "Replace deprecated Query.as_scalar()" 2023-01-24 09:08:00 +00:00
ramishra 9407b4897e Ignore resource_data decryption errors
If the auth_encryption_key changes We can possibly
ignore these errors when deleting stacks.

Task: 42056
Change-Id: I326e415db194a5b9c67acd038d7d2d993293ecb3
2022-11-15 09:45:07 +05:30
Takashi Kajinami ddbf9db52e Set cache_ok to avoid further SAWarning
This change fixes the one remaining usage without the flag which was
missed by the previous attempt[1].

[1] 14d2d859ef

Change-Id: Ic0b34b11d669851e47cae670f5299f934f396257
2022-11-01 15:27:24 +09:00
ramishra af5ffbe019 Replace deprecated Query.as_scalar()
This has been deprecated since SQLAlchemy 1.4.

Change-Id: Id3edcc188bb44d9752fe1bd7625308c5abed4d9c
2022-09-06 10:37:03 +09:00
Stephen Finucane ba81ce0cb1 Fix compatibility with oslo.db 12.1.0
oslo.db 12.1.0 has changed the default value for the 'autocommit'
parameter of 'LegacyEngineFacade' from 'True' to 'False'. This is a
necessary step to ensure compatibility with SQLAlchemy 2.0. However, we
are currently relying on the autocommit behavior and need changes to
explicitly manage sessions. Until that happens, we need to override the
default.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I57b8cce28c2ad7899b831067c94b9d913f3f1334
2022-09-05 17:40:38 +01:00
Takashi Kajinami 14d2d859ef Set cache_ok to avoid SAWarning
Since sqlalchemy 1.4.14[1], the following warning is raised unless
the cache_ok flag is explicitly set in classes inheriting
TypeDecorator.

SAWarning: TypeDecorator Json() will not produce a cache key because
the ``cache_ok`` attribute is not set to True.  This can have
significant performance implications including some performance
degradations in comparison to prior SQLAlchemy versions.
Set this attribute to True if this type object's state is safe to use
in a cache key, or False to disable this warning. (Background on this
error at: https://sqlalche.me/e/14/cprf)

Because the flag was set to True by default in older releases, this
adds explicit cache_ok = True, to restore the previous behavior in
Wallaby and older. (wallaby u-c has SQLAlchemy===1.3.23 while xena u-c
has SQLAlchemy===1.4.23)

[1] 6967b45020

Change-Id: I6b84a626994543468f49373a554de44025576a9a
2022-06-15 10:35:35 +09:00
Ian Wienand bf9186a37e Fallback to upstream for Fedora image
OpenDev infra only keep around the latest two Fedora releases in their
mirrors.  Probe for the image from the local test mirror, but if not
found, fallback to upstream.  This will be much less reliable, but can
avoid gate breakage until new images can be used.

Also, use endpoint_type when creating keystoneclient

Keystone admin endpoint has been removed from devstack with[1].
This would use the public endpoint by default.

Change-Id: I96ab14871ee8c5d5b83cc0cd4abc840ef0218ca8
2021-11-12 15:32:19 +05:30
Brendan Shephard f341395a46 Fix SQLAlchemy use for version 1.4
The change to SQLAlchemy upper constraints has
broken some of the tests in Heat. Syntax and usage
needs to be updated accordingly to ensure tests work
with the 1.4 version of SQLAlchemy.

Reference:
https://storyboard.openstack.org/#!/story/2008974

Story: 2008974
Task: 42616

Change-Id: Id4755c8c58401ebfd42a8c57f94d111de9f74ac1
2021-06-16 07:55:25 +10:00
ramishra bfbaab107d Don't update status for replaced resource
With I04e7ad90944c2d03ce0e59ba16af9d60d6e01222 we allowed
update of previously-existing resource if replacement
creation has failed during last update. However, we force
update the status of the existing resource to 'COMPLETE'.
Therefore, in the next update if properties/type has not
changed for the resource it won't try to update/replace
the previously existing resource resulting in false positive
of stack updated successfully.

Task: 42194
Change-Id: Icc90a921ec67e49aec2c4acfad72235c57c78421
2021-04-01 19:39:58 +05:30
Rico Lin 8daa7e9389 Allow using database configs on db retry
Allow following db configs when calling wrap_db_retry:
    * database.db_max_retries
    * database.db_retry_interval
    * database.db_inc_retry_interval
    * database.db_max_retry_interval
So database cofig can now control db retries.
Please reference [1] for what each config options can do.

[1] https://opendev.org/openstack/oslo.db/src/branch/master/oslo_db/options.py

Change-Id: I034625733c2d22f0f5635f58e9df3d5785e58cf5
2021-02-27 15:42:07 +08:00
Kevin Carter c37a72e471 Fix iter bug with SQlAlchemy 1.3.23
With SQlAlchemy 1.3.23 constraint.copy() fails with error[0].
This seems like a regression after[1].

This patch fixes the issue by providing the ``target_table``
argument to c.copy().

[0] http://paste.openstack.org/show/802486
[1] 7dd3381edb

Change-Id: Ia25b8443bbe576f73fb7debe54f307deadf34e04
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2021-02-18 09:59:50 +05:30
Zane Bitter 2c28e6b978 Rename variables in sort key validation
Avoid unnecessarily divisive terms.

Change-Id: Ida9384c849cd664c5fc5d31449350d7cb6a68b78
2020-07-14 14:53:59 -04:00
Zuul 0e7d99ac3f Merge "Logging Consistency" 2020-05-04 10:12:13 +00:00
Andreas Jaeger 7f770b95b8 Update pep8 rules
Remove E402 from flake8 list, issues are fixed or addressed with noqa.
Remove E305 as comment, the warning is enabled.
Enable E241 and fix all problems

Change-Id: Ifd2bbce98be4206908da769df686a370328c7ee1
2020-04-30 16:33:43 +02:00
Marc Methot 156f276665 Logging Consistency
A simple matter of having log lines fit
with all the others. It will now match the
ambiguous standard.

Also change comments inline to alias with log fixes.

Change-Id: I4a2ed6134f70c2965811f75ccb6ab8221fa0e960
Story: #2007319
Task: 38830
2020-04-27 06:34:31 +00:00
Zuul a996a13713 Merge "Retry transactions for DBConnectionError" 2020-04-26 08:47:04 +00:00
Hervé Beraud 5877da06a1 Remove six and python 2.7 full support
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.

Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.

To review this patch you can use the six documentation [1] to obtain help and
understand choices.

Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.

Latin-1 is equal to iso-8859-1 [3].

This encoding is the default encoding [4] of certain descriptive HTTP
headers.

I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).

HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].

Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'

[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp

Patch 4 of a serie of 28 patches

Change-Id: I871c2dad10abc35790e730c7c4c5272f499b7623
2020-04-22 12:23:44 +02:00
Rabi Mishra 96c4dbe3ea Keep db model and migration scripts in sync
We seem to be using sqlalchemy.Text for 'status_reason' column for
some tables in model and heat type LONGTEXT in migration scripts.

Change-Id: I8b792936722f6fed2dd4e579aa41f6bf6aa27f7c
2020-04-07 20:33:50 +05:30
Rabi Mishra 999a4d8bcb Retry transactions for DBConnectionError
Like DbDeadlock retry in DBConnectionError as oslo.db and SQLAlchemy
already automatically reconnect, if the database was disconnected
and a new transaction is beginning.

Change-Id: I479408fc47753e5971b347b379079b30e035cc08
2020-03-13 09:18:46 +00:00
Zuul 68f233a5f0 Merge "DB: Never retry a subtransaction" 2019-12-19 03:42:51 +00:00