We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.
Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc
Swift removed sha1 from supported digests with [1] and
that broked ironic tinyipa job. Temorary add sha1 to
allowed_digests until it's fixed in ironic.
[1] https://review.opendev.org/c/openstack/swift/+/525771
Story: 2010068
Task: 45539
Change-Id: I68dfc472ce901058b6a7d691c98ed1641d431e54
Previously, Swift's backing disk were not be mounted after reboots,
causing swift-proxy-server service to fail with cryptic error
messages such as 'proxy-server: ERROR Insufficient Storage'. Now,
we use Dan Smith' create_disk function from functions to create
the backing disk for us and add it to /etc/fstab.
Change-Id: I9cbccc87bc94a55b58e9badf3fdb127d6f1cf599
We can use the devstack-admin cloud configuration everywhere now
and don't need to set environment variables with cloud credentials
any longer.
Fix the swift setup, where some more options need to be explicitly
specified now and the default OS_CLOUD setting overridden.
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I86ffa9cd52454f1c1c72d29b3a0e0caa3e44b829
Keystone no longer has any special functionality hidden behind the admin
site. KEYSTONE_AUTH_URI which used to point to the admin site has long
ago been changed to be a copy of KEYSTONE_SERVICE_URI, which points to
the public site.
Drop all KEYSTONE_AUTH_* variables except KEYSTONE_AUTH_URI which may
still be in use in some plugins.
This also allows to finally drop the fixup_keystone() function.
Change-Id: I549f3cadc27d137e014241cdd47e90267859c848
This change fixes the empty value set to the gid option in rsyncd.conf,
which was caused by reference to the invalid USER_GROUP variable, and
ensures the option is set to the group which STACK_USER belongs to.
This also fixes duplicate declaration of the local user_group variable.
Closes-Bug: #1940742
Change-Id: Ifd0a5ef0bc5f3647f43b169df1f7176393971853
Since the introduction of I8f24b839bf42e2fb9803dc7df3a30ae20cf264
s-proxy is no longer able to launch as keystonemiddleware (listed under
test-requirements.txt) has not been installed.
keystonemiddleware is listed as extras requirements in swift
- e0d46d77fa/setup.cfg (L79)
Let's install swift keystone extra requirements also.
Closes-Bug: #1909018
Change-Id: I02c692e95d70017eea03d82d75ae6c5e87bde8b1
Keystone no longer has any special functionality hidden behind the admin
endpoint. Stop referencing it in consumers, so it can later be dropped
completely.
Change-Id: I04a5d77908005268cc7c59e7e9ddeea70f6732e2
We need an appropriate auth_uri for s3token to be able to contact
Keystone.
Since tempauth is always enabled, we want to delay the auth decision
until after tempauth has had a chance to try.
Change-Id: Ie4ff33a617b9dc74d51d037ec8ebd0d9787dd76d
For a long time, swift3 recommended a pipeline like
... swift3 s3token authtoken keystoneauth ...
This led to inefficiencies where the proxy would first contact Keystone
to validate the S3 signature and issue a token, then contact Keystone
*again* to validate the token ID that was just issued.
After s3token moved into the swift3 repo, it was improved to be able
to put all of the headers into the WSGI environment that Swift's
keystoneauth middleware expected and the recommended pipeline was
changed to something like
... authtoken s3api s3token keystoneauth ...
At the time, the old order would still work, it would just be less
efficient. When support was added for Keystone v3, however, the new
order became mandatory.
All of that happened before swift3 moved back into Swift as s3api, but
the pipeline placement problems are the same: Keystone users won't be
able to use the S3 api with the current order.
Change-Id: Id0659f109cc2fc12ddb371df0b26812ba8c442d9
Related-Change: I21e38884a2aefbb94b76c76deccd815f01db7362
Related-Change: Ic9af387b9192f285f0f486e7171eefb23968007e
This is no longer being used due to Keystone PKI tokens no longer
being implemented.
In order to not break backward compatibility we create a new function
that is to be used instead and deprecate the old one. Modify the old
function to ignore the 3rd argument and display a deprecation warning.
Adjust callers to no longer create and set that directory, calling the
new function instead.
Change-Id: Id0dec1ba72467cce5cacfcfdb2bc0af2bd3a3610
swift3 is no longer actively maintained in the upstream.
That has been moved to Swift repository as s3api so we should
use s3api middleware instead. As well as swift3, s3token is
also maintained in Swift upstream.
Change-Id: I4582d81da066ab53e6f11ad1df7af91425f2b0ca
Fix a few path issues where we didn't properly use NOVA_BIN_DIR /
SWIFT_BIN_DIR.
This is part of the effort to start using a virtualenv for openstack
services.
Change-Id: I6eb383db65cc902c67c43e5cb1a16a9716a914b2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This adds the necessary fixes to pass a devstack run
on openSUSE Tumbleweed. Also removes opensuse 42.2 as it
is EOL for some time already and no longer actively tested
in the OpenStack infra.
Depends-On: I1b68c08c07cf6653ea58506f738cbe0054b38f3a
Change-Id: I2894482deef063fd02b0818c695a2ddbf6767039
- There are some locations where we need the raw IPv6 address instead of the
url-quoted version enclosed in brackets.
- Make nova-api-metadata service listen on IPv6 when we need that.
- Use SERVICE_HOST instead of HOST_IP for TLS_IP.
Change-Id: Id074be38ee95754e88b7219de7d9beb06f796fad
Partial-Bug: 1656329
All the evidence from [1] suggests that on opensuse swift-init is not
detaching the daemon process correctly. It's possible there's a pipe
still in play that somehow holds our ansible-streamer open.
This is a minimal fix to avoid swift-init. Although it's possible in
non-default paths to still use swift-init (and hence possibly hit
another variant of this issue), after discussions with swift
developers it was decided the intersection of tests running under our
current ansible, on suse, that would enable these services is
sufficiently small that this is the best course for now.
[1] https://storyboard.openstack.org/#!/story/2001528
Change-Id: I1b68c08c07cf6653ea58506f738cbe0054b38f3a
This reverts commit ef5ebed6c9.
The problem here is a backwards-incompatible change to
configure_auth_token_middleware. Plugins are still passing a
"signing_dir" which is interpreted now as the "section" argument
... this leads to an interesting red-herring issue; because "v" is a
gnu sed command for checking the version, a signing_dir of "/var/..."
(as done in most plugins) gives the weird error:
sed: -e expression #1, char 32: expected newer version of sed
I think we'll either need a new function, or dummy arguments to get
this back in.
Change-Id: I2098d4eb2747282622cf486fa7dbf216f932f58b
PKI tokens have been actively deprecated from keystone and there are
deprecations being emitted from keystonemiddleware. Because of this we
no longer need an auth cache directory in the services where the PKI
certifcates used to be stored.
Remove the creation and use of all these AUTH_CACHE directories.
Change-Id: I5680376e70e74882e9fdb87ee1b95d5f40570ad7
The pre-existing configuration for swift on devstack set's the
*-server's devices option (the root of the servers list of devices) to:
devices = /opt/stack/data/swift/1
where "1" is the node_number, and will be 2, 3, ... N if the devstack
machine is built with more than one swift node/device (pretty sure no
one does that on devstack ever).
The device(s) in the rings are named (perhaps confusingly similar to the
swift loopback image) just "sdb1", so all storage servers expect to have
a $STACK_USER writeable file system at:
os.path.join(<devices_root>, "sdb1")
That directory does not exist when you start up a devstack [1].
Currently Swift's object-server's require that directory exist before
they write data into it (even with mount_check = false!).
Unfortunately however, with mount_check=false the account/container
servers are able to create the device directory when it does not exist
[2]. Which can lead to some unfortunate results with permissions on
some deployments using mount_check = false (e.g. testing or
containerized environments). Fixing this issue [3] uncovered the
previously benign [4] mis-configuration in devstack.
Attempting
1. It was lost a long while ago I7c65303791689523f02e5ae44483a6c50b2eed1e
2. Essentially they want to:
mkdir -p /opt/stack/data/swift/1/sdb1/containers/<part#>
... but end up creating the "sdb1" dir too!
3. I3362a6ebff423016bb367b4b6b322bb41ae08764
4. Benign because the object-server share their device with the
account-container devices and they would create the dirs before trying
to write an object. It was incorrect, but worked by happenstance, which
is nearly as good as worked on purpose.
Change-Id: I52c4ecb70b1ae47e613ba243da5a4d94e5adedf2
... to be used with domain_remap. Swift will start functionally testing
domain_remap in I63428132283986bda9e5c082ffe85741449b71ba.
Change-Id: I4c1ab06d040d91fd8c314d0aa2cecbbb00adf8ad
Otherwise, we've seen intermittent "Unable to establish connection"
failures, with the main devstack log reporting things like
2017-07-19 13:54:29.973 -> start proxy service
2017-07-19 13:54:30.082 -> start OSC to store temp url key
2017-07-19 13:54:31.908 -> OSC reports failure
Meanwhile, the s-proxy screen session tells us things like
Jul 19 13:54:31.919988 -> start child worker
Jul 19 13:54:32.206598 -> still loading the WSGI app
... and ports aren't actually bound until *after* the app is loaded.
Add a wait_for_service call to wait for the proxy to come up.
Change-Id: I1a722de31b144797230991700e110353a2d937dd
The old implementation for is_$service_enabled simply checked if any of
the subservices were enabled and if so the service was considered to be
enabled. This makes disabling services complicated as it means you have
to list every single subservice which can and do change over time.
Instead also check if the generic service name is in the disabled
services list and if so don't treat the service as enabled.
Change-Id: I7fe4dfca2cd9c15069d50a04161a29c5638291cb
As part of getting swift's functional testing to work properly through
the tls-proxy we need to increase the allowed request header size in
apache. This was a non issue without tls proxy as requests hit the
eventlet webserver directly which was configured via the swift config
which sets this relatively large limit (by default devstack configures
swift to have a header size limit of 16384).
Now we pass in an optional parameter to start_tls_proxy that includes
the desired header size. lib/swift then passes in the value it also
configures in its swift.conf.
If not explicitly set we default to 8190 which is apache2's default.
Change-Id: Ib2811c8d3cbb49cf94b70294788526b15a798edd
Because the swift functests (which use test.conf) run out of a
virtualenv they don't get access to the system wide trust of the
devstack CA. Handle this by explicitly configuring the cafile to trust
in the test.conf file.
We also set the web_front_end to apache2 as that is what is terminating
TLS for us. The tests handle different web server behaviors using this
flag.
Swift's functests will need to read these values in and properly
configure things on its end.
Change-Id: I4cdba36ccab6acd76205184882ee29e4f1e12333
This defines a new function get_notification_url, which returns the URL
of RabbitMQ when you want connect to it, and uses in
ceilometermiddleware. This fixes an issue when we try to use AMQP for
RPC, but not for notifications.
Change-Id: I14450b2440806a17a90e5ddefc243868fdbe4f2c
Swift proxy logs to syslog during the devstack-gate tempest runs. To
better capture the swift logs increase the rsyslog buffer size to 6k
bytes allowing for longer messages like tracebacks.
This was setup by openstack-infra previous during our diskimage
builds.
I03e42964e14d9f930c07ed047851bdf775639c59
Change-Id: Iaa232335865410600c93f47d4777ed4f1bce08e2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Until we can test with a version of swiftclient that knows how to eat
auth_uri, swift still needs a working gate.
Change-Id: I09f9ad5c87b542df962a79898e06fbf1e968b1e3
Related-Change: I46294fb24e3c23fa19fcfd7d6c9ee8a932354702
Related-Change: Ie427f3b0b9eb834ff940fa5d52444a5a6cdcab15
tls-proxy is the way we're now doing a standard install using https
between services. There is a lot more work to make services directly
handle https, and having python daemons do that directly is a bit of
an anti pattern. Nothing currently tests this in project-config from
my recent grepping, so in the interest of long term maintenance,
delete it all.
Change-Id: I910df4ceab6f24f3d9c484e0433c93b06f17d6e1
The swift functional tests use a config which requires keystone ports,
we're about to make those go away. This exposes the actual auth_uri to
swift for consumption.
Change-Id: I5868dfdb8e5f0972ba04e359d212b04351502436
Currently Devstack starts all Swift services, including those
in charge of "consistency convergence" (remember Swift is eventually
consistent), data scrubbing, hard-deletion (*-reaper services)
cleanup.
But when running with Replication Factor 1 some of those services
are not needed at all. Besides, the fonctionnalities provided by
some of these services are not tested at all (neither in Tempest
nor in Swift functional tests).
Thus, in light of saving some Mo of RAM, this patch introduces a config
flag to start only a minimal set of Swift services, just what's required
to make all of our current tests pass.
The default value for this new config flag is set to start all services,
that is to maintain Devstack's current behavior.
For sake of completeness, here is the list of services that are not
going to be started is the config flag is toggled, and the associated RSS
according to our peakmem_tracker
40004 swift-object-replicator /etc/swift/object-server/1.conf
34320 swift-container-replicator /etc/swift/container-server/1.conf
33584 swift-object-auditor /etc/swift/object-server/1.conf
33328 swift-object-reconstructor /etc/swift/object-server/1.conf
31936 swift-object-updater /etc/swift/object-server/1.conf
31492 swift-account-reaper /etc/swift/account-server/1.conf
31076 swift-account-replicator /etc/swift/account-server/1.conf
29540 swift-container-updater /etc/swift/container-server/1.conf
29220 swift-account-auditor /etc/swift/account-server/1.conf
29036 swift-container-auditor /etc/swift/container-server/1.conf
So we are looking at saving at most ~350Mo of RAM (could be less
because RSS doesn't account for shared memory).
A follow-up patch will soon be proposed in devstack-gate to not run
those additional services in our Gate jobs.
Change-Id: I8a0d03ac0296a74e38efd185beb8513866eaf0c4
During the PTG there was a discussion that the screen developer
workflow wasn't nearly as useful as it once was. There were now too
many services to see them all on one screen, and one of the most
common service restart scenarios was not restarting one service, but a
bunch to get code to take effect.
This implements a 3rd way of running services instead of direct
forking via bash, or running under screen, which is running as systemd
units.
Logging is adjusted because it's redundant to log datetime in oslo.log
when journald has that.
Swift needed to have services launched by absolute path to work.
This is disabled by default, but with instructions on using it. The
long term intent is to make this the way to run devstack, which would
be the same between both the gate and local use.
Some changes were also needed to run_process to pass the run User
in. A hack around the keystone uwsgi launcher was done at the same
time to remove a run_process feature that only keystone uwsgi uses.
Change-Id: I836bf27c4cfdc449628aa7641fb96a5489d5d4e7
The proliferation of internal/admin endpoints is mostly legacy and
based on some specific deployment patterns. These are not used by
everyone, and for the devstack case aren't really that useful. We
should simplify our service catalog down to the minimum we need for
development.
Change-Id: Ided7a65c81b3a0b56f0184847fc82e17c29a771e
* iniuncomment followed by iniset for reseller_prefix just adds a
duplicate line in the config file that configparser does not like
so just remove the uncomment
* fall back to http:// url for glance->swift keystone authentication
* insecure flag to talk to swift
Depends-On: I51d56d16a5b175bd45dee09edc0b2748d72a5d06
Change-Id: I02ed01e20f8dce195c51273e8384130af53384ce
This commit switches how scripts we use to launch the installed version
in the path. Previously the scripts were manually executed in the source
repo, but this has issues if you're trying to run with py3 in a system
where python == py2. Setuptools already does the shebang magic for us
at install time, so we just need to use the installed version of the
script.
Change-Id: Iaa4d80ec607a2aa200400330e16cad3a4ca782ac
Stud is now abandonware (see https://github.com/bumptech/stud) and is
not packaged in xenial. Lets use Apache for SSL termination since its
there already.
Change-Id: Ifcba410f5969521e8b3d30f02795541c1661f83a
lib/keystone sets KEYSTONE_AUTH_URI and KEYSTONE_SERVICE_URI that
other projects should use rather than building the URL themselves.
This will allow us to more easily drop the port altogether.
Change-Id: I7467aae680215f3045d32a088af2187e1eba8169
This patch enables account management by default in Swift. This will be
leveraged by Tempest test cases validating account management APIs.
Depends-On: Id29f5ca48f92cd139535be7064107b8a61b02856
Change-Id: Ic01432939ed9b4cf0cbf20e3244d4d76847f539f
This reverts commit 7d1ec43004.
This broke the sahara and layer4 dsvm jobs. The layer4 job
is voting on tempest changes so tempest is also broken.
Change-Id: Ide69f10cd85bf7ff0d86bc8cba56dedd26850362
Partial-Bug: #1573868
Martin Kopec
yatinkarel