Commit Graph

121 Commits

Author SHA1 Message Date
Corey Bryant 72e22a94ed Mock datetime instead of redefining standard library
This fixes up the test to mock the datetime calls specifically in
the test that needs to mock them, rather than the prior approach
which was redefining the behavior of the standard library function
globally.

Closes-Bug: #2045588
Change-Id: I9c585c91c5527a61e4a8dbcba0a8a01108cd9b2e
2023-12-04 15:49:36 -05:00
Alex Kavanagh 955f7e710c Improve platform mocking
Patch out charmhelpers.osplatform.get_platform() and
charmhelpers.core.host.lsb_release() globally in the unit tests to
insulate the unit tests from the platform that the unit tests are being
run on.

Change-Id: I3ca0e47cc8cd0bc8475c0c4b0ef8d013d52401ed
2023-10-23 18:12:50 +01:00
Zuul 51bfe2adca Merge "Add package-upgrade action" 2023-05-31 21:19:51 +00:00
Corey Bryant 3a5b5872c5 Add package-upgrade action
The package-upgrade action performs package upgrades for the current
OpenStack release.

The code path used is similar to the openstack-upgrade action, with the
difference being that package-upgrade will not execute if an openstack
upgrade is available (based on the openstack-origin setting).

This change includes a charm-helpers sync.

Change-Id: I0c7184bba29731354e52dc28e3a4dd6f282fa843
2023-05-26 15:07:07 +01:00
James Page d9a8025ac3 conntrack: set sane sysctl defaults
swift-storage is often deployed alongside nova-compute where it
inherits some sane defaults for sysctl values, specifically around
conntrack configuration.

If its deployed standalone it does not recieve the same tuning,
but its very much applicable as access between swift-storage nodes
for rsync traffic is firewall limited so makes a high demand
on conntrack on the server.

Lift and shift the defaults from the nova-compute charm.

Change-Id: Iedd27a51ff93fd1670a418e1434c94875fe21643
Closes-Bug: 1879121
2022-07-11 14:40:52 +01:00
Erlon R. Cruz 4337b753f5 Fix swift replication errors
There are several errors related to the swift replication service.
The swift storage charm is not properly restarting the services
after configuration changes, the correct object_lockup_timeout
value (that per the behaviour observerd in our environments must
be greater than object_rsync_timeout) and we also needed to fix
the object replicator config file to honor the
object-handoffs-first configuration.

This patch along with the swift proxy-change should fix the
currently known replication problems.

Closes-bug: #1903762
Depends-on: I87eb23de94e3f2f5b06d44df1f8bd9d2324456a0
Change-Id: I87eb23de94e3f2f5b06d44df1f8bd9d2324c8470
2021-01-05 11:19:13 -03:00
Alex Kavanagh 494908cee7 Sync libraries & common files prior to freeze
* charm-helpers sync for classic charms
* charms.ceph sync for ceph charms
* rebuild for reactive charms
* sync tox.ini files as needed
* sync requirements.txt files to sync to standard

Change-Id: I28bc6125b817c2f1608157509c27d54dab9541a5
2020-09-27 19:20:00 +01:00
Andrea Ieri 66c14b2aff Ensure lockup_timeout is never < 2*rsync_timeout
Change-Id: Id0afc210b7be8ac7c740753ef06b51db0bae410c
Closes-Bug: 1888725
2020-07-23 15:57:46 -04:00
Zuul f57587b2ff Merge "Add config option for fallocate_reserve" 2020-07-23 16:03:44 +00:00
Drew Freiberger d31095d482 Add config option for fallocate_reserve
In order to prevent disks from filling up, add the option to configure
fallocate_reserve parameter in account-server.conf,
container-server.conf, and object-server.conf, as described in
https://docs.openstack.org/swift/latest/admin_guide.html#preventing-disk-full-scenarios

This change adds the config option file-allocation-reserve with a
default of 0.

Change-Id: Ib396a151250bb8d1733f9b5b9cab7eb506c9f6c6
Closes-Bug: 1872069
2020-07-22 17:30:18 -04:00
Drew Freiberger 74daa465d6 Add monitoring for object-replicator logs disappearing
As noted in lp#1691570, there is an issue with storage I/O during
coro-thread cleanup that affects the swift-object-replicator's ability
to complete replication successfully.  This is most easily witnessed by
the lack of the every 5 minute replicated percentage complete messages
that should come from the swift-object-replicator daemon to syslog.
This patch monitors for and alerts on the condition of the "replicated"
line missing from syslog within the past 15 minutes.

Change-Id: Ieb15da3f3f67fa9bcad03151e36c70faae4c36c9
Closes-Bug: 1691570
2020-07-21 15:31:06 -05:00
Hervé Beraud 3dd824c5b9 Use unittest.mock instead of mock
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.

Change-Id: Id4029239c345655ea7ffc90b5969b29b47a84955
2020-06-09 00:02:15 +02:00
Drew Freiberger 9809a3b347 Add support for object replication handoffs_first
In extreme cases of object rebalance or node failure, swift environments
with millions of objects may have a hard time with objects landing in
handoff or misplaced partitions and the operator of the cloud may need
to prioritize migrating data in handoff partitions back to primary
partitions over the 3 primary partitions replicating amongst each other.

To allow for this, the object-server.conf [object-replicator]
configuration can have handoffs_first set to True which will force
the object-replicators to focus on handoff partitions before syncing
primary partitions for any given object partition.

Change-Id: I8b44c287567a0e6d634def0b13baf0fe4ad4aa7b
Closes-Bug: 1878087
2020-05-11 15:01:22 -05:00
Tytus Kurek fc261a5adc Swift Global Cluster
This patchset adds a support for Swift Global Cluster feature as
described at:

https://docs.openstack.org/swift/latest/overview_global_cluster.html

It allows specifying 'region' config option as part of the deployment.
Theregion is reflected later on in the Swift rings.

Change-Id: I11b6c7802e5bfbd61b06e4d11c65804a165781b6
Partial-Bug: 1815879
Needed-By: I406445493e2226aa5ae40a09c9053ac8633a46e9
2019-12-17 10:08:13 +00:00
Edward Hope-Morley 60dd2f0189 Charmhelpers sync to get vaultlocker fixes
Also gate checking vault context completing on whether
dependencies are installed.

Change-Id: Ib424abe608081da21207db262fb82362f23fe6ca
Closes-Bug: #1849323
2019-12-11 14:30:27 +00:00
Corey Bryant fb6e21b51f py3: Switch packages to Python 3 for train and later
Switch package install to Python 3 for OpenStack Train and later.

When upgrading, remove any python-* packages that were explicitly
installated and then autoremove --purge any dependencies that are
no longer required.

This patch also includes the following related changes:
  * Use the common files package, swift, rather than python-swift
    when using a package name to determine release.

Also add OS_* to tox.ini to allow functional tests to execute.

Change-Id: I121af845bf11c22052479a497b196a4670021256
2019-10-14 08:29:59 +00:00
Alex Kavanagh c9b558b0ff Ensure that get_host_ip() is mocked out in test
The test test_setup_ufw() didn't mock out get_host_ip() which meant that
the test leaked and used an actual IP hostname lookup.  This failed when
ubuntu.com's IP address changed.  This patch mocks it out properly.

Change-Id: Ida7aaa7d085cc0316aa496d0a0e24333c5828cee
Closes-Bug: #1832829
2019-06-14 15:11:04 +01:00
Zuul 173f1a99db Merge "Use UUIDs of storage devices in fstab" 2019-05-24 08:46:17 +00:00
tpsilva dc63810cb4 Fix import errors on Nagios check file
After changing the charm runtime to Python 3, some imports are failing
on check_swift_storage.py. This can be fixed by explicitly imporgin the
modules inside the package, instead of just importing the package.

Change-Id: I746071042b669a1725744d4e32ea733eb40b309a
Closes-bug: #1822334
2019-03-29 13:15:10 -03:00
Tytus Kurek edd00bee63 Use UUIDs of storage devices in fstab
This patchset implements a logic to put UUIDS of storage devices in the
fstab instead of device names.

Change-Id: I2170c54b25ac9b3faacb3fd0b2889d86c531395f
Closes-Bug: 1729478
2019-03-13 21:19:14 +00:00
Alex Kavanagh fc81200315 Upgrade the charm to py3 runtime
Change-Id: I98f4e6664080407a045ca5e76db59d46ffa9c38a
2018-10-24 14:57:05 +01:00
Andrew McLeod 0598fd20fb Add nofail to fstab options for loop dev
Some machines will not boot if nofail is excluded

Change-Id: Ib978709943c4aec8f7d405c0b71e1c4d81f3764c
Closes-Bug: 1768666
2018-05-21 10:43:42 -06:00
Zuul b03a8bc5bd Merge "Add support for block device encryption" 2018-05-15 09:56:23 +00:00
Zuul de60721367 Merge "Updated nrpe check for swift replication lag >1 day" 2018-05-11 09:10:19 +00:00
Zuul 5a7c46f897 Merge "fix a typo" 2018-05-09 12:58:28 +00:00
Zuul 2f60ad0e98 Merge "Using assertIsNone() instead of assertEqual(None)" 2018-05-09 12:51:54 +00:00
James Page 591c084532 Add support for block device encryption
Add new secrets-storage relation to vault, supporting the
use of block device encryption using dm-crypt/vaultlocker.

Prepared devices are now recorded in the local unit kv
store; this information is used to provide a list of
configured block devices to the swift-proxy charm,
rather than the previous best guess provided by
determine_block_devices.  This allows us to use the
dm-crypt device name, rather than the underlying
block device.

Encrypted block devices are unlocked on boot using
vaultlocker-decrypt systemd units (enabled by vaultlocker);
/etc/fstab entries for such devices make use of a
x-systemd.requires option to ensure that the block device
is unlocked prior to attempting to mount it.

Add new storage binding to allow charm to be used with
Juju storage.

Add new ephemeral-unmount configuration option to allow
cloud ephemeral storage to be used for testing purposes;
update functional testing to use this option.  The behaviour
of 'overwrite' was changed to accomodate the use of
encrypted block devices.

Change-Id: I9b3f8cd2de412ee96e0139dba4d4abdf998ecaf2
2018-05-08 12:52:37 +01:00
David Ames 18d0a891db Allow GRE traffic in converged architecture
In a converged architecture with storage and compute on the same
host, UFW can get in the way of tunneled traffic interpreting it as
INVALID. UFW makes solving this more difficult than it needs to be.
See http://northernmost.org/blog/gre-tunnels-and-ufw/index.html for
context.

This change updates /etc/ufw/before.rules to add GRE as an allowed
input.

Also, guarantee ufw is installed for LP #1763716

Please review and merge charm-helpers first:
https://github.com/juju/charm-helpers/pull/170

Change-Id: I789854c33e3af12f7412633dbf7c921beb0ed2b5
Closes-Bug: #1757564
Closes-Bug: #1763716
2018-05-03 21:42:46 +02:00
wangqi f9a6b36c93 Using assertIsNone() instead of assertEqual(None)
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)

Change-Id: Ib5dcf2f389dd76f99d87ad22268695f2a702e6b3
2018-04-20 07:33:26 +00:00
Drew Freiberger 9a5314afad Updated nrpe check for swift replication lag >1 day
Change-Id: Ib1104ea86290cf271e63d1a5aa81e503ed5965c3
Closes-Bug: 1758119
2018-04-11 14:53:12 -05:00
Liam Young 6346a7458b Use the loop device in fstab instead of image file
When adding an fstab entry for a loopback device use the explicit
loopback device name rather than the source image file. This prevents
a new loopback device being created implitcitly when mounting the
image.

The unit tests needed updating to reflect that the loopback
device name is used when creating mountpoint names rather than than
the name of the image file. This was pre-existing behaviour.

Change-Id: Ide074310bf7121f1179e0b5237dff6f3da88e24e
Closes-Bug: #1762390
2018-04-10 12:58:05 +00:00
Liam Young f75bd9762a Fix broken fstab entry
Whitespace between the fstab option is not valid, remove it.

Change-Id: I1be789acc7bf92811f8d4e2c0f0661ebf3be1678
Closes-Bug: #1762105
2018-04-10 11:25:14 +00:00
wangqi 1cc0e5dec5 fix a typo
Change-Id: I1581f783a5fab546019e60b35c1df43a1403cd66
2018-04-02 08:16:47 +00:00
Felipe Reyes 1d28ceb648 Resolve hostnames if needed to allow access in ufw
In some cases juju 1.25 (with maas 1.9) may return a hostname in the
private-address field breaking the assumption that private-address will
always be an IP address. This patch uses get_host_ip() to assure an IP
address is given to ufw.

Change-Id: Ib99852c645517cf316adcd02d04428b248fc9724
Closes-Bug: 1747516
2018-03-05 18:16:57 -03:00
Frode Nordahl b819bde429 Revert "Resolve hostnames if needed to allow access in ufw"
We need a new proposal that uses get_host_ip from charmhelpers.contrib.network.ip

Reference: https://review.openstack.org/#/c/542381/

This reverts commit 4896ac5b5b.

Change-Id: Idd16369be776a10ac6332d7d8b2ce65f9f1ad972
2018-03-01 15:19:49 +00:00
Felipe Reyes 4896ac5b5b Resolve hostnames if needed to allow access in ufw
In some cases juju 1.25 (with maas 1.9) may return a hostname in the
private-address field breaking the assumption that private-address will
always be an IP address. This patch uses socket.getaddrinfo() to assure
an IP address is given to ufw.

Change-Id: I99b0110beed6075164eb549ec2433071af699c04
Closes-Bug: 1747516
2018-02-05 18:32:52 -03:00
Ryan Beisner af2eaad19b Sync charm-helpers
Notable issues resolved:

openstack_upgrade_available() broken for swift
https://bugs.launchpad.net/charm-swift-proxy/+bug/1743847

haproxy context doesn't consider bindings
https://bugs.launchpad.net/charm-helpers/+bug/1735421

regression in haproxy check
https://bugs.launchpad.net/charm-helpers/+bug/1743287

Change-Id: Ie6b66f56614d81c600b66bfe00cf9b9bb462a20c
2018-01-19 16:13:12 +02:00
David Ames 5368af6302 Swift storage ACLs
Ensure that only the swift-proxy units and swift-storage peers have
access to direct communication with swift storage daemons.

Charm-helpers sync to include ufw module and the ingress_address and
iter_units_for_relation_name functions.

Please review and merge first:
https://github.com/juju/charm-helpers/pull/35

Closes-Bug: #1727463

Change-Id: Id5677edbc40b0b891cbe66867d39d076a94c5436
2017-11-07 10:24:53 -08:00
Zuul f953a6aa09 Merge "Add statsd metrics support to the swift-storage charm" 2017-11-05 05:27:08 +00:00
James Hebden 0876b4fb73 Add statsd metrics support to the swift-storage charm
By default, statsd metrics can be sent by the swift account,
container and object storage services for diagnostic and
monitoring purposes, but are disabled by default. This change exposes
charm config settings that allow it to be enabled by setting
'statsd_host' to a non-empty value. 'statsd_port' and
'statsd_sample_rate' are also supported for changing the destination
port and rate at which metrics are collected.

Closes-Bug: #1729770

Change-Id: If1bf3ced8a9ed07af81f352eb0263659d147e3aa
2017-11-05 15:23:38 +11:00
James Page b32f68af0f Add network-space support for swift-storage relation
Resolve correct private-address for use on the swift-storage
relation, supporting 'prefer-ipv6' as well as Juju 2.0 network
spaces.

Change-Id: I3ee111c6abdd028c2c29e80dceb99178443da45a
Closes-Bug: 1697491
2017-09-29 11:33:48 +01:00
Edward Hope-Morley 656e79da18 Catch blkid error when device is not yet formatted
When a new device is added to the ring we first try to
identify whether the device is already in the ring by
polling for an fs uuid. If the device has never been
used this is expected to fail so lets catch the error.

Also fixes log message.

Change-Id: I20354dedfa27a6b8dec92828cabb50a20d0d8838
Closes-Bug: 1567198
2017-09-14 13:22:25 -06:00
zhangyangyang f87d83b1ab Change assert(Not)Equals to assert(Not)Equal
According to http://docs.python.org/2/library/unittest.html
assert(Not)Equals is a deprecated alias of assert(Not)Equal.

Change-Id: I4b9e966f4c7cd3df6440f38b56785a38a812e002
Closes-Bug: #1329757
2017-09-11 19:19:52 +08:00
Jenkins 56cf0e8210 Merge "Support null values on recon/replication module" 2017-08-16 16:32:32 +00:00
Alvaro Uria d59759259e Support null values on recon/replication module
Update on nrpe check check_swift_storage:
 * Refactor check_replication function
 * Raise STATUS_CRIT on null values
 * Clean code per flake8 and previous review
 * Add unit_test for nrpe check check_swift_storage.py

Change-Id: Ie076b8ea56f66c1e15a9bfe2e400b49f968fa4ed
Closes-Bug: #1673441
Signed-off-by: Alvaro Uria <alvaro.uria@canonical.com>
2017-08-11 19:24:24 -07:00
Xav Paice 18419a87c5 Add object-rsync-timeout option
Adds the rsync_timeout option to to object-server.conf, so we can adjust
it away from the default of 900s.

If there are a number of large partitions needing replication,
occasionally one needs to adjust the timeout in order to allow the rsync
to complete rather than timeout and retry.

Change-Id: I2d895741cb0528836a675deb6399005a5bf59ab5
Closes-bug: 1702039
2017-07-04 07:57:22 +12:00
Billy Olsen d6061caa2c Only change owner/permissions of new devices
Do not change owner and permissions of already existing
devices in the setup_storage() function as this runs
during every config-changed hook invocation.

Change-Id: I21f23aee34d315ccb4df303527b4d791fc043f58
Closes-Bug: #1676728
2017-06-08 16:03:16 -07:00
Chris MacNaughton 47a4ac6e48 Remove /srv/node from updatedb
This stops updatedb from indexing the storage locations

Change-Id: I6ca6b8667fb06d3b52cedd151531fc0033cf2526
Closes-bug: 1520226
2017-01-25 09:21:31 -05:00
Luong Anh Tuan bca8efdcd5 Replace assertEquals with assertEqual
The method assertEquals has been deprecated since python 2.7.
http://docs.python.org/2/library/unittest.html#deprecated-aliases

Also in Python 3, a deprecated warning is raised when using assertEquals
therefore we should use assertEqual instead.

Change-Id: If7a404da609eb3cae22627f3b9d71292b8865674
Closes-Bug: #1218185
2016-11-22 10:23:25 +07:00
vnathan b268ef82fa Fixed handling duplicate block devices specified in config
Change-Id: I5c96c49d47b762fecc16c8700ef6ed65bcd39bf5
closes-bug: 1582317
2016-11-14 22:13:57 +05:30