Commit Graph

50 Commits

Author SHA1 Message Date
Takashi Kajinami 0f7bdd733a replace validate_legacy
the validate_legacy function is marked for deprecation in
v9.0.0 from puppetlabs-stdlib.

Change-Id: I920294342c9c2c0567796f345cbfa9e39bb1f1d3
2023-09-19 13:33:00 +00:00
Takashi Kajinami 9bd09b15bb Allow defining a section without settings
(account|container|object)-server.conf requires sections for each
services but these sections sometimes don't require settings when
service defaults are used.

This change allows defining only sections using *_config resources.

Depends-on: https://review.opendev.org/849235
Change-Id: I827c06cc994e13b900a9eeaeaec7849a243732e2
2023-05-11 13:53:32 +09:00
Zuul c17989e647 Merge "Remove logic for old swift-ring-builder" 2022-11-29 18:52:11 +00:00
Takashi Kajinami 8ac0db4928 Remove logic for old swift-ring-builder
swift 2.10.0 was released in 2016. We no longer need to support even
older versions. The output format depends on the version currently
installed, thus the latest logic should work as long as the same
release (or even a bit older release) is installed.

This also updates the logic to skip the header lines. The new logic
is considered to be more robust, because it works even when additional
lines are added.

Change-Id: I694ca4c046f490e4ffecf2101f3f147a69045162
2022-11-24 04:23:03 +09:00
Takashi Kajinami 249f466f94 Ensure consistent IPv6 address format
This change ensures the shortened address representations are always
used for ring devices to avoid broken idempotency caused by different
representations (shortened[1] vs canonical[2]).

[1] ::1
[2] 0000:0000:0000:0000:0000:0000:0000:0001

Closes-Bug: #1997313
Change-Id: I762f0780ba25536aa616482a49d59c7db69924d6
2022-11-22 14:44:48 +09:00
Takashi Kajinami efcef86de7 Support customizing internal-client.conf
This change introduces capability to customize internal-client.conf,
which is used by several daemon processes like container-sharder.

Closes-Bug: #1986599
Related-Bug: #1921319
Change-Id: I200e3a4ff323a07778ec6c4d7dd96824c72a12cf
2022-08-16 11:26:43 +09:00
Zuul bcc867acd2 Merge "swiftinit: Remove support for upstrart" 2022-08-01 11:08:21 +00:00
Takashi Kajinami 41b842c7de swiftinit: Remove support for upstrart
... because it was used in Ubuntu < 16.04 which is no longer supported.

Change-Id: Icedace16756158498d5199285ec775095636980b
2022-06-28 22:26:29 +09:00
Zuul ecd42a3bda Merge "Fix typos in parameter descriptions and tests" 2022-06-28 12:41:55 +00:00
Rajesh Tailor 4d937a4a02 Fix typos in parameter descriptions and tests
Change-Id: Ic91d910b973303b0d4db5164fdf9b5fd807a4f70
2022-06-27 13:39:37 +00:00
Rajesh Tailor 9fcfcb2f2a Fix typo in swiftinit service type
Change-Id: I28ac3aacdeca8dc5803fd053e72e80256d82868f
2022-06-27 17:04:54 +05:30
Zuul 58db6b4ba6 Merge "Allow single alias in swift_storage_policy provider" 2022-05-24 10:55:15 +00:00
Simeon Gourlin 19fa1ca37f Allow single alias in swift_storage_policy provider
Regex validation for aliases in swift_storage_policy provider does not
allow single alias value, like for exemple:
  swift::storage::policy { '1':
      policy_name    => '3-Replica-Policy',
      policy_aliases => 'green',
This fixes the accepted pattenr to allow single-value for policy_aliases
parameter.

Also, the whole expression is updated to more strictly validate
the pattern, especially when more than 2 aliases are given.

Change-Id: I62c86097c717d2a74c100386cf20e07976571805
2022-05-15 23:41:05 +00:00
Takashi Kajinami 350778570c Fix missing argument of builder_file_path
The change 63688a14e5 introduced support
of storage policies and since the change was merged builder_file_path
takes one argument which is policy_index .

This change fixes the function call without argument to avoid failure
caused by missing argument.

Closes-Bug: #1949334
Change-Id: I987a37cb0606b6c217f906fbae17a765e40f4580
2021-11-01 16:44:12 +09:00
Thomas Goirand 4d92952345 Add a swift_drive_audit_config puppet type
This patch adds what's necessary to configure drive-audit.conf

Change-Id: I9bee191412ead21f8fd765b8839b6d63f628bb9c
2021-05-19 21:21:53 +09:00
Thomas Goirand 51f4ad1cb3 Add support for swift_{proxy,account,container}_uwsgi_config in Debian
This patch is adding the configuration of the number of workers,
and the size of the listen queue in Debian, which uses uwsgi to run
Swift. Therefore, this patch adds new swift_proxy_uwsgi_config,
swift_account_uwsgi_config, and swift_container_uwsgi_config, providers
as well as e new swift::wsgi::uwsgi_proxy, swift::wsgi::uwsgi_container,
and swift::wsgi::uwsgi_account classes.

Change-Id: Ib18f3479bc6a4513a7b3d823cda480c4ace2f2c0
2021-04-09 23:50:00 +02:00
Zuul 0252254230 Merge "Fix incorrect comments about updating region/zone of a device" 2020-05-12 09:38:49 +00:00
Takashi Kajinami 26c8746d7b Fix incorrect comments about updating region/zone of a device
Updating region/zone of a device is actually implemented and supported
in swift, while puppet-swift doesn't support that operation yet.
This patch updates error messages when an operator tries to update
region/zone of a device, so that they notice more clearly that the
cause exists in puppet-swift, not in swift.

Change-Id: I7c7f633630ac52578a0cf6beb2033465a16bd251
2020-05-10 00:40:14 +09:00
Takashi Kajinami 115be330eb Use anchor to require necessary packages
... so that correct packages are required without re-defining them in
resource implementations.

Change-Id: I5be94c01b9ec1bbfdae37b3a7e282f1b457fe3e0
2020-05-04 12:00:45 +09:00
Rocky 0a6ac4addc Support to configure memcache.conf
Change-Id: Iab2520403edd47004c3edbeda66b4eb16eed587a
2020-04-24 06:54:17 +10:00
Thiago da Silva c85beff334 Adding barbican configuration to swift
Adding barbican configuration to Swift will
allow enabling data-at-rest encryption in Swift.

Change-Id: I78c6003f5f599a422193dc47422ee607ce05c715
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2018-02-07 12:03:19 -05:00
zhangyanxian 2628d0fae8 Fix some typos in swift_storage_policy.rb
TrivialFix

Change-Id: Idf396283dc0d1c0c9dfd28c702bba826cffc2414
2017-03-17 02:55:40 +00:00
Marcus Furlong d4922bee39 memcache_servers support in object-expirer and container-reconciler
This commit adds support for the memcache_servers option in
object-expirer and container-reconciler. Without this, both
currently default to 127.0.0.1:11211.

Closes-Bug: #1627927
Change-Id: Ie139f018c4a742b014dd4d682970e154d66a8c6d
2016-12-05 15:30:35 +11:00
Adam Vinsh 63688a14e5 Add support for swift storage policies
This change implements storage policies as defined by swift:
http://docs.openstack.org/developer/swift/overview_policies.html

There are two primary areas of change in this review
- Add storage policy support to the ringbuilder and ring devices, along with
  associated spec tests.
- Adding storage policy support and enforcing rules in swift.conf using the
  swift_storage_policy type and provider. Also: updated spec tests and an
  update to swift acceptance test to use storage policies to configure an
  additional 3 replica based ring.

See release notes and the README update for more details/instructions.

Change-Id: I2b8db751790704df3f1027a14f61e231591537f3
2016-11-22 16:17:52 +00:00
Emilien Macchi dcc8ca54ee Add support for Swift Ring Builder 2.9.1+
This patch: https://review.openstack.org/#/c/319387
modified the output of rings, so we need to add the case where this
regex is found and how to use it.

Change-Id: Id9f1a478670e7b618ce9d222c5bc9739e3dd6286
2016-08-11 21:57:41 -04:00
Adam Vinsh d239e198a9 swiftinit provider write file directly
Do not use the puppet file resource from within
the swiftinit service provider.  In puppet versions
4.4.0+ it has unexpected results.  Instead use ruby
File to write the init/service files directly.

Change-Id: If0dc56e0742e591e3db3ed18154edf3489d5ef1a
2016-06-14 17:07:26 -04:00
Denis Egorenko ec4db849c8 Allow to configure container_sync middleware
Add possibility to configure container_sync middleware. Also
allow to configure swift container-sync-realms, which specifying
the allowable clusters and their information, which is required
for container_sync.

Change-Id: I183841cca3daf72a30db160b88c9e3a9335ad7dc
2016-05-30 16:03:44 +03:00
Adam Vinsh 5323bf83ac Swift-ring-builder skip line 'Ring up-to-date..'
If a ring is balanced or does not need an update, swift-ring-builder
will print an additional line:
"Ring file /etc/swift/container.ring.gz is up-to-date"
Skip this line in parsing the output to avoid "Warning: Unexpected line:"

Change-Id: Ibc021f62f699c94f36c76b11acefe1000f6a2288
2016-05-17 15:59:07 +00:00
Adam Vinsh d705401750 Support xenial with swift-init service provider
Added function to check if ubuntu is < 16.04.
If ubuntu < 16.04 then default provider was upstart.
If ubuntu = 16.04 then use systemd support.

Closes-Bug: #1578408

Change-Id: Icb2d6ff5152e782066b957d5e6e74357496c6b28
2016-05-16 10:09:43 -04:00
Jenkins dae0934002 Merge "Add support to change meta entry for ring devices" 2016-01-07 04:35:50 +00:00
Adam Vinsh f3c1832bbb swiftinit provider check init/systemd file content
Compare contents of the init/systemd file on disk against
swiftinit provider template contents.  If they don't match
then the distro provided init/systemd file has not been replaced.
swiftinit provider will call service enable and write out correct
init/systemd file.

Change-Id: I722f6cfa8c537d62a607880cc37863f2d1139636
2015-12-17 20:15:09 -05:00
Christian Schwede 67c0f6ed22 Add support to change meta entry for ring devices
Change-Id: If425694f49b8604db8ac88e21ee54075478cae3c
2015-12-16 22:56:56 +01:00
Adam Vinsh 1dfe3d29a9 swift-ring-builder 2.2.2+ output parse fix
Starting in swift 2.2.2 an extra line for overload is printed
by swift-ring-builder.
Check for this and skip that line to avoid avoid warning messages.

Change-Id: I2d81db0a29249101116c1899565f024aaac69000
Closes-Bug: #1524936
2015-12-10 16:57:00 -05:00
Adam Vinsh 5a7d18975f Manage swift with swiftinit service provider
The 'swiftinit' provider is a custom provider of the service type
that manages swift services using swift-init.

Use of the swiftinit service provider is optional, the default is
to use service providers specified in params file. This provider
also manages swift services starting at boot by adding or removing
a templated init or services file and making systemctl calls.
See README for more detail.

A wrapper defined type 'swift::service' has been created to simplify
use of the swiftinit provider without adding logic to every class.
this wrapper also aids in input validation and testing of the
swiftinit provider.

Two extra runs of apply_manifest have been added to the
basic_swift_spec acceptance test. The service_provider
is set to "swiftinit".  The first run catches any errors upgrading
to the swiftinit service provider and the second run tests
idempotency.

This patch is an initial step towards using swift-init to manage
multiple swift services out of different configuration files such
as is needed to run a separate replication network.

Change-Id: I2f71c82c7a6c463f8c76a193409c0a17daa15bda
2015-12-07 16:00:50 -05:00
Cody Herriges dbfa88f684 Make sure Facter is only executed on agent
This commit moves conditionals based on values obtained from Facter
  inside code blocks that are only executed on agents and replaces
  the use of Facter['osfamily'].value with Facter.value:(:osfamily).

  Without this change we are returning values based off the master's
  facts, so if a Debian agent checks into a RedHat master then the type
  is currently making a decision based off of RedHat and not the actual
  OS of the agent running the catalog.  Code ran outside blocks is
  evaluated on masters and inside a block is evaluated by the agent when
  the catalog is executed.  We do not notice this because all our
  testing uses "puppet apply" and autorequire only matters when they
  find a matching resource name in the catalog.  The latter results
  in no error because the relationship is simply ignored on the Debian
  agent if a package resource with the RedHat name is not found.

  This issue was found by running facter 3, the C++11 re-implementation
  under the jruby and clojure implemented puppetserver.  The clojure
  jni shim written so that facter can be ran inside of puppetserver does
  not implement the [] method for the Facter module but the ruby shim
  does.  So I noticed that the code had an issue when
  Facter['osfamily'].value was executed on the master, which returned a
  not method error.  Once again, something we didn't notice becuase we
  do not test against a master.  I decided to migrate to
  Facter.value(:osfamily) to simply keep to an API that is consistent
  across both puppet and puppetserver but fixing where the code runs
  does actually solve the not method found error.

Change-Id: I24b0b20b2839c7bc33a76ac14849783f2285579f
2015-11-12 15:09:58 -08:00
Javier Pena d8a8b70547 Add support for swift-object-expirer service
The swift-object-expirer service was not supported by
puppet-swift. Adding support by creating a new class
(swift::objectexpirer), and its associated custom type/provider.

Change-Id: I498ffe525a7316c0091e4c9d8b7d9658234231f6
2015-09-30 10:09:09 +02:00
Yanis Guenane e6e8c9d836 Reflect provider change in puppet-openstacklib
With the creation of the new openstack_config provider, some processing
that was done in swift_config has been centralized in
openstack_config.

Impacted methods are :

  * section
  * setting
  * separator

Also, this commit adds the fact that, when passing a specific string
(ensure_absent_val) the provider will behave as if ensure => absent was
specified. '<SERVICE DEFAULT>' is the default value for
ensure_absent_val.

The use case is the following :

swift_config { 'DEFAULT/foo' : value => 'bar' } # will work as usual

swift_config { 'DEFAULT/foo' : value => '<SERVICE DEFAULT>' } # will mean absent

That means that all the current :

if $myvar {
  swift_config { 'DEFAULT/foo' : value => $myvar }
} else {
  swift_config { 'DEFAULT/foo' : ensure => absent }
}

can be removed in favor of :

swift_config { 'DEFAULT/foo' : value => $myvar }

If for any reason '<SERVICE DEFAULT>' turns out to be a valid value for
a specific parameter. One could by pass that doing the following :

swift_config { 'DEFAULT/foo' : value => '<SERVICE DEFAULT>',
ensure_absent_val => 'foo' }

Change-Id: I9281d2cae81f9799327f7f6e04498d6bc723f233
Depends-On: I0eeebde3aac2662cc7e69bfad7f8d2481463a218
2015-08-19 12:00:34 +02:00
Yanis Guenane a72e27f83c Rely on autorequire for config resource ordering
Currently we specify the ordering of config resources wherever it is
necessary based on the presence of the file it will write to, or the
presence of the package in charge of providing the file it will write
to.

Those kind of ordering can be specified directly at the resource level
using the autorequire mechanism. With this patch, any config resource
will make sure the package in charge of providing the file will be
installed first.

Change-Id: Icb3464cc0a747d40326d610d38806d059c9a0fc3
2015-08-13 09:54:06 +02:00
Lukas Bezdicka b6ba9f09b2 Fix ipv6 support
One shold never parse uri with split(':') as it was in types
ring_account_device, ring_container_device and ring_object_device.
This is fixed by using uri and just setting variables from that.
Also swift_ring_builder didn't add brackets to ipv6 address before
adding port information to it.

Change-Id: I55b7032143c8446abeed4f6c266c0fff0cb6b3b3
2015-04-07 12:54:14 +02:00
David Moreau Simard bb3007615f Add swift-ring-builder multi region support
Since Swift 1.8.0, there is the region layer of topology above zones.
swift-ring-builder supports this natively and this commit allows a user to
specify a region when creating devices.
We maintain backwards compatibility by defaulting to region '1'.
The coverage of the ring builder spec tests were also improved as part of
this commit.

Change-Id: I67cbe6b87c84778f71df59cf00f2c5175342bc1b
2014-05-09 15:45:28 -04:00
Dan Prince c89e327fb8 Use swift_config to configure swift.conf
This change also adds an autorequire in the swift_config
type to ensure the /etc/swift/swift.conf exists.

Change-Id: Ic591a3ceb040c4599339ed7f750a5273d2052ac1
Implements: blueprint puppet-swift-ini-settings
2013-08-26 20:54:06 -04:00
Mathieu Gagné e41a552afe Add ini_setting based custom types for configs
Previous configuration file management method used templates.
This method had limited flexibility and was cumbersome
to use and maintain between upstream versions.

This change introduces new custom types based on ini_setting:
  * swift_account_config: /etc/swift/account-server.conf
  * swift_bench_config: /etc/swift/swift-bench.conf
  * swift_config: /etc/swift/swift.conf
  * swift_container_config: /etc/swift/container-server.conf
  * swift_dispersion_config: /etc/swift/dispersion.conf
  * swift_object_config: /etc/swift/object-server.conf
  * swift_proxy_config: /etc/swift/proxy-server.conf

The use of ini_setting offers multiple advantages:
  * Ability to individually manage configuration settings without
    the need to modify templates. This allows the user to manage or
    override settings that aren't handled by the module.
  * Use upstream default values if none is provided.
  * Remove the need to maintain and synchronize templates
    to match upstream versions. Such maintenance could become
    problematic and complicated if multiple upstream versions
    are supported but have different base configuration files.

Implements: blueprint puppet-swift-ini-settings
Change-Id: Iba7be64581b81b8ab4f33cf7096df37e64ccc4b4
2013-07-25 10:19:30 -04:00
Dan Prince ee4a9d4859 Update swift_ring_builder to support replicator...
Updates swift_ring_builder rebalancing parsing to support the
latest upstream replicator_id and replicator_port changes.

This correspondes with swift 1.8+ upstream development.

Fixes LP Bug #1182197.

Change-Id: Ibbaf75ca22cad95b22f9b5f03b547db8c47a9ea4
2013-05-20 15:46:28 -04:00
Mehdi Abaakouk a9ec8e1c2d Fix swift-ring-builder for swift 1.8
In some case, there are no spaces before - of the balance in swift 1.8.
The change fix this.

/etc/swift/object.builder, build version 9
32768 partitions, 3.000000 replicas, 1 regions, 3 zones, 9 devices, 200.00 balance
The minimum number of hours before a partition can be reassigned is 24
Devices:    id  region  zone      ip address  port      name weight partitions balance meta
             0     1     1      10.68.0.45  6000       sdd 100.00      32768 200.00
             1     1     2      10.68.0.46  6000       sdd 100.00          0-100.00
...

Change-Id: Ic7757a22ce94e4d51ae11780d4626c90a9ed430f
2013-05-02 14:17:26 +02:00
Dan Prince bee081b5b6 Fix swift_ring_builder output parsing.
Updates the swift_ring_builder so that it parses ring builder output
from the most recent Swift releases correctly (1.7.6 or greated).

Backward compat for the previous parsing is still supported.

Change-Id: I4263b349310e16f6825960805eb27e4bfe3ff002
2013-04-11 11:37:38 -04:00
Dan Bode 09f726c363 Allow multiple devices per endpoint.
Previously, the Puppet modules only supported a single device per
endpoint.

This commits allows multiple devices per endpoint by making the device
name a part of the namevar for the swift ring devices.

it also updates the relevent configs.
2012-06-05 16:45:47 -07:00
Dan Bode c9389516ba Ensure swift hash is rebuilt for every puppet run
Previously, the swift caches were being built on the class
instances themselves.

This could result in stale values when puppet runs multiple times
using the same process (like in the case of puppet agent)

This commit uses prefetch to ensure that the retrieved swift ring
values are updated per puppet run.
2012-04-13 13:41:12 -07:00
Dan f93ba72fbb Munge weights to be in the format %.2f
The swift ring db converts integers into floats with 2 decimal
points percisions. This was causing the ring resources to constantly
resync.

This commit munges weight in the type to the proper format.
2012-03-14 05:41:41 +00:00
Dan Bode b6b13ce1b2 Handle case where ring_builder has no rows
The code was not robust when no
rows were returned from
  swift_ringbuilder(builder_file_path).

The calls to retrieve an array range from the
array of this commands output was returning nil
which caused the failure: undef method each on nil

This change explicitly checks for the case where
there are no rows returned by swift-ring-builder.
2012-01-23 11:44:54 -08:00
Dan Bode 917426380e Implement ringbuilder
This commit adds a class, defined resources and
native types that can be used together to manage
the creation, configuration, and rebalancing of
rings.

Creates the following native types:

  ring_account_device
  ring_container_device
  ring_object_device

The implementation of these types is provided
by swift-ring-builder and most of the functionality
is provided by the parent provider (SwiftRingBuilder)

Each of the providers is hard-coded to find the
relevent ring database in /etc/swift. This directory
location can currently not be configured.

Each provider implements self.instances so the
current state can of the ring can be queried from
puppet.

These providers do not handle either the creation
or the rebalancing of the rings, they are intended
to be used in combination with defined resource types
that perform that functionality:

  swift::ringbuilder::create
  swift::ringbuilder::rebalance

The swift::ringbuilder class is also created as
a part of this commit. This class does the following:

  - creates object, account, and container rings
(it uses the same parameters to create these rings)
  - creates the rebalance defines
  - sets up the following relationships:
    for each ring type, create should be applied before
    the native types which should refresh the rebalance
2012-01-23 11:42:42 -08:00