The repo is Python using both Python 2 and 3 now, so update hacking to
version 2.0 which supports Python 2 and 3. Note that latest hacking
release 3.0 only supports version 3.
Fix problems found.
Remove hacking and friends from lower-constraints, they are not needed
for installation.
Change-Id: I9bd913ee1b32ba1566c420973723296766d1812f
I can't imagine us *not* having a py3 proxy server at some point, and
that proxy server is going to need a ring.
While we're at it (and since they were so close anyway), port
* cli/ringbuilder.py and
* common/linkat.py
* common/daemon.py
Change-Id: Iec8d97e0ce925614a86b516c4c6ed82809d0ba9b
Repeated calls to each co-builder's _update_last_part_moves() are
unnecessary and have the unfortunate side effect of resetting the
_last_part_moved bitmap. When a component builder has zero
min_part_hours this results in it not preventing its co-builders from
moving parts that it has already moved.
This patch changes the CompositeRingBuilder to call each component
builder _update_last_part_moves() *once* before rebalancing any
component builder. CooperativeRingBuilder's no longer forward calls to
their _update_last_part_moves() method. Each component's
_last_part_moved bitmap is therefore preserved until for the duration
of the composite rebalance.
The initialisation of the RingBuilder _last_part_moves array is moved
to the RingBuilder __init__ method, so that calls to
_update_last_part_moves() are effective even when rebalance() has
never been called on that builder. Otherwise, during a composite
rebalance, a component that has not previously been rebalanced will
not have its _last_part_moves_epoch updated during rebalance and as a
result may report erroneous min_part_seconds_left after its first
rebalance.
Related-Change: I1b30cb3d776be441346a4131007d2487a5440a81
Closes-Bug: #1714274
Change-Id: Ib165cf974c865d47c2d9e8f7b3641971d2e9f404
... but only if we *have* to!
During the initial gather for balance we prefer to avoid replicas on
over-weight devices that are already under-represented in any of it's
tiers (i.e. if a zone has to have at least one, but may have as many of
two, don't take the only replica). Instead we hope by going for
replicas on over-weight devices that are at the limits of their
dispersion we might have a better than even chance we find a better
place for them during placement!
This normally works on out - and especially so for rings which can
disperse and balance. But for existing rings where we'd have to
sacrifice dispersion to improve balance the existing optimistic gather
will end up refusing to trade dispersion for balance - and instead get
stuck without solving either!
You should always be able to solve for *either* dispersion or balance.
But if you can't solve *both* - we bail out on our optimistic gather
much more quickly and instead just focus on improving balance. With
this change, the ring can get into balanced (and un-dispersed) states
much more quickly!
Change-Id: I17ac627f94f64211afaccad15596a9fcab2fada2
Related-Change-Id: Ie6e2d116b65938edac29efa6171e2470bb3e8e12
Closes-Bug: 1699636
Closes-Bug: 1701472
This is follow up for the related change [1].
Just adding small docs and removing unnecessary arg in the test.
[1] Related-Change: I1b30cb3d776be441346a4131007d2487a5440a81
Change-Id: I48c21b6028bd2ef454e084032a2bd8c0d4239e4b
- Add a CooperativeRingBuilder subclass of RingBuilder. The subclass takes
a reference to a parent CompositeRingBuilder which is consulted about
whether a part can be moved during rebalance. The parent builder in turn
consults all component CooperativeRingBuilder's to decide if a part can
be moved.
- Make CompositeRingBuilder load CooperativeRingBuilder instances.
- Add rebalance() method to CompositeRingBuilder class.
- Add a load_components() method to CompositeRingBuilder class.
- Change the CompositeRingBuilder compose() method to NOT by default
raise a ValueError if component builders have not been modified since
last loaded. With the load_components method being added it makes
less sense insist by default on loaded components being modified, and
it is desirable to have the same semantic for all methods that load
components. Previously it has been necessary to use the 'force' flag
with compose() to prevent these errors being raised, which has the
unfortunate side effect of also disabling all other checks on
component builders. A new 'require_modified' parameter is added to
compose() which defaults to False but can be set to True if the
previous default behaviour is required.
Change-Id: I1b30cb3d776be441346a4131007d2487a5440a81
This is a follow up for https://review.openstack.org/#/c/441921
all of this patch is for minor fixes, I found in my self-review.
Change-Id: Ib3a1dc983c3da69dea592114e25a5047ec91a2b9
* Adds a composite_builder module which provides the functionality to
build a composite ring from a number of component ring builders.
* Add id to RingBuilder to differentiate rings in composite.
A RingBuilder now gets a UUID when it is saved to file if
it does not already have one. A RingBuilder loaded from
file does NOT get a UUID assigned unless it was previously persisted in
the file. This forces users to explicitly assign an id to
existing ring builders by saving the state back to file.
The UUID is included in first line of the output from:
swift-ring-builder <builder-file>
Background:
This is another implementation for Composite Ring [1]
to enable better dispersion for global erasure coded cluster.
The most significant difference from the related-change [1] is that this
solution attempts to solve the problem as an offline tool rather than
dynamic compositing on the running servers. Due to the change, we gain
advantages such as:
- Less code and being simple
- No complex state validation on the running server
- Easy deployments with an offline tool
This patch does not provide a command line utility for managing
composite rings. The interface for such a tool is still under
discussion; this patch provides the enabling functionality first.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
[1] Related-Change: I80ef36d3ac4d4b7c97a1d034b7fc8e0dc2214d16
Change-Id: I0d8928b55020592f8e75321d1f7678688301d797
Andreas Jaeger