This commit is part of a series to retire the Packaging Deb
project. Step 2 is to remove all content from the project
repos, replacing it with a README notification where to find
ongoing work, and how to recover the repo if needed at some
future point (as in
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project).
Change-Id: Ic0c68767aff92861baade50fe504541d46e3fa83
This came up when running the py3 unit tests after adding a test that
(erroneously) tried to encode native strings. At some point, we should
probably enforce tighter type-checking, but I'm not sure what all types
we need (or want) to support.
Change-Id: Ibb1f4f52207be83f842740d9f3c39c2a03fb1396
Currently, there are several implementations of erasure codes that are
available within OpenStack Swift. Most, if not all, of which are based
on the Reed Solomon coding algorithm.
Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction. In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.
Please contact support@phazr.io for more info on our technology.
Change-Id: I9377fa32426a190efd0a7f0675ecf13d7e90367d
Now, if liberasurecode.so exposes a liberasurecode_get_version function,
we'll use that; otherwise, we'll fall back to reporting the version that
PyECLib was built against.
Note that if your liberasurecode.so doesn't support
the liberasurecode_get_verion function, pyeclib will the version at built
as well as older behavior.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I54823183cce6775a83e913baf6bb1eeb94aabc13
These will either pass if the backend is available, or skip if it isn't.
What we definitely *don't* want is a situation where someone runs tests,
sees all passes with no skips, and assumes they must therefore have all
backends available.
Change-Id: I1d2c691d01f38d5e8f7bc9cf4ec4b9c50ace32b9
Since we allocate the result buffer in hex_encode_string, we're
responsible for freeing it later.
Why didn't we catch it before? Turns out, our tests didn't
loop enough. I wasn't reliably seeing a test failure at 200k
iterations, but 400k seems to be enough?
Closes-Bug: #1634006
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: I678e10008c3c5bc04640f7f19498334d94cb0cd6
Change-Id: I0f6e922ba25ad56142f3d095896b9856a436a81c
This is for supporting ISA-L cauchy based matrix. The difference
from isa_l_rs_vand is only the matrix to use the encode/decode
calculation.
As a known issue, isa_l_rs_vand backend has constraint for the
combinations of the available fragment to be able to
decode/reconstuct. (See related change in detail)
To avoid the constraint, this patch adds another isa-l backend to use
cauchy matrix. The reason I try to add this intead of changing at
sa_l_rs_vand, the isa_l_rs_vand backend may be already used and if we
change the matrix, the current users won't be able to decode the data.
To avoid the problem and keep the backward compatibility, this is in
another isa_l_rs_cauchy namespace.
NOTE: this depends on tag may be meaningless because the depending target
is a c lang project so probably no scripts to install it we have and then,
almost of the tests for isa-l will be skipped due to a lack of isa-l
backend installation.
Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87
Depends-On: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
Change-Id: I3a5516545d17ab7ac67a9a3627c85243e2110764
To apply the fix for a liberasurecode issue [1], we need hard depencency
of liberasurecode requires >=1.3.1. However current binary dependency
maintainance tool "bindep" works only for packagers' repository. (i.e. it
refers the version of apt/yum/etc...) And nothing is cared for the binary
built from source.
This patch provides a way to detect incompatible liberasurecode and
makes a warning log line to syslog which suggest "you're using older
liberasurecode which will be deprecated, please upgrade it".
NOTE:
- This dependency managemnet depends on erasurecode_version.h header
file in liberasurecode. i.e. it cannot care of overwritten .so library
after PyECLib built once.
Partial-Bug: #1639691
1: Icee788a0931fe692fe0de31fabc4ba450e338a87
Change-Id: Ice5e96f0a59096cc9067823f0d62d6c7065ed62f
* Test flat_xor_hd decoding/reconstruction
* Use random input
* Make py3-friendly
Additionally:
* Accept iterables as fragment payloads, rather than just lists
* Add a __repr__ for ECDrivers
Change-Id: Ic5b5e5ef2420afdc318b403fcbea1ff106e16a33
Once using erasure code scheme in storage system, we have to trust the
backend's result consistency from decode/reconstruct. Otherwise, the
storage system can be broken easily. To trust the result, this patch
adds greedy combination testing for decode/reconstruct for read-solomn
based backends.
Testing:
- decode result from any k fragments must be consistent with original
data
- The Nth fragment reconstructed from other k fragments must be
consistent with the encoded Nth fragment
With this patch, we can confirm the consistency, at least, the testing
spec defined at get_pyeclib_testspec() method, and if you want to test
either other parameters or backends, you can add the spec you want to
test.
Change-Id: Ieb46f4ab5c8f6cb60f35c632728cbcf4a1727c53
He is not working at swift/pyeclib anymore and he doesn't want to keep
his name in the README.md which causes contacts/questions to him.
Change-Id: I0d76fa97dbb8c7f65b1dfa7cf3d0f1771893e1ef
PyDict_SetItems doesn't steal the reference count so that the reference
count of each item in the dict should be decremented via Py_DECREF
*before* returning the dict to Python VM. Otherwise, those items can be
leaked which can never be collected as garbage.
The evicence for memory leak is avaialble for using debug mode python VM like:
https://gist.github.com/bloodeagle40234/f4c0cd267e085cc6224ffdc1b1822631
Plus, this patch add one unit test to check the memory increasement for
many call of get_segment_info.
Closes-Bug: #1604335
Change-Id: I6780efb9718017d296606f3c7e60684910584a1a
Once we have liberasurecode older than depend-on patch, it could make
an invalid mem access if the backend descripter is not available.
(I'm not sure this situation happen in fact, but any other functions in
liberasurecode expecting that failure actually, FWIW)
To recover this failure, this patch fixes to catch the return value from
liberasurecode and if it's an error code, set an error for the return
value to python.
Depends-On: I489f8b5d049610863b5e0b477b6ff70ead245b55
Change-Id: I47d4ec4fa1647c5a4c8d50f0aa6a2df86ca68c80
Right now ECDriver doesn't check if k, m exist in the kwargs for init.
That can tigger unfortunately success to init and that will fail at
encode/decode (or result in odd with get_segment_info)
Once, we have jerasure_rs_vand in default, that worked becuase jerasure
did the assertion for the k, m parameters but we can reproduce the
failures which this patch want fix with like:
driver = ECDriver(ec_type='liberasurecode_rs_vand') # <- this is
default and it succeeded!
driver.encode(' ') # <- And then, this will result in ECDriverError:
Out of memory, whooa!
This patch fixes this to check the k, m existence in the init process.
Change-Id: I0757c0a4e510ba42f357db0cac22861919d0ca26
- Add coverage to test-requirements which was causing import error when
running `make test`
- Add "make" to bindep.txt for rpm package
- Add "python-dev/python-devel" to bindep.txt
- Add some instration guide example, how to install those dependency
packages
Change-Id: Ib4a19e734e68ac4d3be7ea13dc72a97b4e209b2b
Plus, we need liberasurecode version handling for a few place because
some tests/engine itself is broken with a lack of backword
compatibility.
Closes-Bug: #1586220
Change-Id: I72adaefa10875a73e3e5304eb40fe5d9f6d2598a
Tony Breeds