Commit Graph

118 Commits

Author SHA1 Message Date
Tim Burke 3f8ead5a29 tests: Fix random "invalid range" errors
random.randint is inclusive on both ends, unlike random.randrange.
As a result, we could previously (rarely) select an end value
*exactly equal* to len(whole_file_bytes), causing an error like

    pyeclib.ec_iface.ECInvalidParameter: pyeclib_c_decode invalid
    range ERROR: Invalid arguments. Please inspect syslog for
    liberasurecode error report.

Change-Id: I842fbbc2eb1f8c4a1d4eb53597ad14cce4eb3dcf
2024-02-29 11:52:06 -08:00
Takashi Kajinami bed3b55540 pep8: Use default line width
We can use the default value by fixing the only one error.

test/test_pyeclib_api.py:111:80: E501 line too long (83 > 79 characters)

Change-Id: Idcdd6f643330b76328b5cd28c32900eeceacafb8
2024-01-26 01:49:00 +09:00
Tim Burke 415efe08f2 Fix gate
Depends-On: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/881890
Change-Id: I9ebf7457e0fc0c98711ba1b093b0f1b8a9f25dff
2023-05-01 08:49:55 -07:00
Tim Burke 2884a6b882 Use hacking checks
Change-Id: I5ed0829d1124c813fb3457cf45152a0f175cebea
2022-04-08 20:29:20 -07:00
jinyuanliu e82d2926e0 Clean up extra spaces
Although these errors are not important, they affect the code
specification.

Change-Id: Ibfe897a406db027ac716fa216a9b8fba5885ec19
2021-09-15 02:05:47 -04:00
Tim Burke 2265056ff0 Fix gate
* Update pike to train
* Fix pep8
* Mark jobs based on legacy jobs non-voting until we can replace them

Change-Id: Ia83ad102bb5c096b01ce73bffb86d44732d2c784
2020-09-22 16:55:54 -07:00
Tim Burke e6bb3e98d5 Make TestBackendsEnabled work on py3
Change-Id: I483488fd53516d44074f18168db6f3445be455b5
2019-02-13 16:35:43 -08:00
Tim Burke 0e3afd3b94 Fix gate
Change-Id: I1c9bd3a4706a1d7235981f15cb6e1f4cc8d7ff55
2019-02-13 09:43:04 -08:00
Zuul 3b7a278123 Merge "Switch from pep8 to flake8 for linting" 2018-03-13 20:15:31 +00:00
Jenkins aa7a2c0cf2 Merge "Clean up ECDriver" 2017-09-13 21:06:46 +00:00
Jenkins c564b69731 Merge "Avoid segfault when raising exceptions" 2017-05-05 23:19:09 +00:00
Jenkins c4312307ab Merge "Fix error message with invalid parity number" 2017-05-03 22:52:25 +00:00
Tim Burke 292856defa Switch from pep8 to flake8 for linting
...and clean up the unused variables, unused imports, and inconsistent
spacing that flake8 revealed.

Change-Id: I3ae1e834f01f477be88dfc48de954eeaf19e43cf
2017-05-03 15:07:20 -07:00
Tim Burke 95c7423142 Avoid segfault when raising exceptions
This came up when running the py3 unit tests after adding a test that
(erroneously) tried to encode native strings. At some point, we should
probably enforce tighter type-checking, but I'm not sure what all types
we need (or want) to support.

Change-Id: Ibb1f4f52207be83f842740d9f3c39c2a03fb1396
2017-05-03 14:32:14 -07:00
Tim Burke f990e62dc0 Clean up ECDriver
- Add a docstring, since I can never remember the **kwargs.
 - Drop the (unused) *args.
 - Raise an error earlier if neither ec_type nor library_import_str
   is provided.

Change-Id: I38b71ede59b6b0e249223792207b0f48438f7b97
2017-05-02 16:07:56 -07:00
Jenkins ea1b479a81 Merge "Add explicit info for immeasurable throughput" 2017-04-26 00:30:51 +00:00
Jim Cheung 69cb73f9d1 Add Phazr.IO libphazr backend support
Currently, there are several implementations of erasure codes that are
available within OpenStack Swift.  Most, if not all, of which are based
on the Reed Solomon coding algorithm.

Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction.  In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.

Please contact support@phazr.io for more info on our technology.

Change-Id: I9377fa32426a190efd0a7f0675ecf13d7e90367d
2017-03-01 19:19:22 -08:00
Kota Tsuyuzaki 5ebb2cd307 Fix error message with invalid parity number
Change-Id: I620df09cf8ddd03e4a4749e9a1d31f8cbe45c752
2017-02-07 23:11:53 -08:00
Kota Tsuyuzaki a60186ace4 Add explicit info for immeasurable throughput
And remove an unnecessary assignment

Change-Id: I81e124ebf761a943e75bf9568085668362b7c50e
2016-12-20 22:56:13 -08:00
Ondřej Nový ee6e409d83 Prevent division by zero in tests
If avg_time is 0, tests will fail:
throughput = (size / 1000.0) / avg_time
ZeroDivisionError: float division by zero

This happens for me on Hurd:
https://buildd.debian.org/status/fetch.php?pkg=python-pyeclib&arch=hurd-i386&ver=1.3.1-1&stamp=1475967332

Change-Id: I0a7812977173b37918f40891f6ec0a2bc7c1d023
2016-12-15 16:37:14 +01:00
Jenkins 32107b63ed Merge "Add tests for the availability of individual backends" 2016-12-08 04:40:34 +00:00
Jenkins b1156356ba Merge "Fix checksum memory leak" 2016-12-07 21:51:15 +00:00
Tim Burke 624f422819 Add tests for the availability of individual backends
These will either pass if the backend is available, or skip if it isn't.
What we definitely *don't* want is a situation where someone runs tests,
sees all passes with no skips, and assumes they must therefore have all
backends available.

Change-Id: I1d2c691d01f38d5e8f7bc9cf4ec4b9c50ace32b9
2016-12-07 18:50:58 +00:00
Tim Burke 288d1e07de Fix checksum memory leak
Since we allocate the result buffer in hex_encode_string, we're
responsible for freeing it later.

Why didn't we catch it before? Turns out, our tests didn't
loop enough. I wasn't reliably seeing a test failure at 200k
iterations, but 400k seems to be enough?

Closes-Bug: #1634006

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: I678e10008c3c5bc04640f7f19498334d94cb0cd6
Change-Id: I0f6e922ba25ad56142f3d095896b9856a436a81c
2016-12-05 22:11:23 -08:00
Kota Tsuyuzaki 22ae03758d ISA-L Cauchy support
This is for supporting ISA-L cauchy based matrix. The difference
from isa_l_rs_vand is only the matrix to use the encode/decode
calculation.

As a known issue, isa_l_rs_vand backend has constraint for the
combinations of the available fragment to be able to
decode/reconstuct. (See related change in detail)

To avoid the constraint, this patch adds another isa-l backend to use
cauchy matrix. The reason I try to add this intead of changing at
sa_l_rs_vand, the isa_l_rs_vand backend may be already used and if we
change the matrix, the current users won't be able to decode the data.

To avoid the problem and keep the backward compatibility, this is in
another isa_l_rs_cauchy namespace.

NOTE: this depends on tag may be meaningless because the depending target
is a c lang project so probably no scripts to install it we have and then,
almost of the tests for isa-l will be skipped due to a lack of isa-l
backend installation.

Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87

Depends-On: I6eb150d9d0c3febf233570fa7729f9f72df2e9be

Change-Id: I3a5516545d17ab7ac67a9a3627c85243e2110764
2016-12-01 17:18:42 -08:00
Clay Gerrard 685bc06180 Cleanup lint errors in test suite
Add some more linting to test modules and clean 'em up

Change-Id: I8bece8e1e2971ef508c058220dd2646ae880fe35
2016-11-29 18:40:53 -08:00
Tim Burke cc6f4bba26 Fix formatted get_metadata memory leak
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>

Change-Id: I678e10008c3c5bc04640f7f19498334d94cb0cd6
2016-11-29 11:57:59 -08:00
Tim Burke 9cc72f7ce8 Assorted combinatoric-test fixes
* Test flat_xor_hd decoding/reconstruction
* Use random input
* Make py3-friendly

Additionally:

* Accept iterables as fragment payloads, rather than just lists
* Add a __repr__ for ECDrivers

Change-Id: Ic5b5e5ef2420afdc318b403fcbea1ff106e16a33
2016-11-17 09:57:35 -08:00
Kota Tsuyuzaki ea46179001 Add greedy test for decode/reconstruct result solid
Once using erasure code scheme in storage system, we have to trust the
backend's result consistency from decode/reconstruct. Otherwise, the
storage system can be broken easily. To trust the result, this patch
adds greedy combination testing for decode/reconstruct for read-solomn
based backends.

Testing:
- decode result from any k fragments must be consistent with original
  data
- The Nth fragment reconstructed from other k fragments must be
  consistent with the encoded Nth fragment

With this patch, we can confirm the consistency, at least, the testing
spec defined at get_pyeclib_testspec() method, and if you want to test
either other parameters or backends, you can add the spec you want to
test.

Change-Id: Ieb46f4ab5c8f6cb60f35c632728cbcf4a1727c53
2016-11-10 05:34:19 -08:00
Kota Tsuyuzaki cbb3d9364f Ref count for dict item should be Py_DECREF
PyDict_SetItems doesn't steal the reference count so that the reference
count of each item in the dict should be decremented via Py_DECREF
*before* returning the dict to Python VM. Otherwise, those items can be
leaked which can never be collected as garbage.

The evicence for memory leak is avaialble for using debug mode python VM like:
https://gist.github.com/bloodeagle40234/f4c0cd267e085cc6224ffdc1b1822631

Plus, this patch add one unit test to check the memory increasement for
many call of get_segment_info.

Closes-Bug: #1604335

Change-Id: I6780efb9718017d296606f3c7e60684910584a1a
2016-09-07 21:40:47 -07:00
Jenkins f53a0d5700 Merge "Fix arg k, m to be required" 2016-09-07 05:41:45 +00:00
Kota Tsuyuzaki 25bc26b914 Fix arg k, m to be required
Right now ECDriver doesn't check if k, m exist in the kwargs for init.
That can tigger unfortunately success to init and that will fail at
encode/decode (or result in odd with get_segment_info)

Once, we have jerasure_rs_vand in default, that worked becuase jerasure
did the assertion for the k, m parameters but we can reproduce the
failures which this patch want fix with like:

driver = ECDriver(ec_type='liberasurecode_rs_vand')   # <- this is
default and it succeeded!
driver.encode(' ')  # <- And then, this will result in ECDriverError:
Out of memory, whooa!

This patch fixes this to check the k, m existence in the init process.

Change-Id: I0757c0a4e510ba42f357db0cac22861919d0ca26
2016-09-06 22:36:16 -07:00
Kota Tsuyuzaki 10802bac59 Add tox/requirements settings to pass gate job
Plus, we need liberasurecode version handling for a few place because
some tests/engine itself is broken with a lack of backword
compatibility.

Closes-Bug: #1586220

Change-Id: I72adaefa10875a73e3e5304eb40fe5d9f6d2598a
2016-08-12 00:05:03 -07:00
Kevin Greenan ccc8c17e3c Fix the tests to handle corrupted fragments. 2016-04-21 06:52:23 -07:00
Tushar Gohad e4c48eec2a (Re)Enable flat_xor_hd file EC tests 2016-01-12 16:07:00 -07:00
Tushar Gohad 42f116e97b Fix flat_xor_hd test invocation 2016-01-12 15:52:13 -07:00
Tushar Gohad 25ddefa6d4 Empty check for pyeclib_drivers list in unit tests 2015-12-03 15:31:17 -07:00
Tushar Gohad 39e7c5f0de Add unit test for VALID_EC_TYPES 2015-11-23 02:18:50 +00:00
Tushar Gohad b14ac48319 Make VALID_EC_TYPES a runtime/dynamic property
... this is needed for compatibility with older versions
liberasurecode where a few EC types/backends were not
available
2015-11-20 07:23:47 +00:00
Tushar Gohad 1c1e3749e6 Make liberasurecode_rs_vand tests optional
... for compatibility with older liberasurecode revs
2015-11-19 18:20:43 +00:00
Tushar Gohad 51737a984f Allow 'flat_xor_hd_4' from external API
Add 'hd' argument back to the internal API
2015-09-08 09:24:32 +00:00
Kota Tsuyuzaki 68e5f3e5ed Add shss test spec to test_pyeclib_api 2015-09-08 11:20:47 +09:00
Kevin Greenan 8c825a8e2a Fix integer truncation issue with PyBuildValue on Big Endian systems by explicitly
casting the size argument passed in to Py_ssize_t.

Also fix import issue with the API test where older versions of Python fail to
import.
2015-08-11 09:32:12 -07:00
Tushar Gohad c0811b748e Remove stray print from test code 2015-07-29 05:08:06 +00:00
Tushar Gohad e4c3f290ed Minor indent fix (tabs -> spaces) 2015-07-28 22:04:34 -07:00
Tushar Gohad 43c0fa9477 Add checksum type support for common testspec 2015-07-29 04:59:41 +00:00
Tushar Gohad 80729cef12 Unify pyeclib_driver construction code, other tweaks
... including reconstruct test tweaks for dynamic
index ranges based on k, m values
2015-07-29 02:52:31 +00:00
Tushar Gohad 545cfe6ef8 Make tests unavailable EC types conditional 2015-07-28 01:13:00 -07:00
Victor Stinner de2a25c5fe Fix Python 3 issues
* Use relative imports
* Fix bytes/unicode issues in tests
* Replace long() with int()
* Replace a/b with a//b to get integer division
* Use a key function to sort list
* Replace zip() with list(zip()) in a test to get a list on Python 3
2015-07-21 14:51:53 +02:00
Tushar Gohad f1524070dd Add liberasurecode_rs_vand test cases 2015-07-19 18:56:50 -07:00