Commit Graph

204 Commits

Author SHA1 Message Date
Zuul cfdfc7b30e Merge "Include stdint.h in where it's used" 2024-01-25 18:28:20 +00:00
Takashi Kajinami 9f4e4fcfc5 Include stdint.h in where it's used
stdint.h was added to use exact-width int types[1] but later these were
all moved to .c file.

[1] 89721c5d17
[2] f81a59b592

Change-Id: I06f83b818da1291669c8ab90ac7e8595f144d8a5
2024-01-26 01:41:50 +09:00
Takashi Kajinami a11e6656de Remove trailing whitespaces
We probably want some format tools (like clang-format) in the future,
but for now all these spaces are deleted by sed.

Change-Id: If7e52e75772551406b27f97854cebdebf2da2abf
2024-01-26 00:17:15 +09:00
Tim Burke 86e9f03c31 Restrict ourselves to Python's Stable ABI
This allows us to build shared libraries that work across minor CPython
releases. See also: https://docs.python.org/3/c-api/stable.html

To build abi3 wheels, run something like

   python setup.py bdist_wheel --py-limited-api=cp35

Change-Id: Iaa747d58c6ac9dd64c5e4d3b5fdd4e56e8e2cb5e
2022-04-06 10:10:16 -07:00
Tim Burke 033fc066f3 Clean up compile warning
Previously, compiling would complain

   warning: comparison of integer expressions of different signedness:
   ‘int’ and ‘uint32_t’

Change-Id: Ic839ab02189103975985fc0557d6846052635b14
2021-10-25 21:49:40 -07:00
Tim Burke a7bf798b10 Use Py_ssize_t when calling PyArg_Parse
This has been available since Python 2.5, and our old int-based approach
stopped working in Python 3.10. From https://bugs.python.org/issue36381:

   Raise warning for # use without PY_SSIZE_T_CLEAN.

   * 3.8: PendingDeprecationWarning
   * 3.9: DeprecationWarning
   * 3.10 (or 4.0): Remove PY_SSIZE_T_CLEAN and use Py_ssize_t always

Change-Id: I5394268aaf4c409a75f795a44c092fd5101178cd
2021-05-10 18:10:44 -07:00
Jenkins 4e0f35a34d Merge "Change the version reference" 2017-06-06 21:05:53 +00:00
Tim Burke 95c7423142 Avoid segfault when raising exceptions
This came up when running the py3 unit tests after adding a test that
(erroneously) tried to encode native strings. At some point, we should
probably enforce tighter type-checking, but I'm not sure what all types
we need (or want) to support.

Change-Id: Ibb1f4f52207be83f842740d9f3c39c2a03fb1396
2017-05-03 14:32:14 -07:00
Jim Cheung 69cb73f9d1 Add Phazr.IO libphazr backend support
Currently, there are several implementations of erasure codes that are
available within OpenStack Swift.  Most, if not all, of which are based
on the Reed Solomon coding algorithm.

Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction.  In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.

Please contact support@phazr.io for more info on our technology.

Change-Id: I9377fa32426a190efd0a7f0675ecf13d7e90367d
2017-03-01 19:19:22 -08:00
Kota Tsuyuzaki 47493a0fc8 Change the version reference
Now, if liberasurecode.so exposes a liberasurecode_get_version function,
we'll use that; otherwise, we'll fall back to reporting the version that
PyECLib was built against.

Note that if your liberasurecode.so doesn't support
the liberasurecode_get_verion function, pyeclib will the version at built
as well as older behavior.

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I54823183cce6775a83e913baf6bb1eeb94aabc13
2017-03-01 20:43:17 +00:00
Jenkins a384e107fe Merge "Use Py_BuildValue instead of manually building dicts" 2016-12-15 04:50:34 +00:00
Jenkins b1156356ba Merge "Fix checksum memory leak" 2016-12-07 21:51:15 +00:00
Tim Burke 288d1e07de Fix checksum memory leak
Since we allocate the result buffer in hex_encode_string, we're
responsible for freeing it later.

Why didn't we catch it before? Turns out, our tests didn't
loop enough. I wasn't reliably seeing a test failure at 200k
iterations, but 400k seems to be enough?

Closes-Bug: #1634006

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: I678e10008c3c5bc04640f7f19498334d94cb0cd6
Change-Id: I0f6e922ba25ad56142f3d095896b9856a436a81c
2016-12-05 22:11:23 -08:00
Kota Tsuyuzaki 22ae03758d ISA-L Cauchy support
This is for supporting ISA-L cauchy based matrix. The difference
from isa_l_rs_vand is only the matrix to use the encode/decode
calculation.

As a known issue, isa_l_rs_vand backend has constraint for the
combinations of the available fragment to be able to
decode/reconstuct. (See related change in detail)

To avoid the constraint, this patch adds another isa-l backend to use
cauchy matrix. The reason I try to add this intead of changing at
sa_l_rs_vand, the isa_l_rs_vand backend may be already used and if we
change the matrix, the current users won't be able to decode the data.

To avoid the problem and keep the backward compatibility, this is in
another isa_l_rs_cauchy namespace.

NOTE: this depends on tag may be meaningless because the depending target
is a c lang project so probably no scripts to install it we have and then,
almost of the tests for isa-l will be skipped due to a lack of isa-l
backend installation.

Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87

Depends-On: I6eb150d9d0c3febf233570fa7729f9f72df2e9be

Change-Id: I3a5516545d17ab7ac67a9a3627c85243e2110764
2016-12-01 17:18:42 -08:00
Tim Burke 9b8d558acc Use Py_BuildValue instead of manually building dicts
Change-Id: I6077b1f82a910e0d939cac7eb0d05bcbcdabdeb7
2016-11-29 20:03:49 +00:00
Tim Burke cc6f4bba26 Fix formatted get_metadata memory leak
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>

Change-Id: I678e10008c3c5bc04640f7f19498334d94cb0cd6
2016-11-29 11:57:59 -08:00
Kota Tsuyuzaki cbb3d9364f Ref count for dict item should be Py_DECREF
PyDict_SetItems doesn't steal the reference count so that the reference
count of each item in the dict should be decremented via Py_DECREF
*before* returning the dict to Python VM. Otherwise, those items can be
leaked which can never be collected as garbage.

The evicence for memory leak is avaialble for using debug mode python VM like:
https://gist.github.com/bloodeagle40234/f4c0cd267e085cc6224ffdc1b1822631

Plus, this patch add one unit test to check the memory increasement for
many call of get_segment_info.

Closes-Bug: #1604335

Change-Id: I6780efb9718017d296606f3c7e60684910584a1a
2016-09-07 21:40:47 -07:00
Kota Tsuyuzaki 1d9cea73cb Correct fragment size handling
Once we have liberasurecode older than depend-on patch, it could make
an invalid mem access if the backend descripter is not available.
(I'm not sure this situation happen in fact, but any other functions in
liberasurecode expecting that failure actually, FWIW)

To recover this failure, this patch fixes to catch the return value from
liberasurecode and if it's an error code, set an error for the return
value to python.

Depends-On: I489f8b5d049610863b5e0b477b6ff70ead245b55
Change-Id: I47d4ec4fa1647c5a4c8d50f0aa6a2df86ca68c80
2016-09-06 22:46:44 -07:00
Kota Tsuyuzaki 10802bac59 Add tox/requirements settings to pass gate job
Plus, we need liberasurecode version handling for a few place because
some tests/engine itself is broken with a lack of backword
compatibility.

Closes-Bug: #1586220

Change-Id: I72adaefa10875a73e3e5304eb40fe5d9f6d2598a
2016-08-12 00:05:03 -07:00
Timur Alperovich 4f278195a4 Use the check_backend_available function.
Eliminates the spurrious syslog messages and is a cleaner mechanism
for querying all of the available backends on a system.
2016-03-15 10:34:49 -07:00
Tushar Gohad 4bff4a92ea Remove integrated liberasurecode tarball
... now that liberasurecode-dev(el) packages are available
on most supported distros
2015-12-04 04:35:59 -07:00
Tushar Gohad 4c8c42d3a2 Clarifying comment about ec_desc return during init
... liberasurecode_instance_create returns an int status code
- one of LIBERASURECODE_ERROR_CODES defined in erasurecode.h
2015-11-23 13:43:00 +00:00
Tushar Gohad 35f81cf63d Return liberasurecode error code from pyeclib_c_init
... earlier we were returning -EINVAL
2015-11-23 00:34:03 +00:00
Tushar Gohad 9b46e826da Disable error reporting during VALID_EC_TYPES eval
... this should not be required after upstream
liberasurecode has more configurable logging
2015-11-23 00:33:11 +00:00
Tushar Gohad d3389998c4 Include liberasurecode_helpers.h for backward compatibility
... for folks trying to build pyeclib with liberasurecode
versions prior to 1.1.0
2015-11-19 17:47:43 +00:00
Tushar Gohad c7057a3f01 Update integrated liberasurecode rev to 1.1.0 2015-10-21 08:32:45 +00:00
Tushar Gohad 92f2824bc2 Use explicit path to erasurecode.h 2015-09-20 20:10:21 +00:00
Tushar Gohad f45db64106 Implement alloc/free fns local to pyeclib c_ext
... stop using helpers internal to liberasurecode
2015-09-18 05:53:00 +00:00
Tushar Gohad 0103f6886f Update integrated liberasurecode rev to 1.0.9 2015-09-08 09:24:32 +00:00
Kevin Greenan 8c825a8e2a Fix integer truncation issue with PyBuildValue on Big Endian systems by explicitly
casting the size argument passed in to Py_ssize_t.

Also fix import issue with the API test where older versions of Python fail to
import.
2015-08-11 09:32:12 -07:00
Kevin Greenan aa6dae1ccd Adding a new tarball for the integrated installer 2015-07-23 00:56:42 -07:00
Tushar Gohad 8d1ce1c4aa Check for missing liberasurecode and install if necessary 2015-07-23 00:22:30 -07:00
Victor Stinner a571779616 Support Python int in decode() for ranges on py2
Before only the Python long type was accepted on Python 2.
2015-07-21 17:11:03 +02:00
Tushar Gohad 04700a4fcf Expose liberasurecode error codes in PyECLib
... also move exception handing down to the C backend layer, so
it is easier to add new error codes.  Addresses #61.
2015-07-19 18:49:34 -07:00
Kevin Greenan 169feb5589 Plugging in new internal RS backend into pyeclib 2015-05-19 10:31:54 -07:00
Tushar Gohad 16cbe3c9f6 Guide users to liberasurecode messages in syslog upon errors 2015-03-13 01:07:03 -07:00
Kevin Greenan f48656c5d7 Wrap calls to pyeclib_c in core.py with try/catch and wrote example
test to validate exceptions.
2015-03-10 22:46:38 -07:00
Tushar Gohad d71d280ed7 Return liberasurecode errors in fixed length string buffers 2015-03-09 14:02:01 -07:00
Tushar Gohad 189cf74381 Improve pyeclib error reporting (expose liberasurecode err returns) 2015-03-08 02:17:38 -07:00
Kota Tsuyuzaki 16b49c501a Use liberasurecode_get_fragment_size
Currently both liberasurecode and PyECLib have fragment size
calculation by dividing alignment data length. However it might cause
size mismatch because of backend specific metadata added to each
fragment. To prevent such a mismatch, we should use
liberasurecode_get_fragment_size which allows to retrieve fragment_size
from liberasurecode directly instead of calculation itself.
2015-03-05 15:54:26 +09:00
Kevin Greenan cf0a5be97c This is part of Kota's pull request #17. 2015-02-07 14:55:59 -08:00
Kevin Greenan 30a74e4483 Add tests for decode metadata checks. 2015-02-07 14:25:10 -08:00
Kevin Greenan bcf856f4da Adding support for checking fragment metadata during decode and plumbing
ragne parameter from python down to the C API.
2015-02-07 11:12:52 -08:00
Kevin Greenan 3b36b99799 Changed the API for check_metadata to return more structured success and error
information.
2015-02-03 21:46:19 -08:00
Kevin Greenan b01b3b5c3d Removing 'fragment_metadata_len' from the check metaadta API, as it is not needed. 2015-01-19 10:32:26 -08:00
Kevin Greenan 464da0311d Changed the return values for the check metadata API. 2015-01-19 10:27:06 -08:00
Kevin Greenan 310956e676 Fixing a few bugs in the original checksum formatting commit and unit
test comit.

Also added a unit test for the foramtted get_metadata.
2015-01-19 10:15:17 -08:00
Kevin Greenan 4913e72f99 Fix the checksum unit tests for crc32. 2015-01-19 10:15:17 -08:00
Kevin Greenan 1c33592d2d Add option to return fragment metadata as a dict, instead of an opaque buffer. 2015-01-19 10:15:17 -08:00
Tushar Gohad 786520d13c Refactor pyeclib python src tree for nosetests
pyeclib needs to be installed in order to run unit tests
which shouldn't be the case.  We should rely on in-source
modules.

This addresses issue#56.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2014-12-10 10:25:01 -07:00