We probably want some format tools (like clang-format) in the future,
but for now all these spaces are deleted by sed.
Change-Id: Idcc1cbfa39aecf0d5109488defd89b8a60b786da
Apparently the author of our old crc32 assumed that shifting an int
to the right sign-extends, which is not always the case. Result is,
building and running make test on s390x fails. The fix is to force
a sign-extension using the "xor 0x80; sub 0x80" trick.
N.B. This does not cause a compatibility problem, because by a
miracle the "broken" crc32_alt was actually computing a stock
crc32, same that zlib has. Therefore, if someone, somewhere,
ran a Swift cluster on s390x with erasure coding policy,
the data in it is already stored with zlib checksums, as we
do it now anyway. This fix only permits the tests pass, which
used the bad data sample from x86.
Change-Id: Ibd5e4e6c02be00540a9648cc7e0f8efda275bf3f
Related-Change: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5
Related-Bug: 1666320
This popped up because Fedora mandates warning-free builds,
and get_chksum triggers a warning because it returns an
unaligned pointer (it is so analigned, static analysis in
the compiler can detect it).
The easiest fix is to remove it altogether. We think it should
be safe, because:
- The function is not listed in any headers
- Its counterpart is called "set_checksum", not "set_chksum"
- PyECLib does not use it
We also hush some doxygen warnings about wrong comments.
Change-Id: Ie5bc736f912706e0ffd507765abd24e7f4761233
... and vice-versa. We'll fix up frag header values for our output
parameter from liberasurecode_get_fragment_metadata but otherwise
avoid manipulating the in-memory fragment much.
Change-Id: Idd6833bdea60e27c9a0148ee28b4a2c1070be148
Each was only really used in one place, they had some strange return types,
and recent versions of clang on OS X would refuse to compile with
erasurecode_helpers.c:531:26: error: taking address of packed member 'metadata_chksum' of
class or structure 'fragment_header_s' may result in an unaligned pointer value
[-Werror,-Waddress-of-packed-member]
return (uint32_t *) &header->metadata_chksum;
^~~~~~~~~~~~~~~~~~~~~~~
We don't really *care* about the pointer; we just want the value!
Change-Id: I8a5e42312948a75f5dd8b23b6f5ccfa7bd22eb1d
Previously, we had our own CRC that was almost but not quite like
zlib's implementation. However,
* it hasn't been subjected to the same rigor with regard to error-detection
properties and
* it may not even get used, depending upon whether zlib happens to get
loaded before or after liberasurecode.
Now, we'll use zlib's CRC-32 when writing new frags, while still
tolerating frags that were created with the old implementation.
Change-Id: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5
Closes-Bug: 1666320
The well-known idiom to compute a required number of data blocks
of size B to contain data of length d is:
(d + (B-1))/B
The code we use, with ceill(), computes the same value, but does
it in an unorthodox way. This makes a reviewer to doubt himself
and even run tests to make sure we're really computing the
obvious thing.
Apropos the reviewer confusion, the code in Phazr.IO looks weird.
It uses (word_size - hamming_distance) to compute the necessary
number of blocks... but then returns the amount of memory needed
to store blocks of a different size (word_size). We left all of it
alone and return exactly the same values that the old computation
returned.
All these computations were the only thing in the code that used
-lm, so drop that too.
Coincidentially, this patch solves the crash of distro-built
packages of liberasurecode (see Red Hat bug #1454543). But it's
a side effect. Expect a proper patch soon.
Change-Id: Ib297f6df304abf5ca8c27d3392b1107a525e0be0
Currently, there are several implementations of erasure codes that are
available within OpenStack Swift. Most, if not all, of which are based
on the Reed Solomon coding algorithm.
Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction. In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.
Please contact support@phazr.io for more info on our technology.
Change-Id: I4e40d02a8951e38409ad3c604c5dd6f050fa7ea0
Can you believe that we ware testing the memory leak with valgrind
to just *bash scripts* instead of actual binaries on liberasurecode_test
and libec_slap?
That is why we cannot find such an easy memory leak[1] at the gate.
Now this patch enable to run the valgrind against to the binaries.
With this fix, we found various memory leak at liberasurecode_test as
follows and this patch also fixes them:
- If we create fake fragments, we're responsible for freeing all of the
frags as well as the array holding the pointers to the frags.
- If we allocate any space, we're responsible for freeing it.
- If we create an EC descriptor, we're responsible for destroying it.
- If we create a fragment or skip array, we're responsible for freeing it.
- If that happens inside a loop, we're responsible for doing it *inside
that same loop*.
In addition to the test fix, this patch fixes following memory leaks at
the code which is affected to other users (pyeclib, OpenStack Swift)
* Refuse to decode fragments that aren't even long enough to include
fragment headers.
* Fix a small memory leak in the builtin rs_vand implementation.
Closes-Bug: #1665242
Co-Authored-By: Tim Burke <tim@swiftstack.com>
1: https://review.openstack.org/#/c/431812
Change-Id: I96f124e4e536bbd7544208acc084de1cda5c19b2
This is for supporting ISA-L cauchy based matrix. The difference
from isa_l_rs_vand is only the matrix to use the encode/decode calculation.
As a known issue, isa_l_rs_vand backend has constraint for the
combinations of the available fragment to be able to decode/reconstuct.
(See related change in detail)
To avoid the constraint, this patch adds another isa-l backend to use
cauchy matrix and keep the backward compatibility, this is in
another isa_l_rs_cauchy namespace.
For implementation consieration, the code is almost same except the matrix
generation fucntion so that this patch makes isa_l_common.c file for
gathering common fucntions like init/encode/decode/reconstruct. And then the
common init funciton takes an extra args "gen_matrix_func_name" for entry
point to load the fucntion by dlsym from isa-l .so file.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87
Change-Id: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
Currently, we have liberasurecode version info in the header and pyeclib
is using the info to detect the version. However it's a bit painful
because it requires to rebuild pyeclib c code for you to see the actual
installed version.
This addition for liberasurecode_get_version enables caller to get the
version integer from compiled shared library file (.so) and it will
rescure to re-compiled operation from pyeclib.
Change-Id: I8161ea7da3b069e83c93e11cb41ce12fa60c6f32
Since the commit a01b1818c8,
we have been to able to test some parameters for each tests.
However, instead, the NULL which means the end of the test parameters
causes skipping a bunch of unit tests which doesn't take args (e.g.
test_create_backend_invalid_args). That is the worse because we have not
tested anymore for the tests since the commit.
This patch fixes to make it tested and more, fix a minor bug for a
case that corrupted header incomming to get_metadata.
Closes-Bug: #1634403
Change-Id: Ib99a8aa6032f02d0c7d1ab94b8da1ebfd9047d74
As like current code, if we do go jump to just "out" section when
memory allocation failed, liberaurecode/PyECLib will report it as
success. However, in fact, the returned binary is not the one
reconstructed. This patch fixes liberasurecode_reconstruct to return
error code (ENOMEM) if either malloc or memset for working space is
failed.
Change-Id: I7bfc481c7a9bbc64288760df2dc6053c57657b41
As well as any other callers, libersurecode_get_fragment_size should
handle the return value of liberasurecode_get_backend_instance_by_desc.
Otherwise, get_by_desc can return NULL and it causes an invalid memory
access in librerasurecode_get_fragment_size.
Change-Id: I489f8b5d049610863b5e0b477b6ff70ead245b55
Removed unnecessary check that metadata index was not negative.
Check is unneeded because type is unsigned and as such can not be
negative.
Change-Id: I72eddf78b25b3ff9bedbd596095f04be855c3817
Closes-Bug: 1587954
Users of liberasurecode <= 1.0.7 used alloc/free helpers
(which they shouldn't have). This change is to make sure
we are still able to those older revs of programs and they
work with newer liberasurecode.
https://bitbucket.org/tsg-/liberasurecode/issue/13/decode-fails-for-many-cases-when-m-k
This fix includes:
1.) Proper buffer allocation for the 'missing_idxs' structure, which was not allocating enough
space when k > m.
2.) Checks to use header fields of parity fragments during decode when *no* data fragments
are available.
3.) Fixed the unit tests to properly handle the case where k <= m.
4.) Extended the unit test framework to support multiple tests per backend
5.) Added tests for all RS implementations: (4,8), (4,4), (10,10)
On the current code, get_fragment_partition might touch the
invlid memory area with minus index (that means a invalid header)
and it causes segmentation fault.
This fixes it to handle the minus index as a EBADHEADER and then
no segmentaition fault appeared on the case.
shss always needs to decode but fragments_to_string
will alloc internal_payload as a decoded data. It causes
duplicated memory allocation and memory leak.
For get_segment_info function of PyECLib, liberasurecode should
support get_fragment_size function because if pyeclib and liberasurecode
have the calculation of fragment size each other, it might cause
the size mismatch (i.e. it'll be a bug) in the future development work.
This patch introduces liberasurecode_get_fragment_size function to return
the fragment_size calculated at liberasurecode accoring to specified
backend descriptor.
It really usefull to help caller knows how large size it have to expect
and all pyeclib has to do for retrieving fragment_size will be just calling
the liberasurecode_get_fragment_size function on get_segment_info.
This patch allows to get correct fragment size includes metadata_adder.
Current implementaion automatically allocates extra bytes for the metadata_adder
in alloc_buffer, and then, no information about the extra bytes will be returned
to the api caller side. It's too confusable because callers couldn't know how size they
assumes as the fragment size.
To be easy to find out the size infomation, this patch adds "frag_adder_size"
variable into fragment metadata and also some functions to get fragment size.
The definitions of these size infomation are here,
fragment_meta:
- size-> raw data size used to encode/fragment_to_string
- frag_adder_size-> metadata_adder of backend specification
And the definitions of functions are here,
- get_fragment_size:
-> return sizeof(fragument_header) + size + frag_adder_size
- get_fragment_buffer_size:
-> return size + frag_adder_size
- get_fragment_payload_size:
-> return size
By using these function above, users could get the size information
directly from fragments. It results in enabling to return fragment_len
to the caller side easily.
On the first consideration[1], metadata_adder is defined as a extra byte
size for "each" fragment. However, current implementation is an element
affects to data_len. (i.e. aligned_data_size for original segment data)
We should make metadata_adder to be a fixed value against to each fragment,
otherwise the extra bytes for the fragments will have flexible length depends
on "K". It will be quite complex for backend implementation to know "How large
bytes the raw data size is", "How large bytes the backend could use as extra
bytes for each fragment".
1: 032b57d9b1?at=master
This patch achieves a couple of things as follows:
- Undoing the liberasurecode_encode_cleanup specification to
expect "fragment" pointers as its arguments.
- Ensuring liberasurecode_encode to pass "fratment" pointers to
liberasurecode_encode_cleanup.
liberasurecode_encode_cleanup is used also in pyeclib so that
it is expected that the argument pointers (i.e. encoded_data and
encoded_parity) should be the collection of the heads of "fragment"
pointers.
However, when the backend encode fails, liberasurecode keeps "data"
pointers behind of fragment_header, and then, goes to "out:" statement
to cleanup its memories. It causes invalid pointer failure.
This patch adds a translation function from "data" pointers to "fragment"
pointers and ensure liberasurecode_encode to pass correct pointers to
libersurecode_encode_cleanup.
Takashi Kajinami