Currently, there are several implementations of erasure codes that are
available within OpenStack Swift. Most, if not all, of which are based
on the Reed Solomon coding algorithm.
Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction. In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.
Please contact support@phazr.io for more info on our technology.
Change-Id: I4e40d02a8951e38409ad3c604c5dd6f050fa7ea0
This is for supporting ISA-L cauchy based matrix. The difference
from isa_l_rs_vand is only the matrix to use the encode/decode calculation.
As a known issue, isa_l_rs_vand backend has constraint for the
combinations of the available fragment to be able to decode/reconstuct.
(See related change in detail)
To avoid the constraint, this patch adds another isa-l backend to use
cauchy matrix and keep the backward compatibility, this is in
another isa_l_rs_cauchy namespace.
For implementation consieration, the code is almost same except the matrix
generation fucntion so that this patch makes isa_l_common.c file for
gathering common fucntions like init/encode/decode/reconstruct. And then the
common init funciton takes an extra args "gen_matrix_func_name" for entry
point to load the fucntion by dlsym from isa-l .so file.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87
Change-Id: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
Currently, we have liberasurecode version info in the header and pyeclib
is using the info to detect the version. However it's a bit painful
because it requires to rebuild pyeclib c code for you to see the actual
installed version.
This addition for liberasurecode_get_version enables caller to get the
version integer from compiled shared library file (.so) and it will
rescure to re-compiled operation from pyeclib.
Change-Id: I8161ea7da3b069e83c93e11cb41ce12fa60c6f32
As well as any other callers, libersurecode_get_fragment_size should
handle the return value of liberasurecode_get_backend_instance_by_desc.
Otherwise, get_by_desc can return NULL and it causes an invalid memory
access in librerasurecode_get_fragment_size.
Change-Id: I489f8b5d049610863b5e0b477b6ff70ead245b55
For get_segment_info function of PyECLib, liberasurecode should
support get_fragment_size function because if pyeclib and liberasurecode
have the calculation of fragment size each other, it might cause
the size mismatch (i.e. it'll be a bug) in the future development work.
This patch introduces liberasurecode_get_fragment_size function to return
the fragment_size calculated at liberasurecode accoring to specified
backend descriptor.
It really usefull to help caller knows how large size it have to expect
and all pyeclib has to do for retrieving fragment_size will be just calling
the liberasurecode_get_fragment_size function on get_segment_info.
This patch allows to get correct fragment size includes metadata_adder.
Current implementaion automatically allocates extra bytes for the metadata_adder
in alloc_buffer, and then, no information about the extra bytes will be returned
to the api caller side. It's too confusable because callers couldn't know how size they
assumes as the fragment size.
To be easy to find out the size infomation, this patch adds "frag_adder_size"
variable into fragment metadata and also some functions to get fragment size.
The definitions of these size infomation are here,
fragment_meta:
- size-> raw data size used to encode/fragment_to_string
- frag_adder_size-> metadata_adder of backend specification
And the definitions of functions are here,
- get_fragment_size:
-> return sizeof(fragument_header) + size + frag_adder_size
- get_fragment_buffer_size:
-> return size + frag_adder_size
- get_fragment_payload_size:
-> return size
By using these function above, users could get the size information
directly from fragments. It results in enabling to return fragment_len
to the caller side easily.
This introduces a new plug-able backend called "shss" made by
Nippon Telegraph and Telephone corporation (NTT).
Note that this produces a just plug-in to shss erasure coding binary
so that users have to install a shss binary (i.e. libshss.so) aside from
liberasurecode when using shss.
Please contact us if you are insterested in the NTT backend (welcome!):
Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Co-Author: Ryuta Kon <kon.ryuta@po.ntts.co.jp>
liberasure_get_fragment_metadata and
liberasurecode_verify_stripe_metadata. In the process removed the
liberasure descriptor from the liberasure_get_fragment_metadata
function's signature as it was not being used in the function's
implementation.
follows a common format and that it accurately reflects the code.
- Cleaned up liberasurecode_instance_destroy to ensure that the return
code is correct and that it is not trying to free data that has
not been removed from the open instances.
- Updated the tests to assert that the liberasurecode_instance_destroy
method is return correctly.
Jim Cheung