Commit Graph

143 Commits

Author SHA1 Message Date
Tim Burke 9c16df272a Fix bounds check in get_fragment_partition
See also: https://github.com/openstack/liberasurecode/pull/4 and
https://github.com/openstack/liberasurecode/pull/5

Co-Authored-By: utree <aproc@qq.com>
Co-Authored-By: Martin Raiber <martin@urbackup.org>
Change-Id: I174160dc1190398b6a8bbf60bec55f72ae1bec37
2023-10-04 15:08:07 -07:00
Kota Tsuyuzaki 15a257e572 Add a few coverage for legacy crc code
it actually doesn't assert the value
becase we now use same if statement
for the assertion but hope it to be
even a little better than no covarage.

Change-Id: I8860a2a8227e43e02afddcbad1e108157c0872f6
2020-10-02 21:18:02 +09:00
Tim Burke 5626cd5791 Be willing to write fragments with legacy crc
...if users *really* want to. They opt-in at run time by setting

    LIBERASURECODE_WRITE_LEGACY_CRC=1

in the environment; leaving it unset, set to an empty string, or set to
the string "0" continues to write zlib crcs.

UpgradeImpact
=============
This option is intended to allow a smooth upgrade from liberasurecode
1.5.0 and earlier in a system with multiple readers and writers:

  * Before upgrade, ensure the environment variable is set on all nodes.
    This will be ignored by earlier versions.
  * Upgrade liberasurecode on each node in the system, restarting any
    services that use it. Every node continues writing CRCs that are
    still usable by nodes that have not yet upgraded.
  * Now that every node is capable of reading zlib CRCs, remove the
    environment variable from each node to start writing new CRCs.

If you are already using 1.6.0 or later, just upgrade normally.

Closes-Bug: #1886088
Closes-Bug: #1867937
Related-Bug: #1666320
Needed-By: https://review.opendev.org/#/c/739164/
Change-Id: I9adfbe631a2dddc592fd08f8a325f3e8331b92f1
2020-09-30 16:38:11 -07:00
Pete Zaitcev 4568188d77 Do not discard a technical return value in tests
Compilers are getting smarter, and we started getting this:

libec_slap.c: In function 'test_hd_code':
libec_slap.c:285:14: error: 'frags.array' may be used uninitialized
 in this function [-Werror=maybe-uninitialized]

The fix is to consume the error code in such a way that the
test proceeds further only when frags are indeed initialized.

Change-Id: I54db0172a36419206d00b22608523a08818f41f6
2020-02-05 23:26:31 -06:00
Corey Bryant 0eb6cd321e Fix create_fake_frags_no_meta to use memset to fill frags
These aren't C strings; we shouldn't be using strncpy and friends.

Change-Id: I50cd7922dfa377ea27f3c9558a8a7268120ec733
Closes-Bug: #1859710
2020-01-28 16:33:01 -08:00
donnydavis 135c3a272f Update test to build on FreeBSD 12
Change-Id: I1ab792c3c509539a73a1808b8c28c6748185ae00
Closes-Bug: 1835071
2019-07-02 11:18:56 -04:00
Tim Burke 7b547e0e46 Allow reading of little-endian frags on big-endian
... and vice-versa. We'll fix up frag header values for our output
parameter from liberasurecode_get_fragment_metadata but otherwise
avoid manipulating the in-memory fragment much.

Change-Id: Idd6833bdea60e27c9a0148ee28b4a2c1070be148
2018-09-11 21:58:34 -06:00
Tim Burke 7247b27d70 Print test that's going to run *before* running test
Should make it easier to diagnose failures.

Change-Id: I32ed3019c556706634aae91844bf57b0954af86a
2017-09-13 15:43:57 -06:00
Jenkins d81c49161b Merge "flat_xor_hd: better validate args" 2017-09-13 21:33:43 +00:00
Jenkins 3ff4b22bc1 Merge "Negative data or parity args are invalid" 2017-09-13 21:24:27 +00:00
Tim Burke 082e4799d8 Negative data or parity args are invalid
While we're at it, tighten up some test_create_backend_invalid_args
assertions.

Change-Id: Id6c70cdb2d86580280ededc3ec6ec648c6cb7d57
2017-09-13 20:52:30 +00:00
Tim Burke cf12c9ca26 flat_xor_hd: better validate args
Previously, we'd segfault with args like k=1, m=3, hd=3.

Change-Id: I8a1d92f388c54fe4f5b357eaf85a27e6b334df11
2017-09-13 20:46:19 +00:00
Jenkins 0dd6e1e41a Merge "Use preprocessor to build test suites" 2017-09-13 20:21:52 +00:00
Jenkins dd1a91968f Merge "Use more backends in existing tests" 2017-09-13 20:21:46 +00:00
Jenkins 7ce7617c37 Merge "Make test format strings consistent with test function names" 2017-09-13 20:20:55 +00:00
Tim Burke f4a16fe929 Use preprocessor to build test suites
Otherwise, we're left with error-prone copy & pasting, which means we'll
constantly be cleaning it up.

Change-Id: I57e2cbef2c9221cffccf2e73b6af9bd003c04968
Related-Change: Ibd72ba4ae609ad77e8808aa1594b0adb62e34ef0
Related-Change: I9ee9ec3d8f86a10c4c7b5d6425a530b9c44d1156
2017-07-13 13:24:01 -07:00
Tim Burke 6124e432a5 Use more backends in existing tests
This allows us to start defining test suites for common capabilities /
allowed failure modes.

Null Backend:
    test_decode_with_missing_parity
    test_decode_with_missing_multi_parity

Jerasure RS Vand:
    test_decode_with_missing_parity

Jerasure RS Cauchy:
    test_decode_with_missing_parity

ISA-L RS Vand:
    test_decode_with_missing_parity
    test_get_fragment_metadata_crc32

ISA-L RS Cauchy:
    test_decode_with_missing_parity
    test_get_fragment_metadata_crc32

SHSS:
    test_decode_with_missing_parity
    test_get_fragment_metadata_crc32

liberasurecode RS Vand:
    test_decode_with_missing_parity

Change-Id: I9ee9ec3d8f86a10c4c7b5d6425a530b9c44d1156
2017-07-13 13:22:07 -07:00
Tim Burke 34ff5208d4 Make test format strings consistent with test function names
* When we specify a particular backend, we don't need to include it in
  the display string; we'll already include it in the output.

* test_destroy_backend_invalid_args shouldn't be reported as
  test_create_backend_invalid_args.

* Consistently include the leading "test_" in the display string.

Change-Id: Ibd72ba4ae609ad77e8808aa1594b0adb62e34ef0
2017-07-13 13:22:07 -07:00
Tim Burke e426aee95b Un-inline get/set_metatdata_chksum
Each was only really used in one place, they had some strange return types,
and recent versions of clang on OS X would refuse to compile with

erasurecode_helpers.c:531:26: error: taking address of packed member 'metadata_chksum' of
      class or structure 'fragment_header_s' may result in an unaligned pointer value
      [-Werror,-Waddress-of-packed-member]
    return (uint32_t *) &header->metadata_chksum;
                         ^~~~~~~~~~~~~~~~~~~~~~~

We don't really *care* about the pointer; we just want the value!

Change-Id: I8a5e42312948a75f5dd8b23b6f5ccfa7bd22eb1d
2017-07-06 18:02:02 +00:00
Tim Burke a9b20ae6a3 Use zlib for CRC-32
Previously, we had our own CRC that was almost but not quite like
zlib's implementation. However,

* it hasn't been subjected to the same rigor with regard to error-detection
  properties and
* it may not even get used, depending upon whether zlib happens to get
  loaded before or after liberasurecode.

Now, we'll use zlib's CRC-32 when writing new frags, while still
tolerating frags that were created with the old implementation.

Change-Id: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5
Closes-Bug: 1666320
2017-07-06 17:40:38 +00:00
Jenkins de984f59e7 Merge "Stop running targeted tests repeatedly" 2017-05-10 20:37:55 +00:00
Jenkins a17e50fd34 Merge "Stop running test_simple_encode_decode_over32 repeatedly" 2017-05-10 20:33:38 +00:00
Jenkins 345f1a93a2 Merge "Jerasure: Handle initialization errors correctly" 2017-05-10 15:19:35 +00:00
Tim Burke 842b4a9bd7 Jerasure: Handle initialization errors correctly
Otherwise we can get backtraces where we try to free something that was
never initialized.

Change-Id: Iaea427b977fd20819e2da5678cc4889d3a42dd65
2017-03-16 17:49:49 +00:00
Jenkins 26e4742140 Merge "Add Phazr.IO libphazr backend to liberasurecode" 2017-03-01 02:21:14 +00:00
Jim Cheung 06e50ea86b Add Phazr.IO libphazr backend to liberasurecode
Currently, there are several implementations of erasure codes that are
available within OpenStack Swift.  Most, if not all, of which are based
on the Reed Solomon coding algorithm.

Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction.  In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.

Please contact support@phazr.io for more info on our technology.

Change-Id: I4e40d02a8951e38409ad3c604c5dd6f050fa7ea0
2017-02-28 11:14:11 -08:00
Tim Burke de6f3bec9c Stop running targeted tests repeatedly
Previously, we would run them four times, despite always using
the same backends and arguments.

Change-Id: Iede8e9e9689cbaf85c66adce72cb78983afb084b
2017-02-25 20:21:19 +00:00
Tim Burke 11244e449f Stop running test_simple_encode_decode_over32 repeatedly
It uses hard-coded backend and arguments, so it should use EC_BACKEND_MAX
and have a name that reflect how it's only targeted to Jerasure.

This is similar to how we handled the targeted test case for ISA-L.

Change-Id: I0a397930b2f49c0775290970c820bc70310a475e
2017-02-25 20:13:11 +00:00
Kota Tsuyuzaki c9136a62b6 Fix valgrind-check and memory leak
Can you believe that we ware testing the memory leak with valgrind
to just *bash scripts* instead of actual binaries on liberasurecode_test
and libec_slap?

That is why we cannot find such an easy memory leak[1] at the gate.
Now this patch enable to run the valgrind against to the binaries.

With this fix, we found various memory leak at liberasurecode_test as
follows and this patch also fixes them:

- If we create fake fragments, we're responsible for freeing all of the
  frags as well as the array holding the pointers to the frags.
- If we allocate any space, we're responsible for freeing it.
- If we create an EC descriptor, we're responsible for destroying it.
- If we create a fragment or skip array, we're responsible for freeing it.
- If that happens inside a loop, we're responsible for doing it *inside
  that same loop*.

In addition to the test fix, this patch fixes following memory leaks at
the code which is affected to other users (pyeclib, OpenStack Swift)

* Refuse to decode fragments that aren't even long enough to include
  fragment headers.
* Fix a small memory leak in the builtin rs_vand implementation.

Closes-Bug: #1665242

Co-Authored-By: Tim Burke <tim@swiftstack.com>

1: https://review.openstack.org/#/c/431812

Change-Id: I96f124e4e536bbd7544208acc084de1cda5c19b2
2017-02-20 07:45:55 -08:00
Jenkins dc77b681aa Merge "Add get_version functionality to liberasurecode lib" 2016-12-08 19:54:37 +00:00
Jenkins 4c8f8a65d2 Merge "ISA-L Cauchy support" 2016-12-07 02:34:04 +00:00
Kota Tsuyuzaki 8d067ab2f6 ISA-L Cauchy support
This is for supporting ISA-L cauchy based matrix. The difference
from isa_l_rs_vand is only the matrix to use the encode/decode calculation.

As a known issue, isa_l_rs_vand backend has constraint for the
combinations of the available fragment to be able to decode/reconstuct.
(See related change in detail)

To avoid the constraint, this patch adds another isa-l backend to use
cauchy matrix and keep the backward compatibility, this is in
another isa_l_rs_cauchy namespace.

For implementation consieration, the code is almost same except the matrix
generation fucntion so that this patch makes isa_l_common.c file for
gathering common fucntions like init/encode/decode/reconstruct. And then the
common init funciton takes an extra args "gen_matrix_func_name" for entry
point to load the fucntion by dlsym from isa-l .so file.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87
Change-Id: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
2016-12-06 17:08:55 -08:00
Kota Tsuyuzaki 09d8bbf361 Add get_version functionality to liberasurecode lib
Currently, we have liberasurecode version info in the header and pyeclib
is using the info to detect the version. However it's a bit painful
because it requires to rebuild pyeclib c code for you to see the actual
installed version.

This addition for liberasurecode_get_version enables caller to get the
version integer from compiled shared library file (.so) and it will
rescure to re-compiled operation from pyeclib.

Change-Id: I8161ea7da3b069e83c93e11cb41ce12fa60c6f32
2016-12-04 20:56:24 -08:00
Jenkins f396208f69 Merge "Beautify fill_buffer()" 2016-11-10 19:07:58 +00:00
Kota Tsuyuzaki 74eaa374c1 Fix error handling on gf_ivnert_matrix in isa-l backend
Current isa-l has possibility to return corrupted decoded data or
corrupted reconstructed data on decode/reconstruct without error code.

That is from the specification of isa-l rs vandermond matrix discussed
at [1]. With many # of parities cases, we may hit the case above due to
failing to get the inverse matrix from the encode matrix.

The isa-l maintener gbtucker suggests a good way to detect the failing
inverse matrix, that we should handle the return value gf_invert_matrix.
If gf_invert_matrix returns not 0, we should stop to decode/reconstruct
and return failure return code to the caller immediately. Otherwise, the
caller regards the garbage data/fragment as correct one.

And this patch adds the specific test case we can hit the issue (it
happens not so general).

1: https://github.com/01org/isa-l/issues/10

Related-Change: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
Change-Id: Icee788a0931fe692fe0de31fabc4ba450e338a87
2016-11-09 23:28:55 +00:00
Kota Tsuyuzaki e8b5a64d8d Fix liberasurecode skipping a bunch of invalid_args tests
Since the commit a01b1818c8,
we have been to able to test some parameters for each tests.
However, instead, the NULL which means the end of the test parameters
causes skipping a bunch of unit tests which doesn't take args (e.g.
test_create_backend_invalid_args). That is the worse because we have not
tested anymore for the tests since the commit.

This patch fixes to make it tested and more, fix a minor bug for a
case that corrupted header incomming to get_metadata.

Closes-Bug: #1634403
Change-Id: Ib99a8aa6032f02d0c7d1ab94b8da1ebfd9047d74
2016-11-04 18:42:11 +00:00
Pete Zaitcev 169a457198 Beautify fill_buffer()
This is completely pointless and speeds up the tests by something
like 4% (less than a second), but my eyes BLEED when I see the
existing code.

Change-Id: Ia1d163e99bd2a2c2196b2669e4faa8707dbfc8d3
2016-10-14 22:44:24 -06:00
Tushar Gohad a6a8d2018d Fragment metadata checksumming support 2016-03-10 07:33:16 +00:00
Timur Alperovich c7a94df072 Add a method to check if a backend is present.
Uses dlopen to check if a backend is present. This may be used by
consumers who need to check which backends are present on a system.

Issue #23
2016-03-08 22:06:19 -08:00
Dirk Mueller 92b7598bd3 Fix prototype declaration / definition mismatch (issue #25) 2016-03-04 20:36:12 +01:00
Tushar Gohad 17a6dbeb93 Check frag idx validity when verifying frag metadata
... also add related test code
2016-02-14 07:15:24 +00:00
Tushar Gohad 02fe1dc7cc Make libtool archive references relative to builddir 2015-12-10 06:14:20 -07:00
Tushar Gohad 398c48c95d Split helpers.h include for backward compatibility
Users of liberasurecode <= 1.0.7 used alloc/free helpers
(which they shouldn't have).  This change is to make sure
we are still able to those older revs of programs and they
work with newer liberasurecode.
2015-09-22 21:47:15 +00:00
Kota Tsuyuzaki c9ce822825 Enforce the (k + m < 32) limit, add unit tests 2015-08-18 08:30:34 +00:00
Tushar Gohad fdb36f1e7a Rename remaining INTERNAL_RS_VAND instances
... to LIBERASURECODE_RS_VAND
2015-07-20 04:19:09 +00:00
Tushar Gohad 9a01459117 Rename liberasurecode_rsvand -> liberasurecode_rs_vand 2015-06-22 02:54:53 +00:00
Tushar Gohad b15717a6c7 Rename rs_vand_internal to liberasurecode_rs_vand 2015-06-22 00:40:05 +00:00
Tushar Gohad 6fac472e8f Fix open/write() and uninitialize var warnings in rs_vand internal 2015-06-22 00:00:07 +00:00
Kevin Greenan e9ca485c7c Plugging new internal RS backend into liberasurecode. 2015-06-18 12:31:17 -07:00
Kevin Greenan fe537605b3 Adding internal tests for the new internal RS codec. 2015-06-18 12:31:17 -07:00