Commit Graph

446 Commits

Author SHA1 Message Date
Tony Breeds 6da609fae9 Retire Packaging Deb project repos
This commit is part of a series to retire the Packaging Deb
project. Step 2 is to remove all content from the project
repos, replacing it with a README notification where to find
ongoing work, and how to recover the repo if needed at some
future point (as in
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project).

Change-Id: I789570f60e2ca4e98b21ece64c995352778049d8
2017-09-12 15:40:21 -06:00
Tim Burke e426aee95b Un-inline get/set_metatdata_chksum
Each was only really used in one place, they had some strange return types,
and recent versions of clang on OS X would refuse to compile with

erasurecode_helpers.c:531:26: error: taking address of packed member 'metadata_chksum' of
      class or structure 'fragment_header_s' may result in an unaligned pointer value
      [-Werror,-Waddress-of-packed-member]
    return (uint32_t *) &header->metadata_chksum;
                         ^~~~~~~~~~~~~~~~~~~~~~~

We don't really *care* about the pointer; we just want the value!

Change-Id: I8a5e42312948a75f5dd8b23b6f5ccfa7bd22eb1d
2017-07-06 18:02:02 +00:00
Tim Burke a9b20ae6a3 Use zlib for CRC-32
Previously, we had our own CRC that was almost but not quite like
zlib's implementation. However,

* it hasn't been subjected to the same rigor with regard to error-detection
  properties and
* it may not even get used, depending upon whether zlib happens to get
  loaded before or after liberasurecode.

Now, we'll use zlib's CRC-32 when writing new frags, while still
tolerating frags that were created with the old implementation.

Change-Id: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5
Closes-Bug: 1666320
2017-07-06 17:40:38 +00:00
Tim Burke 9b4d8bcf8d Stop pretending to support SSE4-optimized CRC-32C
It isn't the CRC we want, and we never used it anyway. While we may
define INTEL_SSE41 or INTEL_SSE42 if CPU seems to support it, we've
never defined INTEL_SSE4.

Change-Id: I04e1dd6458ccde58a0a2f3f4d6947569a31e9697
Partial-Bug: 1666320
2017-07-06 00:08:45 +00:00
Jenkins 24038b3588 Merge "Release 1.5.0" 2017-06-29 17:36:37 +00:00
Jenkins dd31ac647b Merge "Stop using ceill() to compute padded data size" 2017-06-06 19:17:12 +00:00
Pete Zaitcev 960cdd09dc Stop using ceill() to compute padded data size
The well-known idiom to compute a required number of data blocks
of size B to contain data of length d is:

     (d + (B-1))/B

The code we use, with ceill(), computes the same value, but does
it in an unorthodox way. This makes a reviewer to doubt himself
and even run tests to make sure we're really computing the
obvious thing.

Apropos the reviewer confusion, the code in Phazr.IO looks weird.
It uses (word_size - hamming_distance) to compute the necessary
number of blocks... but then returns the amount of memory needed
to store blocks of a different size (word_size). We left all of it
alone and return exactly the same values that the old computation
returned.

All these computations were the only thing in the code that used
-lm, so drop that too.

Coincidentially, this patch solves the crash of distro-built
packages of liberasurecode (see Red Hat bug #1454543). But it's
a side effect. Expect a proper patch soon.

Change-Id: Ib297f6df304abf5ca8c27d3392b1107a525e0be0
2017-06-06 14:53:45 -04:00
Thiago da Silva 063885cc3c Release 1.5.0
Change-Id: I6903e11a24f548a07f924cef8f0bc8ba3c456ef0
2017-06-05 14:32:32 -04:00
Pete Zaitcev 0962144c51 Allow to disable optimizations for portability
We're having trouble on Fedora when the build system runs
on Intel CPUs. The ./configure detects AVX instructions and
builds liberasurecode with them. The resulting library crashes
with SIGILL when users run it on ADM CPUs without AVX.

See the details here:
 https://bugzilla.redhat.com/show_bug.cgi?id=1454543

The patch allows to disable the optimizations, so that
distro packaging then can invoke this options and build
portable libraries.

For the record, "make test" runs about 16% slower on an Intel
CPU if optimizations are disabled. So, there's a measurable
performance impact. However, operators intersted in outright
performance might want to consider Erasure Coding implementations
other than the one  built-in into liberasurecode. The library
supports other back-ends that are significantly faster than
even optimized built-in code.

Change-Id: I09603b97ceeb833ba582cf3217e0be51c019d645
2017-05-24 14:42:36 -06:00
Jenkins de984f59e7 Merge "Stop running targeted tests repeatedly" 2017-05-10 20:37:55 +00:00
Jenkins a17e50fd34 Merge "Stop running test_simple_encode_decode_over32 repeatedly" 2017-05-10 20:33:38 +00:00
Jenkins 0237240926 Merge "Fix warning on automake tool" 2017-05-10 15:35:41 +00:00
Jenkins 345f1a93a2 Merge "Jerasure: Handle initialization errors correctly" 2017-05-10 15:19:35 +00:00
Jenkins 7c4ef60c9d Merge "ISA-L: Only calculate gf tables on init, not every encode" 2017-03-29 06:49:41 +00:00
Tim Burke 842b4a9bd7 Jerasure: Handle initialization errors correctly
Otherwise we can get backtraces where we try to free something that was
never initialized.

Change-Id: Iaea427b977fd20819e2da5678cc4889d3a42dd65
2017-03-16 17:49:49 +00:00
Jenkins 26e4742140 Merge "Add Phazr.IO libphazr backend to liberasurecode" 2017-03-01 02:21:14 +00:00
Jim Cheung 06e50ea86b Add Phazr.IO libphazr backend to liberasurecode
Currently, there are several implementations of erasure codes that are
available within OpenStack Swift.  Most, if not all, of which are based
on the Reed Solomon coding algorithm.

Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction.  In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.

Please contact support@phazr.io for more info on our technology.

Change-Id: I4e40d02a8951e38409ad3c604c5dd6f050fa7ea0
2017-02-28 11:14:11 -08:00
Tim Burke de6f3bec9c Stop running targeted tests repeatedly
Previously, we would run them four times, despite always using
the same backends and arguments.

Change-Id: Iede8e9e9689cbaf85c66adce72cb78983afb084b
2017-02-25 20:21:19 +00:00
Tim Burke 11244e449f Stop running test_simple_encode_decode_over32 repeatedly
It uses hard-coded backend and arguments, so it should use EC_BACKEND_MAX
and have a name that reflect how it's only targeted to Jerasure.

This is similar to how we handled the targeted test case for ISA-L.

Change-Id: I0a397930b2f49c0775290970c820bc70310a475e
2017-02-25 20:13:11 +00:00
Daniel Axtens 4ab1336cab jerasure: plug memory leaks
Jerasure inits some global variables on init.
We need to free them, or we'll leak memory.

Partial-Bug: #1666674

Change-Id: Ie7073738428a71910016e910a66dbd92ca98eb92
Signed-off-by: Daniel Axtens <dja@axtens.net>
2017-02-21 20:50:03 +00:00
Kota Tsuyuzaki 19442df2cd Fix warning on automake tool
This patch fixes automake tools warnings as follows (and a my fault
in previous patch [1]):

- Change INCLUDES variable to APP_CPP_FLAGS and APP_C_FLAGS according to
  the warning message

- Remove non-POSIX variable to suppress that warnings

Note that the latter change may be affected to the testing so please
be careful on your review what's going on your testing environment.
In the past change [2], they ware designed to use the shared
object binaries in the build path. However, Daniel had pointed out the good
thing at [1] if we runs the test/liberasruecode_test (it's shell with linker)
in the dir, we can touch the expected binaries.

My fault in the previous patch is that I didn't replace
./test/.libs/liberasurecode_test to ./test/liberasurecode_test even
we use --trace-children=yes option for the valgrind.

Now, all tests run via ./test/<test name> syntax and IMO no matters
exist even if we remove the non-POSIX variable for test settings.

1: https://review.openstack.org/#/c/434162
2: 93446db941

Change-Id: I0e79bed7755a1f286b746a70fcf56fdc972bfd5d
2017-02-20 14:23:16 -08:00
Kota Tsuyuzaki c9136a62b6 Fix valgrind-check and memory leak
Can you believe that we ware testing the memory leak with valgrind
to just *bash scripts* instead of actual binaries on liberasurecode_test
and libec_slap?

That is why we cannot find such an easy memory leak[1] at the gate.
Now this patch enable to run the valgrind against to the binaries.

With this fix, we found various memory leak at liberasurecode_test as
follows and this patch also fixes them:

- If we create fake fragments, we're responsible for freeing all of the
  frags as well as the array holding the pointers to the frags.
- If we allocate any space, we're responsible for freeing it.
- If we create an EC descriptor, we're responsible for destroying it.
- If we create a fragment or skip array, we're responsible for freeing it.
- If that happens inside a loop, we're responsible for doing it *inside
  that same loop*.

In addition to the test fix, this patch fixes following memory leaks at
the code which is affected to other users (pyeclib, OpenStack Swift)

* Refuse to decode fragments that aren't even long enough to include
  fragment headers.
* Fix a small memory leak in the builtin rs_vand implementation.

Closes-Bug: #1665242

Co-Authored-By: Tim Burke <tim@swiftstack.com>

1: https://review.openstack.org/#/c/431812

Change-Id: I96f124e4e536bbd7544208acc084de1cda5c19b2
2017-02-20 07:45:55 -08:00
Daniel Axtens 90679884ef ISA-L: Only calculate gf tables on init, not every encode
Currently, the Galois Field multiplication tables are recalcuated
every time an encode is done. This is wasteful, as they are fixed
by k and m, which is set on init.

Calculate the tables only once, on init.

This trades off a little bit of per-context memory and creation
time for measurably faster encodes when using the same context.

On powerpc64le, when repeatedly encoding a 4kB file with pyeclib,
this increases the measured speed by over 10%.

Change-Id: I2f025aaee2d13cb1717a331e443e179ad5a13302
Signed-off-by: Daniel Axtens <dja@axtens.net>
2017-02-13 09:43:24 +11:00
Daniel Axtens 10e096fe09 ISA-L: free matrix on exit
isa_l_common_init allocates desc->matrix, but this isn't freed in
isa_l_exit. Instead, the entire isa_l_desc structure is freed,
thus leaking the memory.

Explicitly free desc->matrix.

Change-Id: Ibf672d1a309498591b87d739632a90a1b3704f7e
Signed-off-by: Daniel Axtens <dja@axtens.net>
2017-02-10 13:20:42 +11:00
Jenkins 0794b31c62 Merge "1.4.0 release" 2016-12-08 20:31:18 +00:00
Thiago da Silva 448c7d0d1c 1.4.0 release
Change-Id: I1d8d6b5711a503eaa7c57c70b4c20a329f572af2
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2016-12-08 19:57:09 +00:00
Jenkins dc77b681aa Merge "Add get_version functionality to liberasurecode lib" 2016-12-08 19:54:37 +00:00
Jenkins 4c8f8a65d2 Merge "ISA-L Cauchy support" 2016-12-07 02:34:04 +00:00
Kota Tsuyuzaki 8d067ab2f6 ISA-L Cauchy support
This is for supporting ISA-L cauchy based matrix. The difference
from isa_l_rs_vand is only the matrix to use the encode/decode calculation.

As a known issue, isa_l_rs_vand backend has constraint for the
combinations of the available fragment to be able to decode/reconstuct.
(See related change in detail)

To avoid the constraint, this patch adds another isa-l backend to use
cauchy matrix and keep the backward compatibility, this is in
another isa_l_rs_cauchy namespace.

For implementation consieration, the code is almost same except the matrix
generation fucntion so that this patch makes isa_l_common.c file for
gathering common fucntions like init/encode/decode/reconstruct. And then the
common init funciton takes an extra args "gen_matrix_func_name" for entry
point to load the fucntion by dlsym from isa-l .so file.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87
Change-Id: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
2016-12-06 17:08:55 -08:00
Kota Tsuyuzaki 09d8bbf361 Add get_version functionality to liberasurecode lib
Currently, we have liberasurecode version info in the header and pyeclib
is using the info to detect the version. However it's a bit painful
because it requires to rebuild pyeclib c code for you to see the actual
installed version.

This addition for liberasurecode_get_version enables caller to get the
version integer from compiled shared library file (.so) and it will
rescure to re-compiled operation from pyeclib.

Change-Id: I8161ea7da3b069e83c93e11cb41ce12fa60c6f32
2016-12-04 20:56:24 -08:00
Jenkins fff899253e Merge "Fix posix_memalign handling" 2016-11-29 20:21:53 +00:00
Jenkins 0a79a4889e Merge "Fix reconstruct to return an error when memory allocation failed" 2016-11-24 00:18:09 +00:00
Jenkins bcc98b5ace Merge "Change link to pyeclib in README.md" 2016-11-11 09:52:41 +00:00
Jenkins f396208f69 Merge "Beautify fill_buffer()" 2016-11-10 19:07:58 +00:00
John Dickinson 0a2c06b8b4 Version bump to 1.3.1
Change-Id: Ia45c7b46ea45dee6f306afe291fe6a908eb41d70
2016-11-09 22:43:35 -08:00
John Dickinson d0cfa46390 1.3.0 release
Change-Id: I81ad241db741afda0afee4278b143f2615cfe621
2016-11-09 15:37:32 -08:00
Kota Tsuyuzaki 74eaa374c1 Fix error handling on gf_ivnert_matrix in isa-l backend
Current isa-l has possibility to return corrupted decoded data or
corrupted reconstructed data on decode/reconstruct without error code.

That is from the specification of isa-l rs vandermond matrix discussed
at [1]. With many # of parities cases, we may hit the case above due to
failing to get the inverse matrix from the encode matrix.

The isa-l maintener gbtucker suggests a good way to detect the failing
inverse matrix, that we should handle the return value gf_invert_matrix.
If gf_invert_matrix returns not 0, we should stop to decode/reconstruct
and return failure return code to the caller immediately. Otherwise, the
caller regards the garbage data/fragment as correct one.

And this patch adds the specific test case we can hit the issue (it
happens not so general).

1: https://github.com/01org/isa-l/issues/10

Related-Change: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
Change-Id: Icee788a0931fe692fe0de31fabc4ba450e338a87
2016-11-09 23:28:55 +00:00
Jenkins b96cf4b2cc Merge "Fix liberasurecode skipping a bunch of invalid_args tests" 2016-11-09 23:08:32 +00:00
Jenkins 336185ae84 Merge "Fix clang compile time error" 2016-11-07 05:27:21 +00:00
Kota Tsuyuzaki e8b5a64d8d Fix liberasurecode skipping a bunch of invalid_args tests
Since the commit a01b1818c8,
we have been to able to test some parameters for each tests.
However, instead, the NULL which means the end of the test parameters
causes skipping a bunch of unit tests which doesn't take args (e.g.
test_create_backend_invalid_args). That is the worse because we have not
tested anymore for the tests since the commit.

This patch fixes to make it tested and more, fix a minor bug for a
case that corrupted header incomming to get_metadata.

Closes-Bug: #1634403
Change-Id: Ib99a8aa6032f02d0c7d1ab94b8da1ebfd9047d74
2016-11-04 18:42:11 +00:00
Kota Tsuyuzaki 1b16077193 Remove Ryuta Kon from NTT shss reference
He is not working at swift/pyeclib anymore and he doesn't want to keep
his name in the README.md which causes contacts/questions to him.

Related-change: I0d76fa97dbb8c7f65b1dfa7cf3d0f1771893e1ef

Change-Id: I97607915a5296359eb2142c6cd3eac7b1ee6a74a
2016-10-23 20:52:21 -07:00
Jenkins 0d2056f677 Merge "Fix a typo in the erasurecode file" 2016-10-18 12:36:08 +00:00
Pete Zaitcev 169a457198 Beautify fill_buffer()
This is completely pointless and speeds up the tests by something
like 4% (less than a second), but my eyes BLEED when I see the
existing code.

Change-Id: Ia1d163e99bd2a2c2196b2669e4faa8707dbfc8d3
2016-10-14 22:44:24 -06:00
Kota Tsuyuzaki 10d6fa5f02 Fix reconstruct to return an error when memory allocation failed
As like current code, if we do go jump to just "out" section when
memory allocation failed, liberaurecode/PyECLib will report it as
success. However, in fact, the returned binary is not the one
reconstructed. This patch fixes liberasurecode_reconstruct to return
error code (ENOMEM) if either malloc or memset for working space is
failed.

Change-Id: I7bfc481c7a9bbc64288760df2dc6053c57657b41
2016-10-11 19:35:04 -07:00
Kota Tsuyuzaki cb0daba975 Add get_by_desc ret val handling to get_fragment_size
As well as any other callers, libersurecode_get_fragment_size should
handle the return value of liberasurecode_get_backend_instance_by_desc.

Otherwise, get_by_desc can return NULL and it causes an invalid memory
access in librerasurecode_get_fragment_size.

Change-Id: I489f8b5d049610863b5e0b477b6ff70ead245b55
2016-09-07 20:22:47 -07:00
gengchc2 ec64eeb5b6 Fix a typo in the erasurecode file
there is a spelling error,"instace" should be "instance"

Change-Id: Iddba9e334bf55cc1e7d5cb397db5f3dfc5aac584
2016-08-30 14:49:18 +08:00
Jenkins 89807c5aac Merge "Move other-requirements.txt to bindep.txt" 2016-08-23 12:59:57 +00:00
Jenkins f13100013f Merge "Changed homepage to new one" 2016-08-18 09:02:27 +00:00
Andreas Jaeger 72122cbf0c Move other-requirements.txt to bindep.txt
The default filename for documenting binary dependencies has been
changed from "other-requirements.txt" to "bindep.txt" with the release
of bindep 2.1.0. While the previous name is still supported, it will
be deprecated.

Move the file around to follow this change.

Note that this change is self-testing, the OpenStack CI infrastructure
will use a "bindep.txt" file to setup nodes for testing.

For more information about bindep, see also:
http://docs.openstack.org/infra/manual/drivers.html#package-requirements
http://docs.openstack.org/infra/bindep/

As well as this announcement:
http://lists.openstack.org/pipermail/openstack-dev/2016-August/101590.html

Change-Id: I7231bc3c3e08b0f912919d1d2a4900d4074ad9f7
2016-08-12 20:54:16 +02:00
Kota Tsuyuzaki 74a028858a Add valgind to other-requirements.txt
This is needed to run unit test with "make valgrind-test"

Change-Id: I3c2979c2f3cd141ff8db3e6358fa356b0581e599
2016-07-13 09:58:07 -07:00