Commit Graph

272 Commits

Author SHA1 Message Date
Kota Tsuyuzaki 9e5f83b32a Fix segmentation_fault on shss with no priv_args 2015-02-27 04:54:51 -08:00
Kota Tsuyuzaki 4ca25e9629 Fix isa-l backend to use correct word size 2015-02-27 04:54:51 -08:00
Kota Tsuyuzaki 84cc8cbacb Add contributors into AUTHORS file 2015-02-27 04:54:51 -08:00
Kota Tsuyuzaki 54da656c65 Add liberasurecode_get_fragment_size function
For get_segment_info function of PyECLib, liberasurecode should
support get_fragment_size function because if pyeclib and liberasurecode
have the calculation of fragment size each other, it might cause
the size mismatch (i.e. it'll be a bug) in the future development work.

This patch introduces liberasurecode_get_fragment_size function to return
the fragment_size calculated at liberasurecode accoring to specified
backend descriptor.

It really usefull to help caller knows how large size it have to expect
and all pyeclib has to do for retrieving fragment_size will be just calling
the liberasurecode_get_fragment_size function on get_segment_info.
2015-02-27 04:54:50 -08:00
Kota Tsuyuzaki a8c8ed9adb Fix small things
Small fixes as follows:

- Add is_compatible_with function into shss backend
- Remove encoded data check against to shss at liberasurecode_test.c
- Decrease metadata_adder size on shss backend to be correct fixed value
2015-02-27 04:54:50 -08:00
Kota Tsuyuzaki 0addebdbf6 Enable to get fragment_len includes metadata_adder
This patch allows to get correct fragment size includes metadata_adder.

Current implementaion automatically allocates extra bytes for the metadata_adder
in alloc_buffer, and then, no information about the extra bytes will be returned
to the api caller side. It's too confusable because callers couldn't know how size they
assumes as the fragment size.

To be easy to find out the size infomation, this patch adds "frag_adder_size"
variable into fragment metadata and also some functions to get fragment size.

The definitions of these size infomation are here,

fragment_meta:
- size-> raw data size used to encode/fragment_to_string
- frag_adder_size-> metadata_adder of backend specification

And the definitions of functions are here,

- get_fragment_size:
  -> return sizeof(fragument_header) + size + frag_adder_size

- get_fragment_buffer_size:
  -> return size + frag_adder_size

- get_fragment_payload_size:
  -> return size

By using these function above, users could get the size information
directly from fragments. It results in enabling to return fragment_len
to the caller side easily.
2015-02-27 04:54:50 -08:00
Kota Tsuyuzaki 1b08cb4376 Change metdata_adder type from int into size_t 2015-02-27 15:44:37 +09:00
Kota Tsuyuzaki 72746d1c51 Move backend metadata adding to fragment allocation
On the first consideration[1], metadata_adder is defined as a extra byte
size for "each" fragment. However, current implementation is an element
affects to data_len. (i.e. aligned_data_size for original segment data)

We should make metadata_adder to be a fixed value against to each fragment,
otherwise the extra bytes for the fragments will have flexible length depends
on "K". It will be quite complex for backend implementation to know "How large
bytes the raw data size is", "How large bytes the backend could use as extra
bytes for each fragment".

1: 032b57d9b1?at=master
2015-02-27 15:44:37 +09:00
Kota Tsuyuzaki 57f5c565e6 Ensure fragment pointers passed to cleanup
This patch achieves a couple of things as follows:

- Undoing the liberasurecode_encode_cleanup specification to
  expect "fragment" pointers as its arguments.

- Ensuring liberasurecode_encode to pass "fratment" pointers to
  liberasurecode_encode_cleanup.

liberasurecode_encode_cleanup is used also in pyeclib so that
it is expected that the argument pointers (i.e. encoded_data and
encoded_parity) should be the collection of the heads of "fragment"
pointers.

However, when the backend encode fails, liberasurecode keeps "data"
pointers behind of fragment_header, and then, goes to "out:" statement
to cleanup its memories. It causes invalid pointer failure.

This patch adds a translation function from "data" pointers to "fragment"
pointers and ensure liberasurecode_encode to pass correct pointers to
libersurecode_encode_cleanup.
2015-02-27 15:44:37 +09:00
Kota Tsuyuzaki aa0c960504 Fix get_fragment_partition return value
When num_missing is over than the num of parities (i.e. > m),
get_fragment_partition should return -1 as an error code.

This patch fixes it and adds a test called "test_get_fragment_partition"
into liberasurecode_test.c.
2015-02-27 15:24:04 +09:00
Kota Tsuyuzaki 289099c43d Fix shss to instantiate backend discriptor 2015-02-26 02:15:29 -08:00
Tushar Gohad ac36408e50 README.md edited online with Bitbucket 2015-02-25 18:03:00 +00:00
Tushar Gohad e4d9e53da3 v1.0-rc1
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-02-25 11:00:20 -07:00
Tushar Gohad 59ff594de5 Update README with 'shss' backen
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-02-25 10:47:39 -07:00
Tushar Gohad 9e44a94f15 Convert README.md from dos to unix format
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-02-25 10:43:17 -07:00
Tushar Gohad aade6ef5d9 Fix isa_l backend source filename (botched during 'shss' backend merge)
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-02-25 10:41:27 -07:00
Tushar Gohad e5c629643a Update references to external projects
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-02-25 10:34:16 -07:00
Tushar Gohad aed012b0ec Update isa-l backend to v2.13
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-02-25 10:34:16 -07:00
Tushar Gohad f8eb9bb85e Merged ntt_backend into master 2015-02-25 10:26:15 -07:00
Kota Tsuyuzaki b30a15d87f Add NTT backend called "shss"
This introduces a new plug-able backend called "shss" made by
Nippon Telegraph and Telephone corporation (NTT).

Note that this produces a just plug-in to shss erasure coding binary
so that users have to install a shss binary (i.e. libshss.so) aside from
liberasurecode when using shss.

Please contact us if you are insterested in the NTT backend (welcome!):
Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>

Co-Author: Ryuta Kon <kon.ryuta@po.ntts.co.jp>
2015-02-10 18:28:05 -08:00
Kevin Greenan aa276d4322 This is the fix propsed by Kota to fix the reconstruction errors
we were seeing after the checksum enum changes.
2015-02-07 15:10:31 -08:00
Kevin Greenan 14b1e5bb64 Fix the extended integrity checks in the liberasurecode decode
function.  Previously, it was failing if the number of invalid
fragments was k or greater, which is incorrect.

We should only be able to decode if:

  (num given fragments - num invalid fragments) >= k

This means fail if:

  (num given fragments - num invalid fragments) < k
2015-02-07 14:40:31 -08:00
Kevin Greenan cf490de5e0 Revert "Merged fix-cleanup into master"
This reverts commit dc4e6f07ce, reversing
changes made to 92c39d590c.
2015-02-07 14:39:21 -08:00
Tushar Gohad 6d9c09faeb Adjust checksum enum defintions to match pyeclib
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-02-01 15:03:39 -07:00
Tushar Gohad dc4e6f07ce Merged fix-cleanup into master 2015-01-30 18:54:20 -07:00
Tushar Gohad 92c39d590c doc: Update decode() param force_metadata_checks
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-01-30 18:42:05 -07:00
Tushar Gohad ff5dd39017 Add optional fragment metadata check for decode
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
2015-01-30 18:42:04 -07:00
Kevin Greenan 09efa2dd84 Decreasing the verbosity of the tests and renaming one of the stripe verify
tests to a 'fragment verify' test, since it is using is_valid_fragment and
and validating fragment headers, not stripe-level.
2015-01-19 10:47:41 -08:00
Kevin Greenan da335a6914 The docs were not building/installing for me. This fixed it. 2015-01-19 10:47:13 -08:00
Kevin Greenan 070a219343 Updating gitignore to ignore more AM/AC stuff 2015-01-19 10:46:14 -08:00
Kota Tsuyuzaki c1eb3e6e6c Fix free fragments memories at a backend error
When failing at calling a backend encode (i.e. the backend return a minus value),
invalid pointer error will occur at liberasurecode_encode_cleanup() because it
will work to free the memory from data[i] behind the fragment header.

This patch fixes it to use free_fragment_buffer to free whole memory includes
the fragment header field.

TODO:
Add tests (or stub backend) to verify liberasurecode behaviors when a failure
returnd from the backend.
2015-01-13 01:27:17 -08:00
Kevin Greenan 380731b449 Make install failing to find install-sh when installing docs 2015-01-12 20:35:11 -08:00
Kevin Greenan 2e153037af Changing stripe verification to properly work with clients. 2015-01-11 09:41:27 -08:00
Eric Lambert a11fcb022d merge upstream 2014-11-09 08:54:56 -08:00
Eric Lambert 1b1d18733f add fragment validation 2014-11-09 08:52:11 -08:00
Eric Lambert 2798a65d81 Make include of config_liberasurecode.h conditonal. Have been expermenting with using cmake to build the project and cmake does not generate the config_liberasurecode.h file. 2014-11-08 08:31:34 -08:00
Tushar Gohad 8d7a9c24b7 README.md edited online with Bitbucket 2014-10-28 02:14:37 +00:00
Tushar Gohad ffe8df39c4 README.md edited online with Bitbucket 2014-10-28 02:12:29 +00:00
Eric Lambert 413c6605dd fixed mem leaks in unit tests 2014-10-10 21:09:55 -07:00
Eric Lambert 0ec68bebb7 fixed mem leaks in rs_vand and xor backend 2014-10-10 21:08:48 -07:00
Eric Lambert 1a34dd4147 Fixed memory leak: freeing jerasure_rs_cauchy_descriptor was not completely freeing the schedule array 2014-10-10 14:25:14 -07:00
Eric Lambert e22f3807e2 provide a configure flag to disable -Werror at compile time 2014-10-09 07:15:16 -07:00
Eric Lambert f552b70f71 formatting fix 2014-10-06 16:07:34 -07:00
Eric Lambert 8af7296205 reorder add_fragment_metadata arg list to be consistent with rest of code 2014-10-06 07:36:46 -07:00
Eric Lambert 62fba6ba49 add backend_id and backend_version to fragment metadata 2014-10-03 17:08:05 -07:00
Eric Lambert 581afe3199 libec version is now stored in fragment header 2014-10-02 22:31:34 -04:00
Eric Lambert dbacee6f98 backends now support verion checks 2014-10-02 11:25:28 -04:00
Eric Lambert 22b17d139a Fixed macro definition of version elements 2014-10-01 10:20:55 -04:00
Eric Lambert 047bcf5330 Merged in malloc_check (pull request #6)
Small fix to check the return code when malloc'ing temporary buffers.
2014-10-01 10:15:42 -04:00
Mark Storer 42870dff1f Small fix to check the return code when malloc'ing temporary buffers. 2014-10-01 10:05:59 -04:00