370 lines
21 KiB
XML
370 lines
21 KiB
XML
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE section [
|
|
<!-- Some useful entities borrowed from HTML -->
|
|
<!ENTITY ndash "–">
|
|
<!ENTITY mdash "—">
|
|
<!ENTITY hellip "…">
|
|
<!ENTITY nbsp " ">
|
|
<!-- Useful for describing APIs -->
|
|
<!ENTITY COPY '<command xmlns="http://docbook.org/ns/docbook">COPY</command>'>
|
|
<!ENTITY GET '<command xmlns="http://docbook.org/ns/docbook">GET</command>'>
|
|
<!ENTITY HEAD '<command xmlns="http://docbook.org/ns/docbook">HEAD</command>'>
|
|
<!ENTITY PUT '<command xmlns="http://docbook.org/ns/docbook">PUT</command>'>
|
|
<!ENTITY POST '<command xmlns="http://docbook.org/ns/docbook">POST</command>'>
|
|
<!ENTITY DELETE '<command xmlns="http://docbook.org/ns/docbook">DELETE</command>'>
|
|
]>
|
|
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
|
|
xml:id="large-object-creation">
|
|
<title>Large objects</title>
|
|
<para>By default, the content of an object cannot be greater than
|
|
5 GB. However, you can use segment objects and manifest
|
|
objects to store more content.</para>
|
|
<section xml:id="segment-objects">
|
|
<title>Segment objects</title>
|
|
<para>You can divide your content into segments, and upload
|
|
each segment into its own segment object. Segment objects
|
|
do not have any special features. You create, update,
|
|
download, and delete segment objects just as you would
|
|
normal objects.</para>
|
|
</section>
|
|
<section xml:id="manifest-objects">
|
|
<title>Manifest objects</title>
|
|
<para>A manifest object points to segment objects. When you
|
|
download a manifest object, Object Storage concatenates
|
|
the contents of the segment objects and returns this in
|
|
the response body of the request.</para>
|
|
<para>This behavior extends to the response headers returned
|
|
by &GET; and &HEAD; requests. The
|
|
<literal>Content-Length</literal> response header
|
|
value is the total size of all segment objects. Object
|
|
Storage calculates the <literal>ETag</literal> response
|
|
header value by taking the <literal>ETag</literal> value
|
|
of each segment, concatenating them together, and
|
|
returning the MD5 checksum of the result.</para>
|
|
<note>
|
|
<para>If you make a © request by using a manifest
|
|
object as the source, the new object is a normal, and
|
|
not a segment, object. If the total size of the source
|
|
segment objects exceeds 5 GB, the © request
|
|
fails. However, you can make a duplicate of the
|
|
manifest object and this new object can be larger than
|
|
5 GB.</para>
|
|
</note>
|
|
<para>The manifest object types are:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><emphasis role="bold">Static large
|
|
objects</emphasis>. The manifest object
|
|
content is an ordered list of the names of the
|
|
segment objects in JSON format.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><emphasis role="bold">Dynamic large
|
|
objects</emphasis>. The manifest object has no
|
|
content.</para>
|
|
<para>However, it has
|
|
<literal>X-Object-Manifest</literal> metadata
|
|
header. The value of this header is
|
|
<literal><container>/<prefix></literal>,
|
|
where <literal><container></literal> is the
|
|
name of the container where the segment objects
|
|
are stored, and <literal><prefix></literal> is
|
|
a string that all segment objects have in
|
|
common.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>While both types of manifest objects have similar
|
|
behavior, the following table describes their
|
|
differences:</para>
|
|
<table rules="all">
|
|
<caption>Static and dynamic large objects</caption>
|
|
<thead>
|
|
<tr>
|
|
<th>Object type</th>
|
|
<th>End-to-end integrity</th>
|
|
<th>Upload order</th>
|
|
<th>Removal or addition of segment objects</th>
|
|
<th>Segment object size and number</th>
|
|
<th>Segment object container name</th>
|
|
<th>Manifest Object Metadata</th>
|
|
<th>Making a copy of the manifest object</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td><para>Static large object</para></td>
|
|
<td><para>Assured. The list of segments includes
|
|
the MD5 checksum (<literal>ETag</literal>)
|
|
of each segment. You cannot upload the
|
|
manifest object if the
|
|
<literal>ETag</literal> in the list
|
|
differs from the segment object already
|
|
uploaded. If a segment is somehow lost, an
|
|
attempt to download the manifest object
|
|
results in an error.</para></td>
|
|
<td><para>The segment objects must be uploaded
|
|
before the manifest object.</para></td>
|
|
<td><para>You cannot add or remove segment objects
|
|
from the manifest. However, you can create
|
|
a completely new manifest object of the
|
|
same name with a different manifest
|
|
list.</para></td>
|
|
<td><para>Segment objects must be at least 1 MB in
|
|
size (by default). The final segment
|
|
object can be any size. At most 1000
|
|
segments are supported (by
|
|
default).</para></td>
|
|
<td><para>The manifest list includes the container
|
|
name of each object. Segment objects can
|
|
be in different containers.</para></td>
|
|
<td><para>The object has
|
|
<literal>X-Static-Large-Object</literal>
|
|
set to <literal>true</literal>. You do not
|
|
set this metadata directly. Instead the
|
|
system sets it when you &PUT; a static
|
|
manifest object.</para></td>
|
|
<td><para/></td>
|
|
</tr>
|
|
<tr>
|
|
<td><para>Dynamic large object</para></td>
|
|
<td><para>Not guaranteed. The eventual consistency
|
|
model means that although you have
|
|
uploaded a segment object, it might not
|
|
appear in the container listing until
|
|
later. If you download the manifest before
|
|
it appears in the container, it does not
|
|
form part of the content returned in
|
|
response to a &GET; request.</para></td>
|
|
<td><para>You can upload manifest and segment
|
|
objects in any order. You are recommended
|
|
to upload the manifest object after the
|
|
segments in case a premature download of
|
|
the manifest occurs. However, this is not
|
|
enforced.</para></td>
|
|
<td><para>You can upload new segment objects or
|
|
remove existing segments. The names must
|
|
simply match the
|
|
<literal><prefix></literal>
|
|
supplied in
|
|
<literal>X-Object-Manifest</literal>.</para></td>
|
|
<td><para>Segment objects can be of any
|
|
size.</para></td>
|
|
<td><para>All segment objects must be in the same
|
|
container</para></td>
|
|
<td><para>The <literal>X-Object-Manifest</literal>
|
|
value is the
|
|
<literal><container>/<prefix></literal>
|
|
indicating where the segment objects are
|
|
located. You supply this request header in
|
|
the &PUT; operation</para></td>
|
|
<td><para>The © operation does not create a
|
|
manifest object. To duplicate a manifest
|
|
object, use the &GET; operation to read
|
|
the value of
|
|
<literal>X-Object-Manifest</literal>
|
|
and use this value in the
|
|
<literal>X-Object-Manifest</literal>
|
|
request header in a &PUT; operation. This
|
|
creates a new manifest object that shares
|
|
the same set of segment objects as the
|
|
original manifest object.</para></td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
<section xml:id="dynamic-large-object-creation">
|
|
<title>Dynamic large objects</title>
|
|
<para>You must segment objects that are larger than 5 GB
|
|
before you can upload them. You then upload the
|
|
segment objects like you would any other object and
|
|
create a dynamic large manifest object. The manifest
|
|
object tells Object Storage how to find the segment
|
|
objects that comprise the large object. The segments
|
|
remain individually addressable, but retrieving the
|
|
manifest object streams all the segments concatenated.
|
|
There is no limit to the number of segments that can
|
|
be a part of a single large object.</para>
|
|
<para>To ensure the download works correctly, you must
|
|
upload all the object segments to the same container
|
|
and ensure that each object name is prefixed in such a
|
|
way that it sorts in the order in which it should be
|
|
concatenated. You also create and upload a manifest
|
|
file. The manifest file is a zero-byte file with the
|
|
extra <literal>X-Object-Manifest</literal>
|
|
<code><container>/<prefix></code> header,
|
|
where <code><container></code> is the container
|
|
the object segments are in and
|
|
<code><prefix></code> is the common prefix
|
|
for all the segments. You must UTF-8-encode and then
|
|
URL-encode the container and common prefix in the
|
|
<literal>X-Object-Manifest</literal>
|
|
header.</para>
|
|
<para>It is best to upload all the segments first and then
|
|
create or update the manifest. With this method, the
|
|
full object is not available for downloading until the
|
|
upload is complete. Also, you can upload a new set of
|
|
segments to a second location and then update the
|
|
manifest to point to this new location. During the
|
|
upload of the new segments, the original manifest is
|
|
still available to download the first set of
|
|
segments.</para>
|
|
<example>
|
|
<title>Upload segment of large object request:
|
|
HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/large-object-upload-segment-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<example>
|
|
<title>Upload segment of large object response:
|
|
HTTP</title>
|
|
<literallayout class="monospaced">s</literallayout>
|
|
</example>
|
|
<para>No response body is returned. A status code of
|
|
<returnvalue>2<replaceable>nn</replaceable></returnvalue>
|
|
(between 200 and 299, inclusive) indicates a
|
|
successful write; status <errorcode>411</errorcode>
|
|
<errortext>Length Required</errortext> denotes a
|
|
missing <literal>Content-Length</literal> or
|
|
<literal>Content-Type</literal> header in the request.
|
|
If the MD5 checksum of the data written to the storage
|
|
system does NOT match the (optionally) supplied ETag
|
|
value, a <errorcode>422</errorcode>
|
|
<errortext>Unprocessable Entity</errortext> response
|
|
is returned.</para>
|
|
<para>You can continue uploading segments like this
|
|
example shows, prior to uploading the manifest.</para>
|
|
<example>
|
|
<title>Upload next segment of large object request:
|
|
HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/large-object-upload-next-segment-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<example>
|
|
<title>Upload next segment of large object response:
|
|
HTTP</title>
|
|
<literallayout class="monospaced">w</literallayout>
|
|
</example>
|
|
<para>Next, upload the manifest you created that indicates
|
|
the container the object segments reside within. Note
|
|
that uploading additional segments after the manifest
|
|
is created causes the concatenated object to be that
|
|
much larger but you do not need to recreate the
|
|
manifest file for subsequent additional
|
|
segments.</para>
|
|
<example>
|
|
<title>Upload manifest request: HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/upload-manifest-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<example>
|
|
<title>Upload manifest response: HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/upload-manifest-resp.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<para>The response's <literal>Content-Type</literal> for a
|
|
&GET; or &HEAD; on the manifest is the same as the
|
|
<literal>Content-Type</literal> set during the
|
|
&PUT; request that created the manifest. You can
|
|
easily change the <literal>Content-Type</literal> by
|
|
reissuing the &PUT; request.</para>
|
|
</section>
|
|
<section xml:id="static-large-objects">
|
|
<title>Static large objects</title>
|
|
<procedure>
|
|
<title>To create a static large object</title>
|
|
<step>
|
|
<para>Divide your content into pieces and create
|
|
(upload) a segment object to contain each
|
|
piece. You must record the <literal>ETag</literal>
|
|
response header returned by the&PUT;
|
|
operation. Alternatively, you can calculate
|
|
the MD5 checksum of the segment prior to
|
|
uploading and include this in the
|
|
<literal>ETag</literal> request header. This
|
|
ensures that the upload cannot corrupt your
|
|
data.</para>
|
|
</step>
|
|
<step>
|
|
<para>List the name of each segment object along
|
|
with its size and MD5 checksum in order.
|
|
Create a manifest object. You indicate that
|
|
this is a manifest object by including the
|
|
<parameter>?multipart-manifest=put</parameter> query
|
|
string at the end of the manifest object
|
|
name.</para>
|
|
</step>
|
|
</procedure>
|
|
<para>The body of the &PUT; request on the manifest object
|
|
comprises a json list, where each element contains the
|
|
following attributes:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><code>path</code>. The container and object
|
|
name in this format:
|
|
<code><container-name>/<object-name></code></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>etag</code>. The MD5 checksum of the
|
|
content of the segment object. This value must
|
|
match the <literal>ETag</literal> of that
|
|
object.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>size_bytes</code>. The size of the
|
|
segment object. This value must match the
|
|
<literal>Content-Length</literal> of that
|
|
object.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<example>
|
|
<title>Static large object manifest list</title>
|
|
<para>This example shows three segment objects. You
|
|
can use several containers and the object names do
|
|
not have to conform to a specific pattern, in
|
|
contrast to dynamic large objects.</para>
|
|
<literallayout class="monospaced"><xi:include href="samples/slo-manifest-example.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<para>The <literal>Content-Length</literal> request header must
|
|
contain the length of the json content. Not the length
|
|
of the segment objects. However, after the &PUT;
|
|
operation completes, the <literal>Content-Length</literal>
|
|
metadata is set to the total length of all the object
|
|
segments. A similar situation applies to the
|
|
<literal>ETag</literal>. If used in the &PUT; operation,
|
|
it must contain the MD5 checksum of the json content.
|
|
The <literal>ETag</literal> metadata value is then set to be
|
|
the MD5 checksum of the concatenated <literal>ETag</literal>
|
|
values of the object segments. You can also set the
|
|
<literal>Content-Type</literal> request header and
|
|
custom object metadata.</para>
|
|
<para>When the &PUT; operation sees the
|
|
<parameter>?multipart-manifest=put</parameter>
|
|
query parameter, it reads the request body and verifies
|
|
that each segment object exists and that the sizes and
|
|
ETags match. If there is a mismatch, the
|
|
&PUT;operation fails.</para>
|
|
<para>If everything matches, the manifest object is
|
|
created. The <literal>X-Static-Large-Object</literal>
|
|
metadata is set to <literal>true</literal> indicating
|
|
that this is a static object manifest.</para>
|
|
<para>Normally when you perform a &GET; operation on the
|
|
manifest object, the response body contains the
|
|
concatenated content of the segment objects. To
|
|
download the manifest list, use the
|
|
<parameter>?multipart-manifest=get</parameter> query parameter.
|
|
The resulting list is not formatted the same as the
|
|
manifest you originally used in the &PUT;
|
|
operation.</para>
|
|
<para>If you use the &DELETE; operation on a manifest
|
|
object, the manifest object is deleted. The segment
|
|
objects are not affected. However, if you add the
|
|
<parameter>?multipart-manifest=delete</parameter> query parameter,
|
|
the segment objects are deleted and if all are
|
|
successfully deleted, the manifest object is also
|
|
deleted.</para>
|
|
<para>To change the manifest, use a &PUT; operation with
|
|
the <parameter>?multipart-manifest=put</parameter>
|
|
query parameter. This request creates a
|
|
new manifest object. You can
|
|
also update the object metadata in the usual
|
|
way.</para>
|
|
</section>
|
|
</section>
|
|
</section>
|