summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorRuslan Aliev <raliev@mirantis.com>2016-12-21 17:54:12 +0400
committerRuslan Aliev <raliev@mirantis.com>2016-12-22 18:01:06 +0400
commita853067a13abbc32da4058ba843fd8404042bd44 (patch)
tree3c7b9172d0f67c3cc6ba98183db025e2b3f84b14
parent20256d7277edd145ee5687f1619733431d852e1a (diff)
Block based backup support (rsync)
This is a spec for block based support implementation. Using rsync as approach. Change-Id: I38e73ac970b35bceddfcc382b6e533b4629211b6 Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Notes
Notes (review): Code-Review+2: Saad Zaher <eng.szaher@gmail.com> Code-Review+1: Vitaliy Nogin <vnogin@mirantis.com> Workflow+1: Pierre Mathieu <pierre-arthur.mathieu@hpe.com> Verified+2: Jenkins Submitted-by: Jenkins Submitted-at: Tue, 07 Feb 2017 10:12:51 +0000 Reviewed-on: https://review.openstack.org/413638 Project: openstack/freezer-specs Branch: refs/heads/master
-rw-r--r--README.rst1
-rw-r--r--requirements.txt10
-rw-r--r--specs/ocata/approved/block_based_backup_support.rst229
3 files changed, 235 insertions, 5 deletions
diff --git a/README.rst b/README.rst
index 94c6ba3..509d880 100644
--- a/README.rst
+++ b/README.rst
@@ -4,6 +4,7 @@ Team and repository tags
4 4
5.. image:: http://governance.openstack.org/badges/freezer-specs.svg 5.. image:: http://governance.openstack.org/badges/freezer-specs.svg
6 :target: http://governance.openstack.org/reference/tags/index.html 6 :target: http://governance.openstack.org/reference/tags/index.html
7 :remote:
7 8
8.. Change things from this point on 9.. Change things from this point on
9 10
diff --git a/requirements.txt b/requirements.txt
index 5aaaa59..3d8b678 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,6 @@
1oslosphinx 1oslosphinx>=4.7.0 # Apache-2.0
2pbr>=1.6 # Apache-2.0 2pbr>=1.8 # Apache-2.0
3sphinx>=1.1.2,<1.2 3sphinx>=1.2.1,!=1.3b1,<1.4 # BSD
4testrepository>=0.0.18 4testrepository>=0.0.18 # Apache-2.0/BSD
5testtools>=0.9.34 5testtools>=1.4.0 # MIT
6yasfb>=0.5.1 6yasfb>=0.5.1
diff --git a/specs/ocata/approved/block_based_backup_support.rst b/specs/ocata/approved/block_based_backup_support.rst
new file mode 100644
index 0000000..05002d8
--- /dev/null
+++ b/specs/ocata/approved/block_based_backup_support.rst
@@ -0,0 +1,229 @@
1..
2 This work is licensed under a Creative Commons Attribution 3.0 Unported
3 License.
4
5 http://creativecommons.org/licenses/by/3.0/legalcode
6
7==================================
8Block based backup support (rsync)
9==================================
10
11https://blueprints.launchpad.net/freezer/+spec/rsync
12
13Taking advantage of the rsync to provide a possibility to create
14space/bandwidth efficient backups.
15
16Problem description
17===================
18
19Currently Freezer checks only ctime and mtime inode information
20to verify if files are changed or not (tar functionality). While
21this approach gives speed (time efficient), it is not bandwidth
22and storage efficient. Freezer needs to support both rsync and tar
23approach to execute incremental backups and restore.
24
25Since Freezer will provide two options for incremental backups, it
26would be more convenient to choose the best approach to backup data
27in accordance with each particular case (more speed or storage/bandwidth
28efficient).
29
30Use Cases
31---------
32
33* For developers, this change will not create negative impacts because
34 this code will be gracefully bundled in Freezer engine API and
35 will not cause any major changes in Freezer architecture.
36
37* For Deployers there is no need to install any additional components,
38 Freezer will use it's own implementation of rsync algorithm
39 (written in Python).
40
41* For End User it would be less difficult to select more efficient
42 option for create backups based on dataset (e.g. few big files or a lot of
43 small files) for backup and speed/storage/bandwidth requirements,
44 since Freezer would support both rsync and tar approaches.
45
46Proposed change
47===============
48
49Implementing the new engine classes for rsync (as well as for tar).
50Providing new engine (-e) choice in config.
51
52For this type of backup will be created following metadata structure:
53
54files_meta = {
55 'files': {},
56 'directories': {},
57 'meta': {
58 'broken_links_tot': '',
59 'total_files': '',
60 'total_directories': '',
61 'backup_size_on_disk': '',
62 'backup_size_uncompressed': '',
63 'backup_size_compressed': '',
64 'platform': sys.platform
65 },
66 'abs_backup_path': os.getcwd(),
67 'broken_links': [],
68 'rsync_struct_ver': RSYNC_DATA_STRUCT_VERSION,
69 'rsync_block_size': RSYNC_BLOCK_SIZE}
70
71
72file_meta = {'inode': {
73 'inumber': os_stat.st_ino,
74 'nlink': os_stat.st_nlink,
75 'mode': file_mode,
76 'uid': os_stat.st_uid,
77 'gid': os_stat.st_gid,
78 'size': os_stat.st_size,
79 'devmajor': os.major(dev),
80 'devminor': os.minor(dev),
81 'mtime': mtime,
82 'ctime': ctime,
83 'uname': uname,
84 'gname': gname,
85 'ftype': file_type,
86 'lname': lname,
87 'rsync_block_size': rsync_block_size,
88 'file_status: status
89 }
90 }
91
92Current version of implementation you always can find here [1].
93
94Alternatives
95------------
96
97Because of the flexibility, speed, and scriptability of rsync, it has
98become a standard Linux utility, included in all popular Linux distributions.
99It has been ported to Windows (via Cygwin, Grsync, or SFU), FreeBSD, NetBSD,
100OpenBSD, and Mac OS. De facto, rsync is the default fallback for most data
101transfers. It has a clear algorithm written for 20 years ago and different
102variations (e.g. acrosync, zsync, etc). librsync is used by Dropbox.
103
104Using other alternative (like bbcp or lftp) would not be more effective
105or portable solution.
106
107Data model impact
108-----------------
109
110Changes in data model has already described in oslo.db migration document.
111Actions entity should contain 'engine' field for performing appropriate action
112using particular type of engine (tar, rsync or openstack).
113
114From new relational database schema:
115
116Actions
117 action_id (uuid) [p_key]
118 resource (varchar)
119 type (varchar)
120 name (varchar)
121 application (varchar)
122 engine (varchar) <-- Require this
123 snapshot (varchar)
124 storage (varchar)
125 global_options (JSON)
126 application_options (JSON)
127 storage_options (JSON)
128 snapshot_options (JSON)
129 engine_options (JSON)
130
131
132REST API impact
133---------------
134
135None.
136
137Security impact
138---------------
139
140None.
141
142Notifications impact
143--------------------
144
145There are no special logs will be added, just some info messages about
146start/stop backup process, backup metrics, etc.
147
148Other end user impact
149---------------------
150
151* There are no additional changes to python-freezerclient CLI. To choice
152 appropriate engine for action, end user should specify 'engine' field
153 in provided JSON configuration in case of creating or updating action.
154
155* freezer-web-ui should provide additional 'engine' field in 'Action
156 Configuration' window. It has to be drop-down list with values 'tar',
157 'rsync' or 'openstack'.
158
159Performance Impact
160------------------
161
162None.
163
164Other deployer impact
165---------------------
166
167Will be added new choice to freezer-agent -e (engine) option - 'rsync'.
168
169Developer impact
170----------------
171
172None.
173
174Implementation
175==============
176
177Assignee(s)
178-----------
179
180Primary assignee:
181 Ruslan Aliev (raliev) <raliev@mirantis.com>
182
183Other contributors:
184 Fausto Marzi (daemontool) <fausto.marzi@ericsson.com>
185
186Work Items
187----------
188
189* implementing the new engine (rsync)
190
191* bundling this engine to freezer code (API calls) and mechanism
192 for using this engine ('-e rsync' option)
193
194* implementing the new database schema for actions (oslo.db migration)
195
196* updating freezer-web-ui 'Action Configuration' window
197
198* updating documentation
199
200Dependencies
201============
202
203* This spec depends on Freezer oslo.db migration [2].
204
205* Pluggable engines described here [3].
206
207* There are no additional library dependencies.
208
209Testing
210=======
211
212There is a question - do we actually need separate tempest test
213for this change or we can be satisfied with existing one?
214
215Documentation Impact
216====================
217
218* freezer README doc
219
220* freezer-api README doc
221
222* freezer-web-ui README doc
223
224References
225==========
226
227.. [1] https://review.openstack.org/#/c/409796/
228.. [2] https://etherpad.openstack.org/p/freezer_mysql_migration
229.. [3] https://etherpad.openstack.org/p/freezer_new_archi