As per IRC conversation, backup should be image so it can mean backup
or restore since they will basically be inverses of each other.
Change-Id: Ic0d14cc7bfa9071bf45d70b0731e02c106b58277
This code will be the basis of writing out the data to the backing
storage with the initial driver being on disk storage.
Additionally move compression/encryption code to storage rather than
backup module
./backup.py --backup /dev/loop0 --manifest mani --location /path/to/store/backup/
Change-Id: I22d291ea3ac03b4d5ddb447915c972db4cdc195d
Updates backup.py and adds dump_manifest.py that will print out all
hashes from the underlying blocks. This can be replicated with a
simple python script as follows:
# dump_disk.py
import hashlib
with open('/dev/loop0', 'rb') as f:
while True:
data = f.read(4*1024**2)
if not data:
break
print(hashlib.sha1(data).hexdigest())
python dump_disk.py | grep -v 2bccbd2f38f15c13eb7d5a89fd9d85f595e23bc3
NOTE: 2bccbd2f38f15c13eb7d5a89fd9d85f595e23bc3 == segment of all zero
To perform backup and dump info from manifest (which will match
the output of the above script):
tools/backup.py --backup /dev/loop0 --manifest mani
tools/dump_manifest.py --manifest mani
The backup works on both files and block devices with the raw driver.
Further drivers will be implemented (such as qcow2 and rbd) as time
goes on.
Adjust sectors to equal number of bytes on size instead.
Change-Id: I976f02a27bc13b5774a6088799ca61f65ec04f14
I don't expect these to be plugable, mostly just wrapping my head
around stevedores codebase.
Of note, because of the stevedore-ness, we _must_ install ekko to use
it. No longer will it work in the local dir, it must be installed on
the system (or in a venv)
Change-Id: I5037f6877e623614ef7ecf991b0009f49824ec0a
Upon debugging the difference between the storage with sqlite and OSDK
I discovered the biggest difference was the backupset_id storage.
Converting this back to my original design of a index reference has
brought the two formats within ~5% of each other in size! Additionally
this change creates a 40% reduction in the manifest size.
A 2^24 objects backup will now only take ~500MB instead of ~800MB
Now SQLite is still about 50% slower than OSDK, but since it is about
the same size the slowdown isn't even something worth mentioning. The
exact times are 55s of OSDK and 90s for sqlite at 2^24 objects.
Change-Id: If63380f312691e34c92f8a95fff5e724a4f5a465
Now we can easily test performance and size differences between OSDK
and sqlite. The expectation here is we will use sqlite going forward,
but the extra validation for speed and size will be comforting.
Adds a new --driver option to generate_manifest.py to select between
osdk and sqlite
Change-Id: I59751080b3169d3d7bfd1f60a73edabe307d7fbf
Use class inheritance for driver in case we need something other than
sqlite as time goes on.
SQLite is only 10% larger than the custom format used with osdk. When
compressed the SQLite manifest only fractionally larger than osdk. The
time it takes to generate and process is about 2x longer. However, the
benefit we get from using common contructs (sqlite) and tracking and
changing the manifest schema is enough for me to think this is right
way to proceed.
To generate a manifest with 2^24 objects (64TB disk) run:
tools/generate_manifest.py --backupsize 64000 --manifest /pathtosave/manifest
This will generate a 809MB file
Co-Authored-By: Sam Yaple <sam@yaple.net>
Change-Id: Ic431bfa52b6fcaeb1c6a64cf270cbb36c496335e
This reads the manifest. Its more based on osdk.py that I initially
thought it would be. But hopefully I have cleaned it up a good bit.
Add tool to generate manifest
To generate a manifest with 2^24 objects (64TB for 4MB objects) run:
tools/generate_manifest.py --backupsize 64000 --manifest /pathtosave/manifest
Change-Id: I09a8fb64c0d4b8e4b4b436cca44b57bec87116cd
These tools are not meant to be used as-is They need a full rework.
They are more reference material than anything else right now. I
also had to do some awful stuff to them to get them to pass the gate
in a hurry, code looks bad because of it.
I am cleaning them up as I go.
We will remove these in the future once they are no longer useful to
reference any longer.
Change-Id: I8f13ef282a2a05e72c53a0c307832a46db18d2b4
SamYaple