Commit Graph

38 Commits

Author SHA1 Message Date
OpenDev Sysadmins 6f130aada7 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:51:41 +00:00
Jenkins 7adf88d6b1 Merge "Updated from global requirements" 2016-12-03 02:12:16 +00:00
OpenStack Proposal Bot c33e9b037d Updated from global requirements
Change-Id: I344a323193e5f103bc49c8dcf0ddfa0d827ab16c
2016-11-09 04:15:04 +00:00
Andreas Jaeger 20b78f6c78 Cleanup tox.ini, enable constraints
Remove old and unused constraints environments from tox.ini. Those
have never been used. Use standard environments as default list.

Enable use of constraints for all jobs now.

For more about constraints see:
http://lists.openstack.org/pipermail/openstack-dev/2016-August/101474.html

Change-Id: I9b4c567e8bcf4abd52e1d499f7ec0b22b034c815
2016-08-26 16:39:10 +02:00
SamYaple 34942d3aeb Revert "Add bare bones ekko-api"
This implementation of the API is no longer tested or supported. We
will be using a new one based on flask

This reverts commit a5f99317ac.

Change-Id: I7e1a3f0f341685b2455a4cdfeb0b13b7fba5bfc0
2016-06-15 20:44:51 +00:00
OpenStack Proposal Bot e48a55ba0e Updated from global requirements
Change-Id: I715d21cc460da5ef1ed30f55c8c4d8a844c5897f
2016-06-03 18:12:04 +00:00
OpenStack Proposal Bot 7d5e4a941e Updated from global requirements
Change-Id: I669d71d6a3e9e853888d404295816da641387120
2016-05-10 00:43:33 +00:00
OpenStack Proposal Bot 7e079e4ebe Updated from global requirements
Change-Id: I29b93e260ea679157289ccfa6d0db56030a06526
2016-04-07 17:07:40 +00:00
OpenStack Proposal Bot 487a755676 Updated from global requirements
Change-Id: I98eb3df58871a04e5eabff36688e7d93d2874932
2016-02-26 01:47:11 +00:00
OpenStack Proposal Bot c3cdb4410b Updated from global requirements
Change-Id: I6e5cfde8ce740e93b6d6b6a2c62fd349d15004ff
2016-02-24 08:31:23 +00:00
OpenStack Proposal Bot 52fea3317d Updated from global requirements
Change-Id: Ifc07282a7c1c3141c3c41eafcf7a3dc22fb37e96
2016-02-20 21:54:42 +00:00
Andreas Jaeger b9ba3d9e65 Make pep8 jobs default
According to the PTI (=Python Test Interface,
http://governance.openstack.org/reference/cti/python_cti.html), pep8
is the interface for codestyle checks. Therefore add pep8
as default environment, linters is not needed.

This change will be followed by a change to project-config to use pep8
for testing in the gate.

Needed-By: I8cecb75cdccc73b7fd26e364e11dd91164b20832
Change-Id: I5738e7cc10a9e5b643844f398bd7b5a64f76b40d
2016-02-10 22:23:51 +01:00
OpenStack Proposal Bot 2a5cd5a228 Updated from global requirements
Change-Id: I5822d0d67cbb833af0f38344427b756d649385f2
2016-02-08 02:38:26 +00:00
Paul Bourke a5f99317ac Add bare bones ekko-api
Foundation for ekko-api, ported over from the Ironic project.

The goal here was to copy enough to give a solid foundation that
conforms to the "OpenStack way", without a lot of extra frills that we
either may not need right away, or makes it harder see what's going on
starting out.

After installation the service can be started via 'ekko-api', which
listens on port 6800 by default. This was chosen from the list of
unassigned ports at
http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt

There are still plenty of TODO's, primarily keystone integration and
config file generation. Foundations for these are here but completely
untested as of this commit.

Change-Id: If136a3bb66949ef710d741eaf3691f36f7b60692
2016-02-08 01:07:34 +00:00
Jenkins 4bb6a26ad8 Merge "Add base for compression and encryption" 2016-01-25 15:19:26 +00:00
Jenkins 7c3aeada4b Merge "Rename backup to image" 2016-01-25 15:18:15 +00:00
Jenkins 361c190a4c Merge "Add storage code" 2016-01-25 15:18:01 +00:00
SamYaple 622140b2ba Add base for compression and encryption
nopp, zlib and lzma compression implemented, but not configurable yet

noop encryption implemented

Change-Id: Id649974c3b0bb3852746fd7cdbf578768d7090c7
2016-01-20 23:23:24 +00:00
SamYaple fa9d060094 Rename backup to image
As per IRC conversation, backup should be image so it can mean backup
or restore since they will basically be inverses of each other.

Change-Id: Ic0d14cc7bfa9071bf45d70b0731e02c106b58277
2016-01-19 19:23:04 +00:00
SamYaple 2e389f1ddb Add storage code
This code will be the basis of writing out the data to the backing
storage with the initial driver being on disk storage.

Additionally move compression/encryption code to storage rather than
backup module

./backup.py --backup /dev/loop0 --manifest mani --location /path/to/store/backup/

Change-Id: I22d291ea3ac03b4d5ddb447915c972db4cdc195d
2016-01-19 19:14:23 +00:00
Paul Bourke 13b9dbfcdd Update .gitignore
Source
https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore

with some modifications.

Change-Id: I9eb17bba855381d475a104d222d1c8a803671fb5
2016-01-19 17:03:10 +00:00
OpenStack Proposal Bot a23dc07f41 Updated from global requirements
Change-Id: I1e126516299c27ca6ae4d3a3368a7a45ecdcbd05
2016-01-18 22:40:21 +00:00
Jenkins d687daf5d0 Merge "Update shebang in tools to allow virtualenv" 2016-01-18 20:14:27 +00:00
Jenkins 3e78319bc0 Merge "Add initial backup code" 2016-01-18 20:14:21 +00:00
Paul Bourke 9f9b96ce56 Update shebang in tools to allow virtualenv
Change-Id: Ie3cfc5796b401a683521afc2f872106006825b2e
2016-01-18 17:18:56 +00:00
SamYaple 401c05413c Add initial backup code
Updates backup.py and adds dump_manifest.py that will print out all
hashes from the underlying blocks. This can be replicated with a
simple python script as follows:

    # dump_disk.py
    import hashlib
    with open('/dev/loop0', 'rb') as f:
        while True:
            data = f.read(4*1024**2)
            if not data:
                break
            print(hashlib.sha1(data).hexdigest())

    python dump_disk.py | grep -v 2bccbd2f38f15c13eb7d5a89fd9d85f595e23bc3

NOTE: 2bccbd2f38f15c13eb7d5a89fd9d85f595e23bc3 == segment of all zero

To perform backup and dump info from manifest (which will match
the output of the above script):

    tools/backup.py --backup /dev/loop0 --manifest mani
    tools/dump_manifest.py --manifest mani

The backup works on both files and block devices with the raw driver.
Further drivers will be implemented (such as qcow2 and rbd) as time
goes on.

Adjust sectors to equal number of bytes on size instead.

Change-Id: I976f02a27bc13b5774a6088799ca61f65ec04f14
2016-01-16 05:54:23 +00:00
OpenStack Proposal Bot 99945f6d83 Updated from global requirements
Change-Id: I7e98b8b2f9385273b29f84d615ce00921f120576
2016-01-16 03:26:06 +00:00
OpenStack Proposal Bot 51bfcd893e Updated from global requirements
Change-Id: Iae29b222a37fff65389655ae58f35294cbdba8ca
2016-01-15 08:23:50 +00:00
SamYaple e2ee32ed82 stevedoreize the drivers
I don't expect these to be plugable, mostly just wrapping my head
around stevedores codebase.

Of note, because of the stevedore-ness, we _must_ install ekko to use
it. No longer will it work in the local dir, it must be installed on
the system (or in a venv)

Change-Id: I5037f6877e623614ef7ecf991b0009f49824ec0a
2016-01-13 21:31:25 +00:00
SamYaple 94bf4209b2 Add manifest read functions for sqlite
Change-Id: I73855f446316ddaf31a492c90a6d3b444d77683c
2016-01-11 18:15:17 +00:00
SamYaple 0f62c546d1 Adjust the way backupsets are stored in manifest
Upon debugging the difference between the storage with sqlite and OSDK
I discovered the biggest difference was the backupset_id storage.

Converting this back to my original design of a index reference has
brought the two formats within ~5% of each other in size! Additionally
this change creates a 40% reduction in the manifest size.

A 2^24 objects backup will now only take ~500MB instead of ~800MB

Now SQLite is still about 50% slower than OSDK, but since it is about
the same size the slowdown isn't even something worth mentioning. The
exact times are 55s of OSDK and 90s for sqlite at 2^24 objects.

Change-Id: If63380f312691e34c92f8a95fff5e724a4f5a465
2016-01-11 18:15:17 +00:00
SamYaple d31e2ebef5 Add OSDK driver for manifest
Now we can easily test performance and size differences between OSDK
and sqlite. The expectation here is we will use sqlite going forward,
but the extra validation for speed and size will be comforting.

Adds a new --driver option to generate_manifest.py to select between
osdk and sqlite

Change-Id: I59751080b3169d3d7bfd1f60a73edabe307d7fbf
2016-01-11 18:15:17 +00:00
Michal Jastrzebski a9d13fc32f Create manifest and implement sqlite driver
Use class inheritance for driver in case we need something other than
sqlite as time goes on.

SQLite is only 10% larger than the custom format used with osdk. When
compressed the SQLite manifest only fractionally larger than osdk. The
time it takes to generate and process is about 2x longer. However, the
benefit we get from using common contructs (sqlite) and tracking and
changing the manifest schema is enough for me to think this is right
way to proceed.

To generate a manifest with 2^24 objects (64TB disk) run:
tools/generate_manifest.py --backupsize 64000 --manifest /pathtosave/manifest

This will generate a 809MB file

Co-Authored-By: Sam Yaple <sam@yaple.net>
Change-Id: Ic431bfa52b6fcaeb1c6a64cf270cbb36c496335e
2016-01-11 18:15:17 +00:00
SamYaple 7b8e49f3d8 Add glossary for some common terms
Change-Id: I3868b20fdd2d6cdff73e153d5022ad7c1b4f3c5d
2016-01-11 18:15:16 +00:00
SamYaple 4795802efd Add manifest parser and generator
This reads the manifest. Its more based on osdk.py that I initially
thought it would be. But hopefully I have cleaned it up a good bit.

Add tool to generate manifest

To generate a manifest with 2^24 objects (64TB for 4MB objects) run:
tools/generate_manifest.py --backupsize 64000 --manifest /pathtosave/manifest

Change-Id: I09a8fb64c0d4b8e4b4b436cca44b57bec87116cd
2016-01-09 00:55:38 +00:00
SamYaple e6d67dca74 Add some tools from osdk repo
These tools are not meant to be used as-is They need a full rework.
They are more reference material than anything else right now. I
also had to do some awful stuff to them to get them to pass the gate
in a hurry, code looks bad because of it.

I am cleaning them up as I go.

We will remove these in the future once they are no longer useful to
reference any longer.

Change-Id: I8f13ef282a2a05e72c53a0c307832a46db18d2b4
2016-01-05 03:09:05 +00:00
SamYaple fdfb83197c Initial Commit
Change-Id: Iaa4726694388e24e944a37da52c851bba1ead1ce
2016-01-04 16:28:29 +00:00
OpenStack Project Creator 90fa22688b Added .gitreview 2016-01-04 10:07:07 +00:00