Before removing contents of /var/dump the shotgun now checks that the
directory exists.
Change-Id: Ie503a17144452dcab97b31a4dd4dd5bfa2a4c46b
Closes-Bug: #1574695
Since compression command is actually a pipeline
we cannot use return code to detect out of space condition
so, the only way is to parse stderr
Change-Id: I2a95333bcabb73d0716d9790e14851cfefd47478
Partial-Bug: #1543491
We decode stdout from utf-8 to detect potential unicode problems
as early as possible.
Change-Id: If8fe35801d398eb5dbbd04f5f92317228df10ee9
Partly-bug: 1573638
On the 15G snapshot data gz is 10 times faster than xz while compression
rate is just 36% against 50% with gzip.
For the pattern "create once - download once" this looks like a
reasonable solution.
Change-Id: I5d2ce3681ec6555a8994bb39def907006be8b79f
Closes-Bug: #1546023
In order to correspond to global requirements and integrate
depper with OpenStack CI this patch switches shotgun from
py.test and unittest2 to testrepository and oslotest.
Change-Id: I08e31a5c16f4ad55c096d340eb2e2bff4fdb0a36
Partial-bug: #1469045
This patch allignes both versions and order of packages in requirements.txt
and test-requirements.txt with master revision of OpenStack's global
requirements in Mitaka release cycle.
In order to perform the soft sync all settings from setup.py were moved to
setup.cfg and setup.py was autogenerated using cookiecutter and the default
project template for OpenStack projects.
Change-Id: Iabf7a0fd8e3b01ea1712a663782399631b186f10
Partial-bug: #1469045
* We have several packages:
- nailgun-agent
- nailgun-mcagents
- python-packetary
changes to which should be reflected in changelog reports
* increase number of records to show from packages changelog
(in case full report)
Change-Id: I155f106edeab2e4cf99e5debf2e60bafeb058d28
Partial-bug: #1557162
1. Fix logging for cli part to get use of builtin cliff commands -v, -q
2. Add short-report command for shorter and cleaner output
3. Fix built-in name shadowing
Example usage:
shotgun2 short-report - shows short report version
Change-Id: I465eca3c509196d6bdc850d0b73fed1e5fb3a3d7
Closes-Bug: #1551762
In order to correspond to global requirements and integrate
depper with OpenStack CI this patch switches shotgun from
py.test and unittest2 to testrepository and oslotest.
Partial-bug: #1469045
Change-Id: Ic0abb5e8e62113be300f871b065f9cc48cd381a5
Sometimes is hard to read shotgun2 report, especially when one of
the packages doesn't contain the changelog. This commit adds an
empty line separator in the report output.
Change-Id: I504dcfffed0f88c57aa9f1956abd173eb39d7262
Closes-Bug: #1548541
This patch allignes both versions and order of packages in requirements.txt
and test-requirements.txt with master revision of OpenStack's global
requirements in Mitaka release cycle.
In order to perform the soft sync all settings from setup.py were moved to
setup.cfg and setup.py was autogenerated using cookiecutter and the default
project template for OpenStack projects.
Partial-bug: #1469045
Change-Id: I9d99517b9016f61904ab5e25d8ffb39cfb0c1119
If generating snapshot throws an IOError with 28 error code
(errno.ENOSPC) whole snapshot is deleted to not clutter drive.
Change-Id: I442b6bfe7ea5d3b3661351ed36f973f870d4d95d
Partial-bug: 1529182
Dumping shotgun resources is now done through symlinks. All local
resources are symlinked in dump directory, after that dump is compressed
with tar using -h option (--dereference).
Excluding files from tarball is now done by passing --exclude option to
tar instead of removing files before taring to avoid deleting logs.
Symlinks are created by 'ln -s' command because of wildcards used in
shotgun settings.
Change-Id: Ie9a0ab51d5874cd46a3919179def0aef407e7340
Partial-bug: 1529182
As we got rid of using docker containers on the
Fuel master node, we don't need anymore to
run any commands inside docker containers.
Depends-On: I9c2a3b1f988ba9bb937546eab2a0dbe84d19b18b
Implements blueprint: get-rid-docker-containers
Change-Id: I345fcadc3dd138251657571641bc505b721bd76f
This subcommand is to be used instead of version.yaml
Version.yaml used to provide short info
about what is installed on the master node.
We've switched to the package based
delivery approach and using build time artifact
(version.yaml) while packages could be easily
updated using yum command does not look rational.
This patch makes it possible to gather a short
report about what is installed on the master node
as well as about the content of some file in
a data driven manner.
One can run
shotgun2 report --config /path/to/report_config.yaml
/etc/shotgun/report.yaml is used by default as a config for
the report subcommand. It is to be provided by
shotgun rpm package.
Change-Id: I304a972c0c275d23ce2896a4d4e60870f9504fb4
Related-Bug: #1515517
This patch makes it possible to run not
only commands but multiline scripts.
Change-Id: I5c8692d0dcabe0041c1da67e3b4f7818810e3bce
Related-Bug: #1515517
Introduces a new snapshot driver that is
the same as Command driver but prepends
'docker exec ' to each command in the
object.
Change-Id: I8bf939bdb5a489982bc38257a245b4233103a07a
Related-Bug: #1515517
This patch makes it possible to define
multiple commands in a single object.
- type: command
command:
- cmd1
- cmd2
to_file: result.txt
Change-Id: I4c2052551aeddae78e6d6bdedf8080a0389aa4b3
Related-Bug: #1515517
Shotgun now uses python-cliff to provide
CLI like:
shotgun2 snapshot --config /path/to/snopshot_config.yaml
This change will allow us to add other subcommands
like "shotgun2 report" that is to be used for gathering
short report about the master node (instead of version.yaml)
Previous version of shotgun CLI is still available
and deprecated.
Change-Id: I3bc1f711d0e0fcf06f9414df1533d641a0e46595
Related-Bug: #1515517
Python 2.6 support would be dropped by
Infra team from mitaka,CI would no longer be testing it,
so projects should drop it too.
Change-Id: I5b849d02826f8af1a1b6efc445fa3026eb30ef9c
Set 'output' property for local command. This is required for
'mktemp' to work similarly in both cases (local and remote
command).
Change-Id: Ic4efe586683424c7145e811aa3495cf23e8f550e
Closes-bug: 1528815
The fabric.api.run command returns the string with combined stderr and
stdout. Raw output of the fabric process contains additional line
prefixes to indicate executed command and stdout/stderr stream. Using
raw output to fetch file name of temp file created by 'mktemp' command
is incorrect and causes DB dump to fail.
Add CommandOut.output property and use it to store the output string
produced by fabric.api.run. Use this property to set the name of
temp file in Postgres driver.
Change-Id: I9678a408f5532db1acb8fc36dd4eef4a95b61bfe
Closes-bug: 1526081
This issue was introduced by
https://review.openstack.org/241271. Actually the changes in that
patch doesn't allow to execute a command (or file/dir operation) on a
local host.
This patch fixes:
* execution of a command on a local host (when neither address nor
hostname is specified)
* missing dump for PostgreSQL in a snapshot
* getting network address (address may be absent, but hostname could be
present, so we need to use hostname in that case)
Change-Id: Iafecfa458372458963abd1e16890fe225454a612
Partial-bug: #1519303
* Add 'six' library to 'requirements.txt' and 'shotgun.spec' for Python 2.x/3.x
compatibility.
* The version binding is taken from https://github.com/openstack/requirements
Change-Id: Ic2f160383f6e4ec186ca251ca65e017909b713b0
Closes-bug: #1519688
Shotgun dumps /var/log/shotgun.log as normal part of the snapshot, but
it fills it in as the snapshot is created, which resulted in dumping
shotgun log split in half.
This patch makes shotgun dump it's own logs at the end of snapshot
creation process, just before compressing it.
Change-Id: Ie3e9737b2f430fd2f6ff3083b216f0ee62681c97
Closes-Bug: #1496842
* Avoid usage of `localhost` and `127.0.0.1`;
* For a remote command execution either `address` or `hostname` could
be used;
* Execute a command on a shotgun' host if neither `address` nor
`hostname` is specified;
Change-Id: Iac1d9ea37400e4d202a9bc673c5fa3d3d185701b
Partial-Bug: #1511730
Partial-Bug: #1517844
Apparently, shotgun never tested which nodes are online
prior to execute snapshot method of objects. Therefore, for offline
nodes shotgun will then consume a lot of time waiting for connection
timeouts to be happened on per object basis.
This patch totally rework how shotgun handles the objects:
* objects being processed on per host basis
* on NetworkError, all affected objects on per host basis will be
postponed for processing at the end of queue.
* if postponed objects still couldn't be proccessed on the last
attempt, they will then be converted into offline objects.
* adds `Offline` object driver. Being snapshotted, this object just
creates a file with friendly reminder that particular node was
offline inside of snapshot target directory. So, one which's
looking at snapshot archive will effectively realize that node
was offline without deep diving into shotgun log file.
So, shotgun won't waste time or per object basis, as it was before.
Change-Id: Ib99476c7b67a04d4f472f3fa3803b3fb92d4fec4
Closes-Bug: #1397038
Andreas Jaeger