Commit Graph

146 Commits

Author SHA1 Message Date
Takashi Kajinami 249b795c8c Drop direct execution of collect.py
Now we generate the console script using setuptools, and we don't
expect users may directly execute the *.py file.

Change-Id: Ida036f138d8e7ef64ea0f8f7c83deeeb71dc4f5b
2024-01-29 23:56:44 +09:00
Takashi Kajinami 14ec232280 secret_access_key should be secret
Apparently the option takes a secret credential.

Change-Id: I8802720223c92683e46c5f5f2a8abb76978a176f
2023-11-28 23:09:26 +09:00
Alex Schultz 3a7d05a9e8 Drop six
With python2 not being supported anymore, let's drop six.

Change-Id: I22f3d28e22ad209806b78ad78f46671a74179157
2021-09-24 10:27:29 -06:00
Hervé Beraud 24245ce18e Use unittest.mock instead of mock
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.

Change-Id: Ib353cf9878334d91e551bf75121413c6fb2222ea
2020-06-09 19:28:11 +02:00
ricolin d1607f9f56 Add tls proxy support for Zaqar
Change-Id: I234a3d60333ee89a7c283b425e524d4624191577
2020-05-19 14:06:02 +00:00
Ben Nemec 4fa30ae5d7 Don't ignore SIGPIPE
By default Python configures SIGPIPE to be SIG_IGN, which means to
ignore the signal. We don't want that as it causes problems when
journald restarts and our log calls start triggering SIGPIPEs.
Instead, we want to allow the SIGPIPE to kill the process so it can
be restarted by systemd.

Change-Id: I512139b96b2de8b372efc91e8a3fc8d33553405a
Closes-Bug: 1795030
2018-09-28 16:33:11 +00:00
Feilong Wang 5bebbf1654 Add region support
Now Heat is missing the region support for software deployment
and so did os-collect-config. This patch fixes it but keeps the
backward compatibility.

For changes in Heat pls refer Iec6f3606c9fdf8474f393b0990356f34d38bcf75
and https://review.openstack.org/580470

For changes in Heat agents please refer
I8e0518fa61e237ec055834dd4bebe0fc87cd6627

Story: 2002781

Change-Id: I88182a9a9af74e9760b2ec9b500971f06293f0b8
2018-07-11 07:16:39 +12:00
Alex Schultz 69f0af9ccc Lower local collector log level
The local collector was added as the default but we're not properly
configuring it anywhere so this results in two log messages being
printed. Let's lower one of the messages to info and leave the one about
non-existing directory as a warning to indicate possible
mis-configuration.

Change-Id: I52ead32cce1eaf5f3816507e8e62b270f1a1a1cd
Closes-Bug: #1691003
2018-01-02 08:44:13 -07:00
Jenkins 7b8d598d1c Merge "Use universal_newlines with subprocess.check_output" 2017-04-13 05:34:40 +00:00
Lukas Bezdicka 8566ad549c Don't fail on non-valid json in cache files
Resolves: rhbz#1438096
Closes-Bug: #1678328

Change-Id: Iee1bb4a7769568cfd3bad8b23c9dff440679ac16
Co-Authored-By: Alex Schultz <aschultz@redhat.com>
2017-04-11 07:58:07 -06:00
Corey O'Brien cb171db770 Use universal_newlines with subprocess.check_output
Python 3 will complain when trying to split a byte-string using a string
so this change enables universal_newlines to make the split work in both
python 2 and 3. Also add a py3 tox environment.

Change-Id: I9af74b6c6cc9cfd0c973c85be04791aa6dc0acd4
2017-04-10 09:00:31 -04:00
Alex Schultz 0762fcb0b5 Add splay option to offset polling intervals
When the os-collect-config process is started on multiple systems at the
same time, the polling intervals can line up to cause performance
problems against the configuration source. To reduce the impact, this
change adds a splay option to allow the operator to configure a random
delay prior to the polling to attempt to offset the polling
syncronization.

Change-Id: I1a8be3345d783da9014eca7ea26da19d57e767c0
Closes-Bug: #1677314
2017-03-30 12:44:21 -06:00
Jenkins 0e2a8ab43a Merge "Use pop for checking messages with websocket" 2017-03-11 02:10:01 +00:00
Jenkins 34624661aa Merge "Allow setting minimum polling interval for collect" 2017-03-06 01:12:21 +00:00
ricolin d677da173b Allow setting minimum polling interval for collect
Add `min_polling_interval` in `[default]` config.
This config option allow setting minimum polling interval,
default is 1 second.
Closes-Bug: #1656893
Depends-On: If863d2e190a4e1701ebf1fcb99a783075cda2556
Change-Id: I213afabeeb63f143ce359a3495a6d88b9b74ef4d
2017-03-02 03:17:07 +00:00
Tony Breeds ec8b2576da Update to a newer hacking library
The older hacking library has a cap on pbr <2.0, with the recent 2.0.0
release of PBR it's causing failures in the pep8 job.

hacking isn't kept in sync via the typical proposal-bot updates.

Do it manually to clear the gate issue.

pep8 runtests: commands[0] | flake8
./os_collect_config/merger.py:45:22: W292 no newline at end of file
    return final_list

Change-Id: If863d2e190a4e1701ebf1fcb99a783075cda2556
Related-Bug: #1668848
2017-03-02 12:43:37 +11:00
Jenkins 6ddcd55394 Merge "Only re-exec on config change or HUP" 2017-01-31 05:21:13 +00:00
Thomas Herve 432abd47ec Use pop for checking messages with websocket
I1c94ad82c3b20cc331045c19d4a9987a701b081b fixed the issue in Zaqar where
the pop argument was wrongly named. We can now use it in the zaqar
collector to retrieve messages over websocket.

Change-Id: I0ce894b0eef7018a37db9ef9d1b43d8d20629bf0
2017-01-09 10:58:23 +01:00
Dan Prince b6364df3e5 Add support for Zaqar websockets
This patch adds a new zaqar.use_websockets option which if set to True
in the config file will configure the Zaqar collector to use
the websocket transport instead of wsgi. This can be more efficient
where you want to avoid the continuous polling of o-c-c and instead
just listen on the websocket that is subscribed to a queue.

Like other collectors each iteration creates a new socket object. This
allows us to use the normal re-exec logic in o-c-c and thus gives
the option to re-configure the agent in the future to use other types
of collectors. We could (optionally) look into a higher level option in
the future that would allow o-c-c to avoid re-exec'ing and thus re-use
the same websocket for multiple sets of metadata.

Depends-On: Ia2a8deb599252d8308e44d595eb2bf443999aaad
Change-Id: Id5c7ed590df776844b6c7961eb40f89206cd24e0
2016-11-30 21:12:47 -05:00
Steve Baker 76975f04e5 ec2 collector attempt config-drive before metadata server
This change will first attempt to get ec2 metadata from a v2 config
drive before falling back to the nova metadata server if no config
drive is detected.

Config drive is enabled in overcloud nodes, so using this for
os-collect-config will allow the undercloud nova metadata server to be
disabled.

V2 of the config-drive was released in OpenStack Folsom, so a decision
was made to not support V1 discovery at all. This makes the block
device candidate selection much simpler than the cloud-init one[1].

[1] https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceConfigDrive.py#n219

Change-Id: I2aec0544fe40c2e901aee38c79bb838a627d20da
Closes-Bug: #1619074
2016-10-13 09:38:29 +13:00
Steve Baker a950edb320 Treat ec2 collector data as immutable
Currently the ec2 collector polls the nova metadata service every
$polling_period even though the data is not expected to change and no
known config actions have been written to respond to changes in these
values.

With larger overclouds, this metadata polling will cause noticeable
load on the undercloud, especially nova-api and neutron (for port
lookups).

This change will check for data in /var/lib/os-collect-config/ec2.json
and simply return that if it exists.

Change-Id: I84156f8b005b319fccf8ac17af8943aefb0a8f6d
Closes-Bug: #1619072
2016-10-12 12:16:59 +13:00
Steve Baker 61ff6aaec7 Use TEST-NET-1 for unit tests, not 127.0.0.1
Currently if http requests are not correctly mocked, behaviour of the
unit tests will depend on what happens to be running on localhost.

Jobs like gate-os-collect-config-python27-ubuntu-xenial are currently
broken because discover requests are being made to localhost and
something has change in the environment to cause test failure.

This change does the following:
- replace unit tests addresses from 127.0.0.1 to 192.0.2.1 (TEST-NET-1)
- fix mocking of tests which now stall because 192.0.2.1 isn't accessable
- change the mocking approach to keystoneclient.discover.Discover to
  a simple stub class

Change-Id: I6d9450d6854b9f50dac0f55f54f1ea1d579a27ac
2016-10-12 12:16:52 +13:00
James Slagle 9ee07c96c2 Revert "Treat ec2 collector data as immutable"
All CI jobs are timing out currently during NetworkDeployment
because 99-refresh-completed is not signaling to Heat due to
instance-id being detected as null by os-apply-config.

I suspect this patch is causing the issue.

This reverts commit 8717436db9.

Change-Id: Id0d122d7e855cd828a0742dabc5f7089fb6a913c
Closes-Bug: 1623754
2016-09-14 16:02:22 +00:00
Steve Baker 8717436db9 Treat ec2 collector data as immutable
Currently the ec2 collector polls the nova metadata service every
$polling_period even though the data is not expected to change and no
known config actions have been written to respond to changes in these
values.

With larger overclouds, this metadata polling will cause noticeable
load on the undercloud, especially nova-api and neutron (for port
lookups).

This change allows collect calls to raise a SourceAlreadyCollected
exception if the data for this collector never changes. The ec2
collector raises this if its data has already been cached to
/var/lib/os-collect-config/ec2.json

Change-Id: Ib7df590601132857690c8ab64fe32098a81752d8
Closes-Bug: #1619072
2016-09-01 13:42:11 +12:00
Jenkins 20630c3360 Merge "Merge deployments data for zaqar collectors" 2016-08-30 21:19:47 +00:00
Dan Prince 90cb2721e9 Only re-exec on config change or HUP
There are 2 things at play here. First the logic added in
4cfeb28d12 is no longer needed
because our initial sleep time is very low and increases
gradually up to the max.

2nd, I'm proposing that we avoid reexecing unless the config
file actually changes. Not re-execing will give us the option
to optimize os-collect-config for some long running collectors
(like zaqar websockets).  Also, Os-collect-config updates would
already be handled by packaging restarts and or other deployment
system changes anyways.

Change-Id: I04b2752d007089f72af42c88c4249c3e11c8346f
2016-07-19 11:57:54 -04:00
Dan Prince 08ba371994 Merge deployments data for zaqar collectors
This is required so that a the TripleO undercloud can
switch to polling for metadata from a Zaqar rather than Heat or
Swift.

Change-Id: Ieb076185314a7aeadd8a708ab01b48dfbccef006
Closes-bug: 1604098
2016-07-18 14:13:22 -04:00
Steve Baker e5518c11c7 Set a timeout for all data fetches using requests
A timeout config value is added for each collector which uses requests
to fetch data, and this value is used for any requests calls.

Without a timeout a request may stall indefinitely and
os-collect-config will stop polling.

A timeout default of 10 seconds is chosen as the default. This is used
for both the connection timeout and the read timeout.

Change-Id: I4ad0065b5a85393105c6385a15653d7204b4f880
Closes-Bug: #1600652
2016-07-11 11:14:57 +12:00
Ben Nemec f092ea133f Switch to oslo.log again
We're currently still using unmaintained oslo-incubator code for
our logging, which is bad.  This switches us to oslo.log as
described in [1].

1: http://docs.openstack.org/developer/oslo.log/migration.html

Change-Id: Ibce86ab4ee24eeb55d0de1b0d5ff4ee4ea6ef66f
2016-03-22 15:38:05 +00:00
Ben Nemec ab5c66df45 Revert "Switch to oslo.log"
This isn't quite right and broke on stable/liberty.  Pushing
a revert in case I3c22d77dece399d21ab94783b74990789a1e1481
doesn't actually fix the problem.  We should probably merge
whichever passes first.

This reverts commit 69653318f4.

Change-Id: I9304429f25d28ca756e50b1788e149c5bb46b1d6
2016-03-21 20:50:25 +00:00
Ben Nemec 69653318f4 Switch to oslo.log
The old oslo-incubator log module isn't maintained (and doesn't even
exist anymore), so we don't really want to be using it.  It appears
this was the only incubator module we were actually using, so this
allows us to remove all of the unmaintained incubator code.

Change-Id: Ib4ad3b231360987a1ef4f95b5b5a8b656232efc4
2016-03-17 14:40:46 +00:00
Swapnil Kulkarni (coolsvap) 8b28ced627 Replace assertEqual(None, *) with assertIsNone in tests
Replace assertEqual(None, *) with assertIsNone in tests to have
more clear messages in case of failure.

Change-Id: I2cc0d913d83ea56c323475daddb66a0047564b42
2016-01-04 19:41:10 +05:30
Dan Prince b63188eb25 shorter sleeps if metadata changes are detected
This patch updates os-collect-config so that the sleep interval
time is shortened if changes are detected. This should decrease
deployment time when using Heat templates which use depends_on
to step through a sequence of software deployment resources.

The new default sleep is set to 1 and increases
by sleep_time *= 2 until it reaches the default
sleep interval again.

Change-Id: I5cbd0956db2abebec876b15bee72b70ec64d5aef
2015-10-21 16:56:21 -04:00
Thomas Herve b13122918c Add support for zaqar transport
This adds a new zaqar collector able to read configuration from a
specific zaqar queue.

blueprint software-config-zaqar
Change-Id: Ie38af7b59e7a1aa370ac7760bb7819e37c2165c3
2015-08-26 11:33:10 +02:00
Steve Baker acafb1b94a Merge deployments data for collectors heat, request
This is required so that a swift-enabled TripleO undercloud can switch
to polling for metadata from a TempURL rather than heat.

Change-Id: I73ac9e01f85e0c72ce7411e2c61c545322f3dccc
Closes-Bug: #1424913
2015-06-15 09:14:33 +12:00
Steve Baker 4b68dfd53d Refactor merging out of cfn collector
This is a pure refactoring change which moves the os-apply-config
deployment data merging into its own module. This will allow
other collectors (request, heat) to do the same merging.

Partial-Bug: #1424913
Change-Id: Ic78a60e3efebadbb06ebfd262ceb275ca519a3f2
2015-06-15 09:14:29 +12:00
Jenkins e80616d15a Merge "Log with info instead of warn when collector not configured" 2015-06-04 17:15:33 +00:00
Jenkins 163030c081 Merge "Drop use of 'oslo' namespace package" 2015-05-29 09:40:47 +00:00
Steve Baker d2c5bd1b0e Log with info instead of warn when collector not configured
Users who are debugging unrelated issues are frequently interpreting
logged warnings as an indication that os-collect-config is misconfigured
when the warning is actually harmless.

This change uses info instead of warn for this logging.

Change-Id: I28e0fb37b603364eea72cdc5299a29259489b820
Closes-Bug: #1437952
2015-05-15 11:54:23 +12:00
Doug Hellmann 4a41f0b3e5 Drop use of 'oslo' namespace package
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.

The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.

Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.

Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages

Change-Id: If51059c31c82d5235e2ae21143911b5561783ca6
2015-05-06 18:45:17 +00:00
Yanyan Hu f6ca7618c4 Use non-versioned auth_url for keystone
keystoneclient discover can't get v3 auth_url using v2 auth_url. It
accepts an url without version number. This patch fixes this issue.

Change-Id: I45b4c016bd91afc39d39620920951043bb71df05
Closes-Bug: #1398254
2015-04-24 13:52:24 +02:00
Endre Karlson c917665af9 Add ca_certificate option for SSL'd api
Change-Id: I12eb9dbbb8bee24e50ae342ffbc7356d4583a973
2014-12-05 13:59:55 +01:00
Jenkins de6d420f40 Merge "Add unit test for heat_local" 2014-11-25 17:53:11 +00:00
Dougal Matthews 4bc37fd50e Bump hacking to 0.9.x series
Change-Id: I474266c647f31f0ad92ec9878fc03eb5a5c97f43
2014-11-13 13:11:00 +00:00
Gregory Haynes 46e950ddd1 Enable local collector by default
This is a useful collector and it seems odd to not have it on by
default.

Change-Id: Ibc3be6ff27c6290ce32bc01d0e62acf3253545c3
2014-10-30 01:26:09 -07:00
Jenkins 72bd2854f6 Merge "Handle missing local collector directory" 2014-10-29 16:13:22 +00:00
Gregory Haynes d05f57b092 Handle missing local collector directory
We currently explode if local collector is enabled with the data
directory missing. Changing this to a warning allows us to have this
turned on by default without greatly inconveniencing users.

Change-Id: I36364ba1a0706a5a2c820eadd526f2ba424ac665
2014-10-28 13:34:48 -07:00
Steve Baker ad475ee927 Collector which does a simple HTTP GET
This change implements a collector which does an HTTP GET via
python requests to fetch the metadata.

It should work with any GET-able URL, however it is designed to
work with Swift TempURLs.

Swift objects are not consistent, so the Last-Modified header is
checked for each poll and metadata is not fetched if the last
modified is not newer than the previous successful poll.

This collector will be enabled for OS::Nova::Server
software_config_transport: POLL_TEMP_URL which is available
in the Juno release of Heat. Using POLL_TEMP_URL will result
in no metadata polling load on heat, which has historically been
an issue with tripleo scalability.

Change-Id: I22155c22bdcc3c81a5e945ca5436a8f29f196528
2014-10-17 09:40:23 +13:00
Gregory Haynes 6d32511dd0 Exit with error status on failed command
When we detect a failed command we log ERROR but we do not return an
error status. This makes it difficult for programs which may run
os-collect-config to detect whether a run was sucessful.

This only applies to runs which are performed with --one-time argument
as this is a straightforward case.

Change-Id: I168862e8c75c15d1ea405a417908d1284feb7b32
2014-10-09 00:20:03 -07:00
Gregory Haynes 7e913145a3 Fix py3k popen and locale test fails
A popen call is made in the Python 3 platform module which causes our
assertions for our mock'd popen to fail. Also fixing a locale issue to
use defaultlocale() rather than LC_ALL.

Co-Authored-By: Steve Kowalik <steven@wedontsleep.org>
Change-Id: I45cdf7d921fbca5f09e5f65bd69fcee83fb7c8e4
2014-10-09 00:17:43 -07:00