We are no longer piping our service output to logger, so we should be
logging the stdout via upstart. This is useful for detecting tracebacks
and other info which does not go through python logging.
Change-Id: I5d4abb94ff15f74fd2478e70d0fa4d3bb9c09f2d
We currently have a horrible hack for upstart where we run our services
with stdout and stderr piped into logger. There are lots of issues with
this, such as upstart has a hard time knowing which PID to track.
Conveniently, python logging knows how to syslog, so why not just use
that!
This changes the default behaviour of os-svc-daemon to create a python
logging.conf so services log directly to syslog.
This also changes the default upstart respawn limiting to be 2 fails in
10 secs. This is becauce making upstart monitor the actual service
daemon was causing some services to respawn loop due to time it took
them to fail.
Closes-Bug: #1385346
Change-Id: Ibd491881d79160ad1457526d51512f5fc1cf9ea5
We advertise supporting passthrough arguments in os-svc-daemon by using
'--' followed by extra params. This does not work if using the
undocumented old-style positional arguments and fixing this is a
non-trivial parsing problem. Lets just deprecate this old behaviour so
we can remove it.
Change-Id: I0e75f7f975325b5f199daf697004628461a20075
This reverts commit 615c6bb95b.
Using "expect fork" results in hitting a bug causing some service
restarts to hang which in turn is preventing os-refresh-config
from completing.
https://bugs.launchpad.net/upstart/+bug/406397
Change-Id: Ib893b1428e3cac3084d07b4d98c9cb69d60bb3cf
Otherwise upstart doesn't get the correct process id for the service
and in some causes can fail to kill the process during a servcie stop
Change-Id: I138dd2206ed1e511fbabaf43bd3cb1d163c76db4
Closes-bug: #1385346
Debian and Ubuntu use /lib/systemd whereas Fedora usr /usr/lib/systemd as the
default system unit directory. This already leads to conflict when the
debian-systemd is used.
Fedora provides a symbolic link between /lib/systemd and /usr/lib/systemd
to also support /lib/systemd directory. With this patch, we use /lib/systemd
everywhere.
This patch supersedes I413a92284e6a79e7fcdde99c5138bc3ce8e85e80.
Change-Id: I459f7514ab35082d31607968252a9005fa25de2a
See: https://bugs.debian.org/719634
This patch incorporates the svc-map patch[1] into other
elements by integrating svc-map into the os-svc-install
element. This patch adds the -a option to select svc-map
instead of the default of map-services. The -a option
enables a phased transition from map-services to svc-map
usage throughout the rest of the elements.
[1] Id11433ea342aace71a358936a7ca3151ec11d506
Change-Id: I382631746e458f2f8603a7dcf5221a21f4b53255
/var/run/<service> needs to be relabeled with the correct file
contexts. With systemd, these directories are created anew each
boot by /lib/systemd/system/<service>-create-dir.service.
os-svc-daemon was updated to correct the file contexts.
Change-Id: I6487d0726db08912eed3062dfee2552940eadcfb
Closes-Bug: 1346559
Logging to an upstart logfile results in log entries that are not
associated with a timestamp(e.g. os-collect-config). This isn't
consistent with the log files produced in other openstack CI tests
and means we can't index them in logstash (at least consistently
with how the other projects are indexed).
As a result of this we will need to remove the double time stamps that
get produced (by taking them out in each of the services configuration)
and extract the log at the end of CI into per-service files so that we
can selectively index the files we want.
In particular the service that doesn't contain timestamps in all of the
log entries is os-collect-config which is arguably one of the most
important service logs we want indexed. We could simply just divert this
service alone to syslog but doing them all is more consistent with other
distros and how this will presumably work when ubuntu moves to systemd.
Change-Id: Ib8c1fc39d56b2b9c6d8e9b64a868def619aa2f1f
As advised in I072cf8bf6748d0c910fecffdf2282bcc4656d038, code should
use 4 spaces for indentation.
This commit enforces the use of 4 spaces indentation.
In order to simplify the review process, this patch only cover the
following elements:
- os-apply-config
- os-collect-config
- os-refresh-config
- os-svc-install
- pacemaker
- pypi-mirror
- qpidd
- rabbitmq-server
- snmpd
- stackuser
- tripleo-cd
Change-Id: I3f365f6a1cd6fd9e56402ad3bd6572192b85d798
In some circumstances we need to be able to create a /var/run
directory without actually installing a service. This behavior
was supported before, but these changes allow the user to specify
the directory name and streamline the process by making -c an
optional parameter instead of requiring a dummy value. It also
makes this functionality available on upstart-based systems.
The existing uses of -d had to be updated as well to reflect this
new behavior.
Change-Id: I34f33f2290eeefb5be82c0cf5198ae53d34332ce
Update os-svc-install with a new -i option which can be used
to specify the virtualenv directory.
Update os-svc-install with a new -s option which can be used
to enable system-site-packages within the virtualenv. By
default this is disabled (matches virtualenv's default...)
In doing this we remove a hard coded check for the 'nova' service
by making nova use the -s option instead.
Update os-svc-daemon with a new -i option which can be used
to specify the virtualenv directory.
NOTE:
I used -i for both os-svc-daemon and os-svc-install since it wasn't
already used and I wanted to make the option the same in both places.
Change-Id: Ief53567711b0153de890d50a28c471b1e5acd56c
For systemd services, we create 2 services: one for the daemon and one
that creates the run directory under /var/run for the daemon.
For the packaged installs, the daemon service is already provided by the
package, so we need the ability to add just the *-create-dir service.
Some might say that the package should create it's run directory...and
it in fact does. However, we are reconfiguring the services to use
/var/run and the package uses /var/lib. Since /var/run is tmpfs mounted,
we need the *-create-dir services for the packages as well.
Change-Id: If26d9d7da04d3d1d97809db0e21e54dd9f8042ca
Install systemd service files into a location that will work on Fedora as well
as on openSUSE. Note, that on Fedora /lib is just a symlink to /usr/lib anyway.
Change-Id: If1dcd0d428d36d587fab501d42b0c8689a972e31
Without this export os-svc-enable.conf never sees the starting event and
thus never does anything to prevent starting the job.
Change-Id: Ib94c817ce3f2e6c3aada1515beea6e7fb41605af
Recent changes landed to set the environment variable
OS_SVC_ENABLE_CONTROL=1 but the env statement was left out, rendering
the upstart job completely broken as the syntax is now invalid.
Change-Id: I88a7a63304d1fe759067e47e2fab88ee27f91914
Now that we call os-svc-enable in the individual elements we
can drop the call here.
With this commit *most* services will no longer autostart on first boot
and will rely on post-configuration scripts in os-refresh-config
to enable services. This should clean up our first boot logs
quite a bit...
Change-Id: Ied3622a5e7fce4c3ac2fbd9a0e513ae877cafc5d
Split out the code which enables services to start on boot into
a new os-svc-enable script. This script is currently called by
os-svc-deamon so that services automatically start on first boot
(as is the case with os-svc-install)
The motivation here is to decouple service installation and enabling
so that we can eventually avoid starting services on first boot
until after they are configured. Auto starting services on boot like
we currently do is causing lots of log ERROR messages, etc.
Change-Id: I66d4660c169f5918e924eab1876134891c3a24a1
These aren't needed during an image build and keep getting
copied each time a new service unit is added.
Change-Id: I498a229b69e78a3eca06bf8691c13157ecd6739d
diskimage-builder provides a helper script + env variable
to determine, which init system is used. We should use them
in elements instead of manual checks after update of raring
cloud images broke "if [ -d /etc/systemd/system ]" test.
Related-Bug: #1251610
Change-Id: I6d27491302ff0c4c6de82d6d60fc80e83fec519e
Systemd services should be created at /lib/systemd/system instead of
/etc/systemd/system, once the service is enabled systemd will create a
link from /lib/systemd... to /etc/systemd...
Also systemd services should contain the absolute path to the executables.
Change-Id: I741fe249de8ecc7b2af100ca6cf55c51f86f84b0
We also migrate os-svc-daemon to supporting switches to modify its
behavior and be more flexible as it grows. The old positional style is
still supported but will be deprecated.
Each service will have a unique way to verify that it has started. The
default 'exec sleep 1' will be sufficient for some, but not all. Quantum
is the first example, which can take more than 5 seconds to open a
listening socket.
Due to a lack of knowledge and testing on systemd, it will simply print
out a warning about the issue until we have a better answer.
Change-Id: I8d0e0842dfa66b2e32b942270ed071e593d3865e
The respawn count is not incremented until post-start exits, so we
need to set the count lower than the interval to allow for the lag
caused by the one second sleep. Two times in 5 seconds seems like
enough to handle the occasional unexpected crash but stay dead when
things are badly broken.
Change-Id: I8eaca6f9ea131861a42d473ffc5c09a41226ce0e
Upstart will return from 'start xxx' immediately upon forking. We still
need to run start-stop-daemon, change users, exec again, and then run
the python code to get to listen() before the service is actually ready.
This all should take < 1s, but the sleep is just a stopgap until we have
a better way to track "is the service ready" in upstart.
Fixes bug #1179766
Change-Id: I3842e572c940e833cb3036b67df0ee71fe5bb129
The 'boot-stack' element is a self-contained, baremetal openstack.
Upon first boot, it will initialize itself with default keystone
users, nova flavors, etc.
Change-Id: Ib0c411627154a3d666f74513c6b8edfbfbf6c07e
Best practice is to use start-stop-daemon rather than sudo, as sudo will
apply user-centric pam limits and create a wtmp entry. Also there is no
need for a script stanza, as we are only execing one command.
Change-Id: I0c2f12536b56d90fd43ab40e74424350efcc0b61
Python package dependency conflicts have been observed
to occur for certain combinations of services at certain
revision.
Running all services in virtualenvs removes the issue.
Change-Id: I100817569b43a5af3427b0ae20cebdc7d55d03a5
This allows openstack services to be git/pip installed
without installing any startup scripts.
This is useful for keystone-db or nova-db elements, for
example, where the service must be installed to perform
database migration, but no service start scripts need
be installed.
Additionally, add a tool to create openstack sql databases.
Use openstack pypi mirror:
Use the openstack pypi mirror for openstack service
installation. It's much faster than pypi.org.
Also, pip install $svc_repo/tools/pip-requires first,
if it exists, which is required to pick up oslo.config.
Change-Id: I72751d4da59f8597d20aea2f27a9dfabe2f63a8f
Ghanshyam Mann