The old peakmem_tracker service has been disabled in [0], now enable
the replacement memory_tracker.
Also fail when the old service is still configured, otherwise
consumers might never notice.
Depends-On: https://review.opendev.org/739995
Change-Id: I583caf3f36a8ff41d7d4106dabc6c5f24243085e
'tools/mlock_report.py' script requires 'psutil' package to be
installed. This ensures it is done before memory_peak service is
started.
Partial-Bug: #1860753
Change-Id: I7b2b6eaf9856c6057e1a4a0054d15074150a6cb6
During the PTG there was a discussion that the screen developer
workflow wasn't nearly as useful as it once was. There were now too
many services to see them all on one screen, and one of the most
common service restart scenarios was not restarting one service, but a
bunch to get code to take effect.
This implements a 3rd way of running services instead of direct
forking via bash, or running under screen, which is running as systemd
units.
Logging is adjusted because it's redundant to log datetime in oslo.log
when journald has that.
Swift needed to have services launched by absolute path to work.
This is disabled by default, but with instructions on using it. The
long term intent is to make this the way to run devstack, which would
be the same between both the gate and local use.
Some changes were also needed to run_process to pass the run User
in. A hack around the keystone uwsgi launcher was done at the same
time to remove a run_process feature that only keystone uwsgi uses.
Change-Id: I836bf27c4cfdc449628aa7641fb96a5489d5d4e7
In order to get memlocked pages this needs to be run as root, just
start it as root so that we don't have issues with the inability to
run sudo later in the run.
Change-Id: I7adab8cbb6d89d4717e427aec22e316d27bea075
The change makes peakmem_tracker list processes that lock memory pages
from swapping to disk. It may be helpful when debugging oom-killer job
failures in gate in case when dstat shows that swap is not fully used
when oom-killer is triggered.
The peakmem_tracker service was renamed into memory_tracker to reflect
its new broader scope.
Needed-By: I5862d92478397eac2e61b8a61ce3437b698678be
Change-Id: I1dca120448ee87930fe903fd81277b58efaefc92
I noticed this when debugging some grenade issues failures.
An include of grenade/functions stores the current value of XTRACE
(on) and disables xtrace for the rest of the import.
We then include devstack's "functions" library, which now overwrites
the stored value of XTRACE the current state; i.e. disabled.
When it finishes it restores the prior state (disabled), and then
grenade restores the same value of XTRACE (disabled).
The result is that xtrace is incorrectly disabled until the next time
it just happens to be turned on.
The solution is to name-space the store of the current-value of xtrace
so when we finish sourcing a file, we always restore the tracing value
to what it was when we entered.
Some files had already discovered this. In general there is
inconsistency around the setting of the variable, and a lot of obvious
copy-paste. This brings consistency across all files by using
_XTRACE_* prefixes for the sotre/restore of tracing values.
Change-Id: Iba7739eada5711d9c269cb4127fa712e9f961695
Future work toward visualization of DevStack and devstack-gate performance
would benefit greatly from the availability of machine-parsable DStat output.
This patch outputs an additional logfile to $LOGDIR, `dstat-csv.log`, using
DStat's built-in CSV logging functionality.
An additional instance of DStat is started during start_dstat that outputs
to CSV-formatted text without `--top-cpu-adv` and `-top-io-adv` enabled, as
these plugins are currently incompatible with CSV output. To facilitate this,
a new `dstat.sh` script is added to $TOP_DIR/tools/ to act as a daemon to
manage the two processes.
Change-Id: I826c94c35b6a109308b4f132c181ff7a1f63bc7b
Depends-On: I534fb1f9356a7948d2fec0aecc7f275e47362a11
We can see at-a-glance memory usage during the run with dstat but we
have no way to break that down into an overview of where memory is
going.
This adds a peer-service to dstat that records snapshots of the system
during peak memory usage. It checks periodically if there is less
memory available than before and, if so, records the running processes
and vm overview.
The intent is to add logic into the verify-pipeline jobs to use this
report and send statistics on peak memory usage to statsd [1]. We can
then build a picture of memory-usage growth over time. This type of
report would have allowed better insight into issues such as
introduced by Idf3a3a914b54779172776822710b3e52e751b1d1 where
memory-usage jumped dramatically after switching to pip versions of
libraries. Tracking details of memory usage is going to be an
important part of future development.
[1] http://graphite.openstack.org/
Change-Id: I4b0a8f382dcaa09331987ab84a68546ec29cbc18
It is not clear to me why this can't use run_process? Currently we
end up with two log-files both with the same thing
- dstat.txt.gz which comes from the "tee" and is symlinked into
SCREEN_LOGDIR, so gets picked-up by the gate scripts
- screen-dstat.txt.gz which comes from screen_it
Change-Id: I00b9e09b8d44f72ff14e69dc6e4a4bd5e2a0439e
This adds the test infrastructure for testing that unstack.sh and
clean.sh do the right thing, and actually stop what's expected. This
is designed to be used in upstream testing to make unstack and clean a
bit more certain.
It includes numerous fixes to make these pass in an errexit
environment with the gate config. The scripts still don't run under
errexit because we don't assume we've handled all possible cleanup safely.
Change-Id: I774dfb2cc934367eef2bb7ea5123197f6da7565b
If the SCREEN_LOGDIR and LOGDIR environment variables point to the
same location, devstack creates a dstat.log which is a symlink
pointing to itself. The second invokation of devstack then fails
trying to reference this broken symlink
Change-Id: I1de2bb7983e7535b41b28f526083a0d77312ff85
This is the first step in the log file cleanup. If SCREEN_LOGDIR
is still set, symlinks will be created in the old screen log directory
so things like the devstack-gate log collector continues to work.
bp:logging-and-service-names
Change-Id: I3ac796e322a18dbd0b8b2310a08310ca159d7613
This renames the log files in logs/screen that contain timestamps to put
the timestamp after '.log' and '.log.summary' in the names. This will
simplify devstack-gate's search for log files to copy to '*.log'.
dstat.txt is also renamed to dstat.log
Make LOGDIR and LOGFILE local
bp:devstack-logging-and-service-names
Change-Id: I02aba9ca82c117a1186dafc1d3c07aa04ecd1dde
With gerrit 2.8, and the new change screen, this will trigger syntax
highlighting in gerrit. Thus making reviewing code a lot nicer.
Change-Id: Id238748417ffab53e02d59413dba66f61e724383
In order to use dstat on the new side of grenade, pull dstat code into
its own lib with the function start_dstat
Change-Id: I5c908d594a6f3a90ed4b3f744002bf606841cf07
Rodolfo Alonso Hernandez