Previously it was possible for an acl push to fail then we would record
successful acl push via our project cache's sha entry for that acl file.
The reason for this is despite success or failure of the acl push we
continue on without "problems". Unfortunately this can lead to projects
failing to update without any indication this has happened.
Update jeepyb to raise an exception if there is a failure pushing. THis
should prevent the project cache file from being updated and will cause
manage-projects to try again later. The downside to this is we will
short circuit our project handling if there are many updates to perform.
THis seems preferable to silently "succeeding".
Note that we special case read only projects by detecting the error
pushing an acl to a read only project and treating this as success. This
will allow us to delete our cache file and update everythin while
passing over read only projects as is.
Change-Id: I0d1c1160f2a28fc023fd71c8010ddcf6555efa14
manage-projects unconditionally set up a local git dir even if the
config option was unset. We change that behavior because new gerrit url
paths make a local git dir option less straight forward. If you've set a
git dir path we'll continue to honor it, otherwise this step is skipped.
Depends-On: https://review.opendev.org/762196
Change-Id: I39ef5ba7cd8747d52487cb0252a9b12878d6247b
By default we set repo HEAD to master but add configuration that allows
us to set it to some other value for new projects. Note this shouldn't
be used until tools for other systems like Gitea are ready.
We explicitly set the default to master as we rely on Gitea and Gerrit
and Git to all be in sync with the default branch. That is the case
today but may not be going forward. Being explicit allows us to avoid
early unexpected change but would also allow us to swap before all tools
update their defaults as well.
Note that we configure our local operating copy with the default head as
well to ensure there isn't additional confusion between the local repo
used to create .gitreview files and the gerrit canonical copy.
Depends-On: https://review.opendev.org/741277
Change-Id: Ie064056ba1722b1bca309dd9f96b61aae0a2fef9
The OpenDev Manual (formerly OpenStack Infra Manual) has moved to
opendev.org, so update the link for it in the welcome message
accordingly. Do the same for the pull request closer utility, and
also reword it for better generality (a bit of de-OpenStacking).
While we're here, update outdated URLs for other content and
services which have moved to opendev.org, refresh some configuration
examples, and correct some lingering OpenStack LLC copyrights to
OpenStack Foundation per the guidance at
https://wiki.openstack.org/wiki/LegalIssuesFAQ#Copyright_Headers
(though I did not touch scripts we no longer use, like create
cgitrepos or openstackwatch).
Change-Id: Ibb34d1ab8ab58dee653c51141a871834b93cd54b
Depends-On: https://review.opendev.org/729362
We're manipulating a Gerrit object not a GerritConnection object.
The username property is found on the Gerrit object.
Change-Id: I9b4c4cc0dacf456c1cb71c3798141ac726b1db71
Return strings from run_command
jeepyb is written expecting strings. Decode the bytes into a
string.
Fix fsck_command for python3
We fixed run_command to return strings, but fsck_command was
looking for bytes.
Filter returns an interator not a list in python3, wrap in a
list.
Change-Id: Ibb4d540a987711bd6b4c804a3f6b7cb2ccfe1baa
For some reason we don't log the git output so failures are opaque to
us. Fix that by adding additonal logging.
Change-Id: If212ae6115c8266e1352431588c04547c897e982
Direct access to the db restricts where we can run manage-projects from
and complicates testing. Instead rely on the list-groups ssh command to
get group uuid info. Note that this should be tested using new
integration testing before we merge it.
Change-Id: I6e3f0881398da8efeab5dfef370ad8d35163a7be
We were skipping the setting of retired project acls in manage-projects
because we were iterating over projects that weren't retired. In
manage-projects we don't want to skip over retired projects so that we
can retire the projects properly.
Address this by adding a new registry property that returns all projects
retired or not then iterate over that in manage-projects.
Change-Id: I7aa89b53696fb0fca262fc7068e5e2ad2b7ad8fc
* Currently, manage_projects always perform
`exit 0`, even if some errors were.
* Patch would add `exit 1` beh., in
case any issues were during processing.
It would affect only exit code , and will not
stop processing.
* Misc: perform auto-ident
Change-Id: I50d425e5a9e02441a50f7b1d32219485f80e6632
Newer gerrit automatically adds the user creating a group to that group.
We don't actually want this behavior as the user used to manage this is
used for nothing else. Check if the current user was added to the group
and if so remove it.
Depends-On: Id5d8915f2c0bf6f6057b414785b8ec1b18e810f2
Change-Id: Ia0ca5faaaa466ae7a930bbbfb78e2975771f3af9
We're caching the data already. If we at this, it'll cause us to update
descriptions in github when they change.
Change-Id: I1941958cca6387bd17fd3dcfd3c519af8b355453
created-in-github being True prevents the subsequent github actions
from happening. If we fail updating the github project after creation
though, with the current code we can be left with half-created projects.
Don't set it until we've actually make it through.
Change-Id: I159eda5d1a3e5a30eed5f322d1e29477a80c5bee
fsck_repo is used in both files, so move it to utils. push_to_gerrit is
only used in manage-projects and sync_upstream is only used in
track-upstream.
Change-Id: Ia81c8a54023d5f0cac6443a6e28475b0d2243aa0
There might be situation when project creation can be done from
other "endpoint" then common users have access. So we need to have
an ability to customise GERRIT_HOST and GERRIT_PORT parameters
inside .gitreview file for newly created project.
This change adds two parameters, that can be re-defined in
projects.ini file to format correct connection string to gerrit:
* GITREVIEW_GERRIT_HOST (by default = GERRIT_HOST)
* GITREVIEW_GERRIT_PORT (by default = GERRIT_PORT)
Change-Id: Ie3fc35c5b6714c579314faa18160594aeae79cb7
This is also shared by manage-projecs and track-upstream - but there was
also a change to its logic in the former patch which would be hard to
see if we moved it and changed it in the same patch.
Change-Id: Iab287596d1e0db529e9557cc057d558d5b7ecccb
We spend a very large amount of time in track-upstream code now because
of all of the debian repos. That's fine - but it makes iterating on
project creation issues exceptionally difficult. Also, now that we're
caching the flag as to whether or not the project is created,
track-upstream can read it and know whether or not it's safe to run for
a given project.
Change-Id: I454b1ba400dc86abcc9b939564eb4eb7c324308c
Saving the state for having created the project was sticking to false.
If we had the git repo locally but it wasn't in gerrit, the logic was
all message up. Checking for homepage and downloads in github kills us -
although it would be fine if we'd started from caching homepage and
description in the first place, so maybe we need a script that sets
status quo in the cache file.
Change-Id: I15802082c0bef9e9362eb9482b058f81d1e566d7
We continue to hit the GH rate limit. But honestly, in most cases, we
should not need to make a bazillion GH api calls.
Change-Id: Iac709a4f7f18d6348b21520bbc7aeacfa991ec1a
Implement a trivial caching system that will allow us to skip even
attempting gerrit and github remote operations for projects that we've
processed successfully in previous passes. This should hopefully reduce
the amount of time we spend processing the projects.yaml file.
In the future, we could extend this to tracking more specific actions-
such as the description set in github, so that we could know whether or
not they have changed and thus need to be processed again.
In this form, acls files will always be processed.
Change-Id: I07b13c8663e6f9ee1255a4e56caf556ea49fb51b
Don't retry checking the DB multiple times on first call to see if group
exists when determining whether the group needs to be created.
If the group exists, a previous iteration would already have waited for
the group to be written down from the cache. So if nothing is returned
then the group does not exist, and there is no point rechecking multiple
times to see if it will appear from the cache.
This speeds up creation of new groups by avoiding a 10 second wait
before creation due to the first call for group uuid unnecessarily
retrying multiple times.
Change-Id: I0afbc716159e8aecf1ade6442d9b02674094fa08
This is a sanity check to properly log when repos are corrupt and not
able to be imported into Gerrit. The import will fail either way but at
least with this change in place we will get more direct logging of the
reason for the import failure.
Change-Id: Ibb1839329065e5d4a9747f95ff21029148801874
A Gerrit bug[1] allowed manage_projects to create a Gerrit
internal group with the same name as a system group
('Registered Users'). This happened after upgrading to
Gerrit 2.11 because there are existing projects with 'Registered
Users' in acls config files. Even though Gerrit shouldn't allow
creation of a group with the same name as a system group,
manage_projects shouldn't even attempt to do it.
This change prevents manage_projecsts from creating
groups with the same name as Gerrit system groups. It
will also add correct entries to group files for system groups.
Below is the group file entries that Gerrit expects for
system groups:
global:Anonymous-Users Anonymous Users
global:Change-Owner Change Owner
global:Project-Owners Project Owners
global:Registered-Users Registered Users
For backward compatibility, the existing group UUID will be returned
if one is found in the database.
[1] https://code.google.com/p/gerrit/issues/detail?id=3831
Change-Id: I17d6e1a5553f98ff910480d41a2ac7b3a2522809
The command line utilities provided only haphazardly set up logging
using command line arguments. Refactor it into a common module, and
make use of it in all scripts that use logging.
This changes expire-old-reviews.py to not hardcode the log file, but
that is fine, it has no scripts calling it.
Change-Id: Ibc85f8e3b47f9c7898ad4334511b44e91ecbd736
The 'groups' file gets created only if there are groups mentioned in
the 'project.config' file. A 'project.config' file is not required to
mention any groups, thus we should only attempt to add the 'groups' file
if it exists.
Change-Id: I333df5156bc27db29143ee412d86e12606faa751
The command being run is more informative than it's output, so raise the
logging level of the command being run to info and lower its output to
debug.
Change-Id: Ic4f764192384e3d0911380e5daf60fcdf0d2a200
Currently info-level logs only contain project names when errors
occur, and otherwise have a lot of information about branch processing
with no information about which projects the branches belong to. This
adds an info-level message with the project name so that well-behaved
runs of manage-projects have enough context to give meaningful logs.
Change-Id: Ibea0a8d8367da1b6e20d244e8117ddc241337e88
Clark Boylan