Run script generate-gitmodules.py to sync with governace repository to
add new repositories and remove retired ones.
Change-Id: Ia42b717ad242958ba4a9689ac814c956f460712f
* Update jeepyb from branch 'master'
- Merge "Inspect all configs in manage-projects"
- Inspect all configs in manage-projects
We were skipping the setting of retired project acls in manage-projects
because we were iterating over projects that weren't retired. In
manage-projects we don't want to skip over retired projects so that we
can retire the projects properly.
Address this by adding a new registry property that returns all projects
retired or not then iterate over that in manage-projects.
Change-Id: I7aa89b53696fb0fca262fc7068e5e2ad2b7ad8fc
* Update jeepyb from branch 'master'
- Decode utf-8 from subprocess.Popen
We use subprocess.Popen directly in these scripts rather than
run_command. We need to decode the output so that we get strings
back and not bytes.
Change-Id: Ie967ceac77ecb898cab24827644b981eed22e141
* Update jeepyb from branch 'master'
- Fix issues from rolling out containers
We need to be able to set the location of GERRIT_GIT_DIR via
env vars. And we need to turn off strict for config parser.
Change-Id: I9d511caa6db6f958a215f131692e185c717b66d5
* Update jeepyb from branch 'master'
- Username is on the connection objet
We're manipulating a Gerrit object not a GerritConnection object.
The username property is found on the Gerrit object.
Change-Id: I9b4c4cc0dacf456c1cb71c3798141ac726b1db71
* Update jeepyb from branch 'master'
- Catch exceptions when checking for groups
Work around gerritlib raising a generic "Exception" exception when
listGroup() finds no group.
Change-Id: I63de5aac729b29366b37f1304369419328f4a051
* Update jeepyb from branch 'master'
- More python3 fixes
Return strings from run_command
jeepyb is written expecting strings. Decode the bytes into a
string.
Fix fsck_command for python3
We fixed run_command to return strings, but fsck_command was
looking for bytes.
Filter returns an interator not a list in python3, wrap in a
list.
Change-Id: Ibb4d540a987711bd6b4c804a3f6b7cb2ccfe1baa
* Update jeepyb from branch 'master'
- Update StringIO to use six
It's StringIO.StringIO in python2 and io.StringIO in python3,
but six.StringIO everywhere.
Change-Id: Ief0b091b716e3ea6224c0ac374a6066f0270bebc
* Update jeepyb from branch 'master'
- Merge "Clean up some more python3 things"
- Clean up some more python3 things
Because system commands are bytes not strings. Because yay.
Change-Id: Ib54b5717c2e2051addc2d7208a70b82a66ca8ccb
* Update jeepyb from branch 'master'
- Merge "Run pep8 job in-repo"
- Run pep8 job in-repo
We removed the pep8 job from project-config, let's add
it back to our local .zuul.yaml.
Depends-On: https://review.opendev.org/714516
Change-Id: I5a9a65569e4103914e64a09c47104d0e3d7c5fe1
* Update jeepyb from branch 'master'
- Merge "Update ConfigParser imports for python3"
- Update ConfigParser imports for python3
We're starting to run jeepyb under python3 in the new gerrit image.
Change-Id: Ia71cb13f3b45055bbb9a311807fa65a89c8be62a
* Update jeepyb from branch 'master'
- Merge "Run gerritlib and Gerrit integration test"
- Run gerritlib and Gerrit integration test
This gives us functional integration testing of gerritlib and gerrit
with manage-projects. This could be expanded further to test more
functionality too.
Depends-On: https://review.opendev.org/#/c/704623/
Change-Id: I739807b26ccd89f9bf97b3706bcf567a9d790dd7
* Update jeepyb from branch 'master'
- Merge "Log errors when committing and pushing acls"
- Log errors when committing and pushing acls
For some reason we don't log the git output so failures are opaque to
us. Fix that by adding additonal logging.
Change-Id: If212ae6115c8266e1352431588c04547c897e982
* Update jeepyb from branch 'master'
- Merge "Don't use the db in manage-projects"
- Don't use the db in manage-projects
Direct access to the db restricts where we can run manage-projects from
and complicates testing. Instead rely on the list-groups ssh command to
get group uuid info. Note that this should be tested using new
integration testing before we merge it.
Change-Id: I6e3f0881398da8efeab5dfef370ad8d35163a7be
* Update jeepyb from branch 'master'
- Overhaul default welcome message for OpenDev
The "Welcome, new contributor!" hook bot should leave a more generic
message relevant to most OpenDev projects, and not something
OpenStack-specific (as this could easily confuse new contributors to
non-OpenStack projects in OpenDev).
Change-Id: Ifc41e0639adf27a476025e66728d547a71ad99d5
* Update jeepyb from branch 'master'
- Don't index retired projects
While they should just have a README, it's still a waste of energy
for codesearch to index them in the first place.
Remove the special condition for deb- repos - they are all retired.
Change-Id: I60d90fccb607fcd48de42739188af5fb241ded7f
* Update jeepyb from branch 'master'
- Hound config: disambiguate key names for projects in new namespaces
Because we use os.path.basename() for the key in the config
dictionary, openstack/project-config and zuul/project-config map to
the same key and overwrite each other, thus we only get indexing on
one or the other.
Use the full project name as the key instead.
Change-Id: I8a32ddc2e24211cc9300a2f90df19e1354251ae5
* Update jeepyb from branch 'master'
- Use opendev and https by default
We are currently defaulting the hound config to using git://.
Change that to https and change the git host to opendev.org.
Also, change the path and anchor so that they will be correct
for gitea.
Change-Id: Icd1eef1a5476fbb7e0fff97e0d401b79e371fb36
- OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.htmlhttp://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html
Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
* Update jeepyb from branch 'master'
- Correct blueprint topic links for modern Gerrit
The Gerrit query URL pattern has changed, so correct how we form it
in the update_blueprint hook to match expectations of current Gerrit
versions.
Change-Id: I368d0d30451051ab88a7b07dc6a2fbe7c4ddb886
- manage_projects: exit with error code
* Currently, manage_projects always perform
`exit 0`, even if some errors were.
* Patch would add `exit 1` beh., in
case any issues were during processing.
It would affect only exit code , and will not
stop processing.
* Misc: perform auto-ident
Change-Id: I50d425e5a9e02441a50f7b1d32219485f80e6632
- Use https for links
Use https instead of http when link to docs.openstack.org.
Change-Id: Iaf643a1c110aac53cd036537a583d58eebbf817d
- Remove leading whitespaces from the Gerrit welcome message
Change-Id: I02f02b361ded18d21013c2d8c58986340a6e5dfb
* Update jeepyb from branch 'master'
- Support cgit alias sites
This allows creation of new top-level sites and aliasing of
existing repos into those sites (under arbitrary names).
For example, this will let us list only the zuul projects under
git.zuul-ci.org. This scheme does not extend to the git protocol,
which we will start to deprecate.
Change-Id: I4c759f02d8d7e77439984d8ad1f012bd381fae59
Story: 2001382
Task: 6092
* Update jeepyb from branch 'master'
- Handle newer gerrit in update blueprint script
Newer gerrit provides the project name and the branch the change is
proposed against as well as the change id itself in the --change
argument to patchset created hooks. This is a behavior change that we
have to handle as old gerrit passed only the change id.
We do this by going to the old behavior of the script by splitting off
the change id from the new string and using only that. Note that this
may not be strictly correct as multiple changes can share a change id
(likely why gerrit made this change in the first place). We can worry
about properly correct behavior in future updates.
Change-Id: Idef56e98ed6c753a58b766024295b2f5147e3aea
* Update jeepyb from branch 'master'
- Merge "Remove self from group after group creation"
- Remove self from group after group creation
Newer gerrit automatically adds the user creating a group to that group.
We don't actually want this behavior as the user used to manage this is
used for nothing else. Check if the current user was added to the group
and if so remove it.
Depends-On: Id5d8915f2c0bf6f6057b414785b8ec1b18e810f2
Change-Id: Ia0ca5faaaa466ae7a930bbbfb78e2975771f3af9
* Update jeepyb from branch 'master'
- Remove a retired reference missed in previous refactor
We pulled the application of retired up into util, but didn't delete
this reference to the variable in the tuple.
Change-Id: I190204fbd4d4c0edf5d116937e8240f94432832d
* Update jeepyb from branch 'master'
- Merge "Remove extra writing of description"
- Remove extra writing of description
This snuck in via bad merge resolution.
Change-Id: If2a737580f7cca16a9e8889d1850657a5ab61490
* Update jeepyb from branch 'master'
- Merge "Include -attic in retired calculations"
- Include -attic in retired calculations
The retired thing works pretty well with acls - but our attic repos
are different. Retire them too.
Change-Id: Id25743db58ce4a1e3ba42e84fc298e737ec5f8a3
* Update jeepyb from branch 'master'
- Merge "Don't include retired projects for codesearch"
- Don't include retired projects for codesearch
They just take up extra space.
Change-Id: I105519f577cefac207743bcbc9bfc096798e4dff
* Update jeepyb from branch 'master'
- Merge "Stop showing retired repos in cgit"
- Stop showing retired repos in cgit
There are projects that are no-longer with us. Let's not broadcast
them to the world in our cgit mirrors.
Change-Id: Ie2d66fb8f41dd7035a209b48721b08c4f55fd75a
* Update jeepyb from branch 'master'
- Merge "Add description to needs_update calculations"
- Add description to needs_update calculations
We're caching the data already. If we at this, it'll cause us to update
descriptions in github when they change.
Change-Id: I1941958cca6387bd17fd3dcfd3c519af8b355453
* Update jeepyb from branch 'master'
- Merge "Don't mark as created in github until we're done"
- Don't mark as created in github until we're done
created-in-github being True prevents the subsequent github actions
from happening. If we fail updating the github project after creation
though, with the current code we can be left with half-created projects.
Don't set it until we've actually make it through.
Change-Id: I159eda5d1a3e5a30eed5f322d1e29477a80c5bee
Project: openstack-infra/jeepyb 63f0ba511572149c41a9affebddb0ef3c47140f9
Update bugs only with active Gerrit accounts
When determining the LP assignee for a bug, constrain the Gerrit DB
query to only active accounts. Otherwise an incorrect OpenId for a
defunct account may be selected instead of the intended one.
Change-Id: I34e64362dd89c1d678a1f641244274fe27b5bdca
Project: openstack-infra/jeepyb 3da5fcba0a204bf4bd41edf99c5c55954b558042
strip out newlines when writing repo.desc
If newlines end up in the description this breaks cgit hard, strip
those out.
Change-Id: I0f77c36503c522b50e6ae690acd5ff9c6caa7bec
Project: openstack-infra/jeepyb 135363a63584601869850eaaa69a36dc031c3f47
Fix startswith usage
Fix my previous change, startswith needs a tuple, not a list.
Change-Id: I3df0e711dae7553be1946ece5b41d6407986f588
Project: openstack-infra/jeepyb b88dafc17d08111cd7b8d6448c8929c69944108a
If we're going to apply acls, we need a dir
We need the git repo to exist if we need to send in acls.
Change-Id: I29b0e8dae152e41c1a9e0556a74caa5f70813706
Project: openstack-infra/jeepyb b83cb8446b736611945879cc514205383ddcc3e0
Codesearch: Ignore dead repos
Ignore all dead repos that are in openstack-attic and stackforge
namespaces - including stackforge-attic.
Codesearch does not need to index and show these as they are dead.
Change-Id: Ie752399791ac1b650045516acb038f7313a182db
Project: openstack-infra/jeepyb f8e4f4450603edd105195a213427891068ae859b
hound: ignore deb-* projects
They are forks for actual projects, and searching against those repos is
wasteful and produces duplicate entries for projects that actually
maintain the code. Upstream developers are usually not interested in
searching through those repos.
This patch ignores those repos completely.
Change-Id: I53d0e129a13f4ca1c7a6dfe189ce33e61f0778b4
Project: openstack-infra/jeepyb 205a1f5dcc91aa6d2ab8c9a316de4f165063d819
Deduplicate some more code
fsck_repo is used in both files, so move it to utils. push_to_gerrit is
only used in manage-projects and sync_upstream is only used in
track-upstream.
Change-Id: Ia81c8a54023d5f0cac6443a6e28475b0d2243aa0
Project: openstack-infra/jeepyb 61ee4bd6737a4a019e3757db02bfb7d78f6dfa46
Allow to customize values in .gitreview file
There might be situation when project creation can be done from
other "endpoint" then common users have access. So we need to have
an ability to customise GERRIT_HOST and GERRIT_PORT parameters
inside .gitreview file for newly created project.
This change adds two parameters, that can be re-defined in
projects.ini file to format correct connection string to gerrit:
* GITREVIEW_GERRIT_HOST (by default = GERRIT_HOST)
* GITREVIEW_GERRIT_PORT (by default = GERRIT_PORT)
Change-Id: Ie3fc35c5b6714c579314faa18160594aeae79cb7
Project: openstack-infra/jeepyb 1a1e6bc75be72abb3eb28d2e86443411de36c016
Only expire open reviews
Closed, Merged, Abandoned reviews don't need to
be re-expired. Only consider patches that are 'open'.
Change-Id: I848a7949a00ef700c1687493a6bd26a7a9508cfd
Project: openstack-infra/jeepyb a9728a54e1fc66796730b75c8978db30b8fd20cd
Move make_local_copy to utils
This is also shared by manage-projecs and track-upstream - but there was
also a change to its logic in the former patch which would be hard to
see if we moved it and changed it in the same patch.
Change-Id: Iab287596d1e0db529e9557cc057d558d5b7ecccb
Project: openstack-infra/jeepyb 9ae0c0629955345f6bdf2e6f44c9e31548598fc9
Skip retry loop on first check for group
Don't retry checking the DB multiple times on first call to see if group
exists when determining whether the group needs to be created.
If the group exists, a previous iteration would already have waited for
the group to be written down from the cache. So if nothing is returned
then the group does not exist, and there is no point rechecking multiple
times to see if it will appear from the cache.
This speeds up creation of new groups by avoiding a 10 second wait
before creation due to the first call for group uuid unnecessarily
retrying multiple times.
Change-Id: I0afbc716159e8aecf1ade6442d9b02674094fa08
Project: openstack-infra/jeepyb a4b8f3668baa5cc065a207a2183898c64fe15f15
Correct a boolean inversion for github updating
The not is important. Also, add in some more logging for when it does
happen.
Change-Id: I3b2a8779a3d699f72682c93f418b498506b86a01
Project: openstack-infra/jeepyb fa743024d37118ae8705fc0420d834e5cc5279c2
Split track-upstream from manage-projects
We spend a very large amount of time in track-upstream code now because
of all of the debian repos. That's fine - but it makes iterating on
project creation issues exceptionally difficult. Also, now that we're
caching the flag as to whether or not the project is created,
track-upstream can read it and know whether or not it's safe to run for
a given project.
Change-Id: I454b1ba400dc86abcc9b939564eb4eb7c324308c
Project: openstack-infra/jeepyb 6d1ba2f1696a9f8f502e39b804505cfa2a690e22
Continue to optimize manage-projects for resiliency
Saving the state for having created the project was sticking to false.
If we had the git repo locally but it wasn't in gerrit, the logic was
all message up. Checking for homepage and downloads in github kills us -
although it would be fine if we'd started from caching homepage and
description in the first place, so maybe we need a script that sets
status quo in the cache file.
Change-Id: I15802082c0bef9e9362eb9482b058f81d1e566d7
Project: openstack-infra/jeepyb 182fede1b43d63280c137bfcb60b022be51953b5
Make a few less github API calls
We continue to hit the GH rate limit. But honestly, in most cases, we
should not need to make a bazillion GH api calls.
Change-Id: Iac709a4f7f18d6348b21520bbc7aeacfa991ec1a