Commit Graph

93 Commits

Author SHA1 Message Date
Andreas Jaeger 0ec5a78f66 Sync with governance
Run script generate-gitmodules.py to sync with governace repository to
add new repositories and remove retired ones.

Change-Id: Ia42b717ad242958ba4a9689ac814c956f460712f
2020-06-07 16:15:34 +02:00
Zuul 40a6cbac3e Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Inspect all configs in manage-projects"
  - Inspect all configs in manage-projects
    
    We were skipping the setting of retired project acls in manage-projects
    because we were iterating over projects that weren't retired. In
    manage-projects we don't want to skip over retired projects so that we
    can retire the projects properly.
    
    Address this by adding a new registry property that returns all projects
    retired or not then iterate over that in manage-projects.
    
    Change-Id: I7aa89b53696fb0fca262fc7068e5e2ad2b7ad8fc
2020-05-04 15:43:38 +00:00
Monty Taylor 656be7c905 Update git submodules
* Update jeepyb from branch 'master'
  - Decode utf-8 from subprocess.Popen
    
    We use subprocess.Popen directly in these scripts rather than
    run_command. We need to decode the output so that we get strings
    back and not bytes.
    
    Change-Id: Ie967ceac77ecb898cab24827644b981eed22e141
2020-04-11 16:24:28 +00:00
Monty Taylor 51cebbd152 Update git submodules
* Update jeepyb from branch 'master'
  - Fix issues from rolling out containers
    
    We need to be able to set the location of GERRIT_GIT_DIR via
    env vars. And we need to turn off strict for config parser.
    
    Change-Id: I9d511caa6db6f958a215f131692e185c717b66d5
2020-04-11 14:25:21 +00:00
Monty Taylor d6e819f602 Update git submodules
* Update jeepyb from branch 'master'
  - Username is on the connection objet
    
    We're manipulating a Gerrit object not a GerritConnection object.
    The username property is found on the Gerrit object.
    
    Change-Id: I9b4c4cc0dacf456c1cb71c3798141ac726b1db71
2020-03-30 14:53:21 +00:00
Jeremy Stanley b42c357b8d Update git submodules
* Update jeepyb from branch 'master'
  - Catch exceptions when checking for groups
    
    Work around gerritlib raising a generic "Exception" exception when
    listGroup() finds no group.
    
    Change-Id: I63de5aac729b29366b37f1304369419328f4a051
2020-03-29 23:14:18 +00:00
Monty Taylor 4e6232baf8 Update git submodules
* Update jeepyb from branch 'master'
  - More python3 fixes
    
    Return strings from run_command
    
    jeepyb is written expecting strings. Decode the bytes into a
    string.
    
    Fix fsck_command for python3
    
    We fixed run_command to return strings, but fsck_command was
    looking for bytes.
    
    Filter returns an interator not a list in python3, wrap in a
    list.
    
    Change-Id: Ibb4d540a987711bd6b4c804a3f6b7cb2ccfe1baa
2020-03-27 23:42:22 +00:00
Monty Taylor fcc8474d09 Update git submodules
* Update jeepyb from branch 'master'
  - Encode the input to hashlib
    
    Another python3-ism.
    
    Change-Id: Ide54c16fd75b1e06b45a926b771d65575d50ca2e
2020-03-25 21:42:12 +00:00
Monty Taylor 42ad9ddd00 Update git submodules
* Update jeepyb from branch 'master'
  - Update StringIO to use six
    
    It's StringIO.StringIO in python2 and io.StringIO in python3,
    but six.StringIO everywhere.
    
    Change-Id: Ief0b091b716e3ea6224c0ac374a6066f0270bebc
2020-03-25 01:12:30 +00:00
Monty Taylor d2bc82daa3 Update git submodules
* Update jeepyb from branch 'master'
  - Trigger new images
    
    Change-Id: I428b46af2c04867582f170b48c64fc6ff087052c
2020-03-24 21:56:41 +00:00
Zuul b40c524d59 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Clean up some more python3 things"
  - Clean up some more python3 things
    
    Because system commands are bytes not strings. Because yay.
    
    Change-Id: Ib54b5717c2e2051addc2d7208a70b82a66ca8ccb
2020-03-24 15:08:48 +00:00
Zuul e3c50e8e74 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Run pep8 job in-repo"
  - Run pep8 job in-repo
    
    We removed the pep8 job from project-config, let's add
    it back to our local .zuul.yaml.
    
    Depends-On: https://review.opendev.org/714516
    Change-Id: I5a9a65569e4103914e64a09c47104d0e3d7c5fe1
2020-03-23 19:21:34 +00:00
Zuul e91a47c9d7 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Update ConfigParser imports for python3"
  - Update ConfigParser imports for python3
    
    We're starting to run jeepyb under python3 in the new gerrit image.
    
    Change-Id: Ia71cb13f3b45055bbb9a311807fa65a89c8be62a
2020-03-22 14:17:56 +00:00
Zuul ac974e13c5 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Run gerritlib and Gerrit integration test"
  - Run gerritlib and Gerrit integration test
    
    This gives us functional integration testing of gerritlib and gerrit
    with manage-projects. This could be expanded further to test more
    functionality too.
    
    Depends-On: https://review.opendev.org/#/c/704623/
    Change-Id: I739807b26ccd89f9bf97b3706bcf567a9d790dd7
2020-03-22 14:17:56 +00:00
Zuul 15aad1d7e8 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Log errors when committing and pushing acls"
  - Log errors when committing and pushing acls
    
    For some reason we don't log the git output so failures are opaque to
    us. Fix that by adding additonal logging.
    
    Change-Id: If212ae6115c8266e1352431588c04547c897e982
2020-03-21 21:44:42 +00:00
Zuul a0958cfd2f Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Don't use the db in manage-projects"
  - Don't use the db in manage-projects
    
    Direct access to the db restricts where we can run manage-projects from
    and complicates testing. Instead rely on the list-groups ssh command to
    get group uuid info. Note that this should be tested using new
    integration testing before we merge it.
    
    Change-Id: I6e3f0881398da8efeab5dfef370ad8d35163a7be
2020-03-21 21:44:41 +00:00
Jeremy Stanley 11411b662e Update git submodules
* Update jeepyb from branch 'master'
  - Overhaul default welcome message for OpenDev
    
    The "Welcome, new contributor!" hook bot should leave a more generic
    message relevant to most OpenDev projects, and not something
    OpenStack-specific (as this could easily confuse new contributors to
    non-OpenStack projects in OpenDev).
    
    Change-Id: Ifc41e0639adf27a476025e66728d547a71ad99d5
2020-02-25 19:49:03 +00:00
Monty Taylor f5e4bf650b Update git submodules
* Update jeepyb from branch 'master'
  - Don't index retired projects
    
    While they should just have a README, it's still a waste of energy
    for codesearch to index them in the first place.
    
    Remove the special condition for deb- repos - they are all retired.
    
    Change-Id: I60d90fccb607fcd48de42739188af5fb241ded7f
2020-01-07 20:50:07 +00:00
Monty Taylor 00990b4e5d Update git submodules
* Update jeepyb from branch 'master'
  - Fix create_hound_config for python3
    
    pep8 did a business!
    
    Change-Id: I1ffc99e30a0b5fc41c1f2485546b10553c525915
2020-01-07 20:50:06 +00:00
Ian Wienand 87d06fd184 Update git submodules
* Update jeepyb from branch 'master'
  - Hound config: disambiguate key names for projects in new namespaces
    
    Because we use os.path.basename() for the key in the config
    dictionary, openstack/project-config and zuul/project-config map to
    the same key and overwrite each other, thus we only get indexing on
    one or the other.
    
    Use the full project name as the key instead.
    
    Change-Id: I8a32ddc2e24211cc9300a2f90df19e1354251ae5
2019-05-13 16:31:54 +00:00
Monty Taylor 25ce38ea2b Update git submodules
* Update jeepyb from branch 'master'
  - Use opendev and https by default
    
    We are currently defaulting the hound config to using git://.
    Change that to https and change the git host to opendev.org.
    
    Also, change the path and anchor so that they will be correct
    for gitea.
    
    Change-Id: Icd1eef1a5476fbb7e0fff97e0d401b79e371fb36
    
  - OpenDev Migration Patch
    
    This commit was bulk generated and pushed by the OpenDev sysadmins
    as a part of the Git hosting and code review systems migration
    detailed in these mailing list posts:
    
    http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
    http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html
    
    Attempts have been made to correct repository namespaces and
    hostnames based on simple pattern matching, but it's possible some
    were updated incorrectly or missed entirely. Please reach out to us
    via the contact information listed at https://opendev.org/ with any
    questions you may have.
2019-04-23 14:54:46 +00:00
Jeremy Stanley df732e6182 Update git submodules
* Update jeepyb from branch 'master'
  - Correct blueprint topic links for modern Gerrit
    
    The Gerrit query URL pattern has changed, so correct how we form it
    in the update_blueprint hook to match expectations of current Gerrit
    versions.
    
    Change-Id: I368d0d30451051ab88a7b07dc6a2fbe7c4ddb886
    
  - manage_projects: exit with error code
    
     * Currently, manage_projects always perform
       `exit 0`, even if some errors were.
     * Patch would add `exit 1` beh., in
       case any issues were during processing.
       It would affect only exit code , and will not
       stop processing.
     * Misc: perform auto-ident
    
    Change-Id: I50d425e5a9e02441a50f7b1d32219485f80e6632
    
  - Use https for links
    
    Use https instead of http when link to docs.openstack.org.
    
    Change-Id: Iaf643a1c110aac53cd036537a583d58eebbf817d
    
  - Remove leading whitespaces from the Gerrit welcome message
    
    Change-Id: I02f02b361ded18d21013c2d8c58986340a6e5dfb
2019-04-09 12:55:05 +00:00
James E. Blair 13ce1523da Update git submodules
* Update jeepyb from branch 'master'
  - Support cgit alias sites
    
    This allows creation of new top-level sites and aliasing of
    existing repos into those sites (under arbitrary names).
    
    For example, this will let us list only the zuul projects under
    git.zuul-ci.org.  This scheme does not extend to the git protocol,
    which we will start to deprecate.
    
    Change-Id: I4c759f02d8d7e77439984d8ad1f012bd381fae59
    Story: 2001382
    Task: 6092
2018-03-23 22:33:38 +00:00
Clark Boylan 5c305d16f0 Update git submodules
* Update jeepyb from branch 'master'
  - Handle newer gerrit in update blueprint script
    
    Newer gerrit provides the project name and the branch the change is
    proposed against as well as the change id itself in the --change
    argument to patchset created hooks. This is a behavior change that we
    have to handle as old gerrit passed only the change id.
    
    We do this by going to the old behavior of the script by splitting off
    the change id from the new string and using only that. Note that this
    may not be strictly correct as multiple changes can share a change id
    (likely why gerrit made this change in the first place). We can worry
    about properly correct behavior in future updates.
    
    Change-Id: Idef56e98ed6c753a58b766024295b2f5147e3aea
2017-09-25 15:38:25 +00:00
Jenkins 64c00db565 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Remove self from group after group creation"
  - Remove self from group after group creation
    
    Newer gerrit automatically adds the user creating a group to that group.
    We don't actually want this behavior as the user used to manage this is
    used for nothing else. Check if the current user was added to the group
    and if so remove it.
    
    Depends-On: Id5d8915f2c0bf6f6057b414785b8ec1b18e810f2
    Change-Id: Ia0ca5faaaa466ae7a930bbbfb78e2975771f3af9
2017-09-22 21:37:32 +00:00
Monty Taylor 1e3b163336 Update git submodules
* Update jeepyb from branch 'master'
  - Remove a retired reference missed in previous refactor
    
    We pulled the application of retired up into util, but didn't delete
    this reference to the variable in the tuple.
    
    Change-Id: I190204fbd4d4c0edf5d116937e8240f94432832d
2017-09-22 18:17:04 +00:00
Jenkins 931cb8a74e Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Revert "Add description to needs_update calculations""
  - Revert "Add description to needs_update calculations"
    
    This reverts commit f12da781327ba6aaa87debb91326bcdcd1c5f426.
    
    Change-Id: I0bdd154deea16f345f3ba1aed15c419393bb6d93
2017-09-22 15:13:15 +00:00
Jenkins 0db8212514 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Remove extra writing of description"
  - Remove extra writing of description
    
    This snuck in via bad merge resolution.
    
    Change-Id: If2a737580f7cca16a9e8889d1850657a5ab61490
2017-09-22 15:10:55 +00:00
Jenkins 7161c93d84 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Include -attic in retired calculations"
  - Include -attic in retired calculations
    
    The retired thing works pretty well with acls - but our attic repos
    are different. Retire them too.
    
    Change-Id: Id25743db58ce4a1e3ba42e84fc298e737ec5f8a3
2017-09-21 16:03:37 +00:00
Jenkins 5c8c5daa1d Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Make retired acl check less greedy"
  - Make retired acl check less greedy
    
    Change-Id: I03ef4483f14b41e52b07e40cabd573a3b4a3eef5
2017-09-20 18:56:15 +00:00
Jenkins 097bafdabf Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Don't include retired projects for codesearch"
  - Don't include retired projects for codesearch
    
    They just take up extra space.
    
    Change-Id: I105519f577cefac207743bcbc9bfc096798e4dff
2017-09-20 18:31:31 +00:00
Jenkins fa1721d487 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Stop showing retired repos in cgit"
  - Stop showing retired repos in cgit
    
    There are projects that are no-longer with us. Let's not broadcast
    them to the world in our cgit mirrors.
    
    Change-Id: Ie2d66fb8f41dd7035a209b48721b08c4f55fd75a
2017-09-20 18:30:00 +00:00
Jenkins 03b3ad02ac Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Add description to needs_update calculations"
  - Add description to needs_update calculations
    
    We're caching the data already. If we at this, it'll cause us to update
    descriptions in github when they change.
    
    Change-Id: I1941958cca6387bd17fd3dcfd3c519af8b355453
2017-09-20 18:28:13 +00:00
Jenkins 6084e29ed6 Update git submodules
* Update jeepyb from branch 'master'
  - Merge "Don't mark as created in github until we're done"
  - Don't mark as created in github until we're done
    
    created-in-github being True prevents the subsequent github actions
    from happening. If we fail updating the github project after creation
    though, with the current code we can be left with half-created projects.
    
    Don't set it until we've actually make it through.
    
    Change-Id: I159eda5d1a3e5a30eed5f322d1e29477a80c5bee
2017-09-20 18:17:55 +00:00
Jeremy Stanley 8e3f44f313 Updated openstack/openstack
Project: openstack-infra/jeepyb  63f0ba511572149c41a9affebddb0ef3c47140f9

Update bugs only with active Gerrit accounts

When determining the LP assignee for a bug, constrain the Gerrit DB
query to only active accounts. Otherwise an incorrect OpenId for a
defunct account may be selected instead of the intended one.

Change-Id: I34e64362dd89c1d678a1f641244274fe27b5bdca
2017-09-06 20:03:47 +00:00
Sean Dague b4fbf4505c Updated openstack/openstack
Project: openstack-infra/jeepyb  3da5fcba0a204bf4bd41edf99c5c55954b558042

strip out newlines when writing repo.desc

If newlines end up in the description this breaks cgit hard, strip
those out.

Change-Id: I0f77c36503c522b50e6ae690acd5ff9c6caa7bec
2017-06-26 16:48:22 +00:00
Andreas Jaeger ff3067a10a Updated openstack/openstack
Project: openstack-infra/jeepyb  135363a63584601869850eaaa69a36dc031c3f47

Fix startswith usage

Fix my previous change, startswith needs a tuple, not a list.

Change-Id: I3df0e711dae7553be1946ece5b41d6407986f588
2017-03-24 17:20:56 +00:00
Jenkins 4dcf99a5fd Updated openstack/openstack
Project: openstack-infra/jeepyb  b88dafc17d08111cd7b8d6448c8929c69944108a

If we're going to apply acls, we need a dir

We need the git repo to exist if we need to send in acls.

Change-Id: I29b0e8dae152e41c1a9e0556a74caa5f70813706
2017-03-23 20:32:53 +00:00
Jenkins 475c4dac6f Updated openstack/openstack
Project: openstack-infra/jeepyb  b83cb8446b736611945879cc514205383ddcc3e0

Codesearch: Ignore dead repos

Ignore all dead repos that are in openstack-attic and stackforge
namespaces - including stackforge-attic.

Codesearch does not need to index and show these as they are dead.

Change-Id: Ie752399791ac1b650045516acb038f7313a182db
2017-03-23 18:48:16 +00:00
Jenkins b01039f705 Updated openstack/openstack
Project: openstack-infra/jeepyb  f8e4f4450603edd105195a213427891068ae859b

hound: ignore deb-* projects

They are forks for actual projects, and searching against those repos is
wasteful and produces duplicate entries for projects that actually
maintain the code. Upstream developers are usually not interested in
searching through those repos.

This patch ignores those repos completely.

Change-Id: I53d0e129a13f4ca1c7a6dfe189ce33e61f0778b4
2017-03-23 18:48:11 +00:00
Jenkins ec9c2d656c Updated openstack/openstack
Project: openstack-infra/jeepyb  205a1f5dcc91aa6d2ab8c9a316de4f165063d819

Deduplicate some more code

fsck_repo is used in both files, so move it to utils. push_to_gerrit is
only used in manage-projects and sync_upstream is only used in
track-upstream.

Change-Id: Ia81c8a54023d5f0cac6443a6e28475b0d2243aa0
2017-03-23 18:39:20 +00:00
Jenkins ba07bcfef5 Updated openstack/openstack
Project: openstack-infra/jeepyb  61ee4bd6737a4a019e3757db02bfb7d78f6dfa46

Allow to customize values in .gitreview file

There might be situation when project creation can be done from
other "endpoint" then common users have access. So we need to have
an ability to customise GERRIT_HOST and GERRIT_PORT parameters
inside .gitreview file for newly created project.

This change adds two parameters, that can be re-defined in
projects.ini file to format correct connection string to gerrit:
  * GITREVIEW_GERRIT_HOST (by default = GERRIT_HOST)
  * GITREVIEW_GERRIT_PORT (by default = GERRIT_PORT)

Change-Id: Ie3fc35c5b6714c579314faa18160594aeae79cb7
2017-02-17 14:46:15 +00:00
Jenkins 9765539bcd Updated openstack/openstack
Project: openstack-infra/jeepyb  9b08e078212e0f960bd8a8b0e43b3a92cd991ef4

Make protocol configurable

Sometimes, someone might want to clone using https

Change-Id: I6cb9f1d698a100742185c7f73e081d37efecf7e6
2017-02-17 14:44:54 +00:00
Jenkins ee0477a9e5 Updated openstack/openstack
Project: openstack-infra/jeepyb  1a1e6bc75be72abb3eb28d2e86443411de36c016

Only expire open reviews

Closed, Merged, Abandoned reviews don't need to
be re-expired. Only consider patches that are 'open'.

Change-Id: I848a7949a00ef700c1687493a6bd26a7a9508cfd
2017-02-17 14:43:04 +00:00
Jenkins a94a52ad0e Updated openstack/openstack
Project: openstack-infra/jeepyb  a9728a54e1fc66796730b75c8978db30b8fd20cd

Move make_local_copy to utils

This is also shared by manage-projecs and track-upstream - but there was
also a change to its logic in the former patch which would be hard to
see if we moved it and changed it in the same patch.

Change-Id: Iab287596d1e0db529e9557cc057d558d5b7ecccb
2017-02-17 14:42:31 +00:00
Jenkins 7f909df1ba Updated openstack/openstack
Project: openstack-infra/jeepyb  9ae0c0629955345f6bdf2e6f44c9e31548598fc9

Skip retry loop on first check for group

Don't retry checking the DB multiple times on first call to see if group
exists when determining whether the group needs to be created.

If the group exists, a previous iteration would already have waited for
the group to be written down from the cache. So if nothing is returned
then the group does not exist, and there is no point rechecking multiple
times to see if it will appear from the cache.

This speeds up creation of new groups by avoiding a 10 second wait
before creation due to the first call for group uuid unnecessarily
retrying multiple times.

Change-Id: I0afbc716159e8aecf1ade6442d9b02674094fa08
2017-02-17 14:42:01 +00:00
Monty Taylor dd4f2348b9 Updated openstack/openstack
Project: openstack-infra/jeepyb  a4b8f3668baa5cc065a207a2183898c64fe15f15

Correct a boolean inversion for github updating

The not is important. Also, add in some more logging for when it does
happen.

Change-Id: I3b2a8779a3d699f72682c93f418b498506b86a01
2017-02-17 14:32:47 +00:00
Monty Taylor 0774a0e827 Updated openstack/openstack
Project: openstack-infra/jeepyb  fa743024d37118ae8705fc0420d834e5cc5279c2

Split track-upstream from manage-projects

We spend a very large amount of time in track-upstream code now because
of all of the debian repos. That's fine - but it makes iterating on
project creation issues exceptionally difficult. Also, now that we're
caching the flag as to whether or not the project is created,
track-upstream can read it and know whether or not it's safe to run for
a given project.

Change-Id: I454b1ba400dc86abcc9b939564eb4eb7c324308c
2017-02-17 14:32:27 +00:00
Monty Taylor 7b947bcc20 Updated openstack/openstack
Project: openstack-infra/jeepyb  6d1ba2f1696a9f8f502e39b804505cfa2a690e22

Continue to optimize manage-projects for resiliency

Saving the state for having created the project was sticking to false.
If we had the git repo locally but it wasn't in gerrit, the logic was
all message up. Checking for homepage and downloads in github kills us -
although it would be fine if we'd started from caching homepage and
description in the first place, so maybe we need a script that sets
status quo in the cache file.

Change-Id: I15802082c0bef9e9362eb9482b058f81d1e566d7
2017-02-17 14:07:53 +00:00
Monty Taylor 2ed0062e2a Updated openstack/openstack
Project: openstack-infra/jeepyb  182fede1b43d63280c137bfcb60b022be51953b5

Make a few less github API calls

We continue to hit the GH rate limit. But honestly, in most cases, we
should not need to make a bazillion GH api calls.

Change-Id: Iac709a4f7f18d6348b21520bbc7aeacfa991ec1a
2017-02-13 17:57:04 +00:00