14 KiB

Raw Permalink Blame History

Importing from upstream: using git-upstream

Note: this guide assumes that you are using a branch named master to maintain your new features or bug fixes that sit on top of the upstream code of some project (probably somewhat related to OpenStack).

It is also assumed you are tracking releases, which is only one of the possible approaches to upstream tracking. Another approach would be tracking the master tip of a project. Of course even other strategies are possible.

Install git-upstream on a development workstation

Clone git-upstream from its git repository.

git clone https://github.com/stackforge/git-upstream.git
cd git-upstream
# Install git-upstream itself
python setup.py install

git clone https://github.com/stackforge/git-upstream.git
cd git-upstream
easy_install .

If you want command line completion (using tab), install the argcomplete package and source the provided "bash completion" file

mkdir ~/bin && cp ./bash_completion/git-upstream ~/bin
echo ". ~/bin/git-upstream" >> ~/.bash_profile
pip install argcomplete

Verify your installation.

pip show git-upstream
---
Name: git-upstream
Version: unknown-version
Location: ../ve-git-upstream/lib/python2.7/site-packages/git-upstream-unknown_version-py2.7.egg
Requires: GitPython

~# git-upstream --help
usage: git-upstream [--version] [-h] [-q | -v] <command> ...

[...]

Initial import of an upstream project

To explain the usage of the git-upstream tool We are going to use a real-world (but trivial) example.

We will be also using environment variables: in case you are keen on doing copy-paste, you will only need to change them to suite your environment.

export REPO_NAME="jenkins-job-builder"
export INTERNAL_REMOTE="https://your.org/my-projects/jenkins-job-builder.git"
export UPSTREAM_REMOTE="https://github.com/openstack-infra/jenkins-job-builder.git"
export FIRST_IMPORT_REF="0.5.0"

Create an empty repository

mkdir $REPO_NAME
cd $REPO_NAME
git init

Add your remotes

We will name it origin and upstream (for the sake of originality).

git remote add origin $INTERNAL_REMOTE
git remote add upstream $UPSTREAM_REMOTE

Fetch objects and refs from upstream remote

git fetch --all

Push refs

Push refs defined upstream to the origin remote (i.e., the internal copy of the repository with local patches) using the string upstream as prefix, also pushing tags.

git for-each-ref refs/remotes/upstream --format "%(refname:short)" | \
  sed -e 's:\(upstream/\(.*\)\)$:\1\:refs/heads/upstream/\2:' | \
  xargs git push --tags origin

You may want to repeat the last two commands before starting any new feature development or a bug fix.

Check-out the first import commit (e.g., tag or SHA1)

This will be the starting point for the internal development.

git checkout -b import/$FIRST_IMPORT_REF $FIRST_IMPORT_REF

Create and switch to the master branch

git checkout -b master

Now the tips of master, $FIRST_IMPORT_REF and import/$FIRST_IMPORT_REF should be pointing to the same commit.

Push local master branch to the remote origin, and make origin master the default when pushing commits.

git push -u origin master

Writing your patches/features

Now start to develop new feature or fix bugs on master, as usual. For this example, we are going to change the .gitreview file in order to use a local Gerrit server.

echo "diff --git a/.gitreview b/.gitreview index bb7e85c..f98d928 100644 --- a/.gitreview +++ b/.gitreview @@ -1,4 +1,4 @@ [Gerrit] -host=review.openstack.org +host=gerrit.my.org port=29418 -project=openstack-infra/jenkins-job-builder.git +project=my-projects/jenkins-job-builder.git" | patch -p1

Don’t forget to commit and push (after this step, you may want to use git review as usual)

git commit -a -m "Set .gitignore content to use internal gating infra"
git push

Our master (local and remote) tip should be now pointing to the last commit.

Importing single patches from upstream

Before implementing any feature or fixing any bug (in short, before reinventing the wheel), check if someone has already implemented the required code upstream.

If not, try not to develop code only for your specific needs, be ambitious and try to develop something that could be useful for the whole community. This way you can propose your patch upstream and save yourself a lot of trouble which arise when there are many local changes to carry on the tip of upstream releases.

In this example, we tried to use our code and we found out that the job filtering isn’t working! Fortunately, Antoine Musso has already fixed this bug, as we can see in the upstream repo.

git show --summary 2eca0d11669b55d4ab02ba609a15aa242fd80d14
commit 2eca0d11669b55d4ab02ba609a15aa242fd80d14
Author: Antoine Musso <hashar@free.fr>
Date:   Mon Jun 24 14:36:52 2013 +0200

    job filtering was not working properly

    When passing job names as arguments to 'update', the command is supposed
    to only retain this jobs.  Due to the job being a dict, the filter would
    never match and the none of the job would be updated.

    This has apparently always been broken since the feature got introduced
    in 85cf7a41.  Using job.['name'] fix it up.

    Change-Id: Icf4d5b0bb68777f7faff91ade04451d4c8501c6a
    Reviewed-on: https://review.openstack.org/34197
    Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
    Approved: James E. Blair <corvus@inaugust.com>
    Reviewed-by: James E. Blair <corvus@inaugust.com>
    Tested-by: Jenkins

We are also interested in the following commit, which adds the Environment File Plugin (finally!).

~# git show --summary bf4524fae25c11640ef839aa422ac81bd926ca20
commit bf4524fae25c11640ef839aa422ac81bd926ca20
Author: zaro0508 <zaro0508@gmail.com>
Date:   Mon Jul 1 11:21:24 2013 -0700

    add Environment File Plugin

    This commit adds the Environment File Plugin to JJB.
    https://wiki.jenkins-ci.org/display/JENKINS/Envfile+Plugin

    Change-Id: Id35a4d6ab25b0440303da02bb91007b459979243
    Reviewed-on: https://review.openstack.org/35170
    Reviewed-by: Arnaud Fabre <fabre.arnaud@gmail.com>
    Reviewed-by: James E. Blair <corvus@inaugust.com>
    Approved: Clark Boylan <clark.boylan@gmail.com>
    Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
    Tested-by: Jenkins

Import those changes simply cherry-picking the two commits. Don’t forget to push (review!) your changes.

git cherry-pick 2eca0d11669b55d4ab02ba609a15aa242fd80d14
git cherry-pick bf4524fae25c11640ef839aa422ac81bd926ca20
git push

Importing new versions from upstream

Days passes and finally a new releases comes out.

git fetch --all
git for-each-ref refs/remotes/upstream --format "%(refname:short)" | \
  sed -e 's:\(upstream/\(.*\)\)$:\1\:refs/heads/upstream/\2:' | \
  xargs git push --tags origin

A lot of work has been done upstream and we need to rebase our master onto the upstream master branch. In this process we must skip all the commits we already cherry-picked some days ago, of course.

Note: the rebasing for this example is trivial but it is just to break the ice. Later in this guide we will address rebasing conflicts that can occur in the real world.

Create a new local branch with the new release tag as a starting point

git branch import/0.6.0 0.6.0

Running git-upstream

Finally, it is time to run git-upstream! Before doing so make sure the current branch is master

git checkout master

~# git-upstream import import/0.6.0
Searching for previous import
Starting import of upstream
Successfully created import branch
Attempting to linearise previous changes
Successfully applied all locally carried changes
Merging import to requested branch 'HEAD'
Successfully finished import:
target branch: 'HEAD'
upstream branch: 'import/0.6.0'
import branch: 'import/0.6.0'

No errors, we have been lucky!

What has just happened?

git-upstream has created a new branch named import/0.6.0-base which tip is set to the commit pointed by the release tag 0.6.0, and has rebased all changes present in our local master which were not already present in the upstream new release (import/0.6.0-base) onto import/0.6.0-base.

You can see that running the following command

git log --graph --oneline --all --decorate

For this trivial example, the only commit not present in the upstream release was about the customisation of the .gitreview file.

The default strategy git-upstream uses to find duplicate entries is the comparison of Change-id entries in commit messages. Of course, it’s not possible to compare directly the SHA1 for a commit because the cherry-picking changes the information used for SHA1 calculation

Note: A git commit SHA1 is generated from the following information:

commit message
author signature (identity + timestamp)
committer signature (identity + timestamp)
tree SHA1 (hierarchy of directories and files within the commit)
list of the SHA1's of the parent commits

The local branch import/0.6.0 now contains our local changes rebased onto the new upstream release. git-upstream has also merged this branch with the local master branch (with "ours" strategy) to allow the normal workflow (committing/merging to master for review).

Note: The "final" merging step is not mandatory. Of course you can keep a separate branch for each new import. On one hand this strategy allows a "cleaner" history as you will always have your local changes rebased on top of the exact copy of the upstream repository. On the other hand you will be creating a new branch every time you want to import upstream code. You can customise the name of the import branch using the --import-branch <branch name> option.

In principle, you could also replace your master branch (history) with the new import branch created by git-upstream... Unfortunately there is no way to do this without requiring ad-hoc intervention on cloned copies of the repository (aka do-not-do-that(TM))

To disable automatic merging, just use the --no-merge flag

 git-upstream import --no-merge import/0.6.0

Handling conflicts

Of course n the real world things are much more complicated. From time to time, during import, you will get rebasing conflict (for instance due to changes from both local and upstream repository to the same piece of code).

In case of rebasing conflict, git-upstream will stop allowing the user to fix the conflict.

~# git-upstream import import/0.5.0 --into master
Searching for previous import
Starting import of upstream
Successfully created import branch
Attempting to linearise previous changes
ERROR   : Rebase failed, will need user intervention to resolve.
error: could not apply f9b4fca... Fixup for openstack review
When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".
Could not apply f9b4fca... Fixup for openstack review
Import cancelled

Let's find out why git-upstream failed and let's try to continue the rebasing manually.

~# git status
# HEAD detached from 8e6b9e9
# You are currently rebasing branch 'import/0.5.0' on '8e6b9e9'.
#   (fix conflicts and then run "git rebase --continue")
#   (use "git rebase --skip" to skip this patch)
#   (use "git rebase --abort" to check out the original branch)
#
# Unmerged paths:
#   (use "git reset HEAD <file>..." to unstage)
#   (use "git add <file>..." to mark resolution)
#
# both modified:      jenkins_jobs/cmd.py
# both modified:      jenkins_jobs/modules/hipchat_notif.py
#
no changes added to commit (use "git add" and/or "git commit -a")

Depending on the type of conflict, you will could:

drop the local change

Issuing git rebase --skip
edit conflicting code

Change conflicting code in order to accommodate local changes to the new upstream code. You can later resume rebasing process issuing git rebase --continue

Currently git-upstream can't resume the rebasing process. So, if needed, the final "merging" steps have to be performed manually:

git merge -s ours --no-commit <import-xxxx>

Replacing tree contents with those from the import branch

git read-tree <import-xxxx>

Committing merge commit

git commit --no-edit

Note: git-upstream performs exactly those steps in order to replace the content of master branch with the import branch preserving the history.

Integration with Gerrit

You may want to use review with Gerrit the output of git-upstream, in order to perform tests, gating, etc.

You have 2 options for doing that:

Re-review every new commit

In this case we want to review every new commit (since the last import). In order to do so, use the --no-merge flag of git-upstream import command, and:

git checkout import-xxxxx
git push gerrit import-xxxxx-base:import-xxxxx
git review import-xxxxx

If there is more than one new commit, git-review will ask to confirm the submission of multiple changes.

Re-review only the final merge commit

This would be possible by using the --import-branch option of import command and pushing directly (i.e.: bypassing Gerrit) the new branch to the local repo. For instance:

TIMESTAMP=$(date +"%Y%m%d%H%M%s")
git upstream import --import-branch "import/import-$TIMESTAMP" upstream/master
git push gerrit import/import-$TIMESTAMP:import/import-$TIMESTAMP

Then, create a valid Change-Id for the merge commit

git commit --amend -C HEAD --no-edit

Locally, git-review will still complain about the presence of N+M commits which would be committed BUT on the remote side all those commits will be recognised as already present in one of the two branch involved in the merge.

git review -R -y master

14 KiB Raw Permalink Blame History Unescape Escape