x
/
git-upstream
Code Issues Proposed changes

Add basic documentation and migrate README & USAGE 

Where documentation is being moved about, corrections will be avoided
to allow for easier following of conversion separated from rewriting.

* Set up some simple documentation to describe basic details of
  git-upstream and installation.
* Migrate details from README into dedicated subcommands rst doc.
* Consolidate installation instructions from USAGE into
  installation rst page.
* Move remainder of USAGE into dedicated workflows rst and match the
  existing markdown layout.

Change-Id: Ie73d64ae7cb0e96060729638da382c901d5d68e7
This commit is contained in:
Darragh Bailey 2016-11-07 18:17:33 +00:00
parent 360d839abe
commit 65cb8e2004
7 changed files with 756 additions and 632 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

177
README.rst
View File

@ -120,185 +120,12 @@ See also https://pypi.python.org/pypi/git-upstream
Using git-upstream
==================
Please see ``USAGE.md``
Please see `workflows <doc/source/workflows.rst>`_
Available commands
==================
import
------
Description
~~~~~~~~~~~
Import code from specified upstream branch. Creates an import branch
from the specified upstream branch, and optionally merges additional
branches given as arguments. Current branch, unless overridden by the
``--into`` option, is used as the target branch from which a list of
changes to apply onto the new import is constructed based on the
specified strategy. Once complete it will merge and replace the contents
of the target branch with those from the import branch, unless
``--no-merge`` is specified.
Usage
~~~~~
::
git upstream import [-h] [-d] [-i] [-f] [--merge] [--no-merge]
[-s <strategy>] [--into <branch>]
[--import-branch <import-branch>]
[<upstream-branch>] [<branches> [<branches> ...]]
Arguments
~~~~~~~~~
::
positional arguments:
<upstream-branch> Upstream branch to import. Must be specified if you
wish to provide additional branches.
<branches> Branches to additionally merge into the import branch
using default git merging behaviour
optional arguments:
-h, --help show this help message and exit
-d, --dry-run Only print out the list of commits that would be
applied.
-i, --interactive Let the user edit the list of commits before applying.
-f, --force Force overwrite of existing import branch if it
exists.
--merge Merge the resulting import branch into the target
branch once complete
--no-merge Disable merge of the resulting import branch
-s <strategy>, --strategy <strategy>
Use the given strategy to re-apply locally carried
changes to the import branch. (default: drop)
--into <branch> Branch to take changes from, and replace with imported
branch.
--import-branch <import-branch>
Name of import branch to use
drop
----
Description
~~~~~~~~~~~
Mark a commit as dropped. Marked commits will be skipped during the
upstream rebasing process.
See also the "git upstream import" command.
Usage
~~~~~
::
git upstream drop [-h] [-a <author>] <commit>
Arguments
~~~~~~~~~
::
positional arguments:
<commit> Commit to be marked as dropped
optional arguments:
-h, --help show this help message and exit
-a <author>, --author <author>
Git author for the mark
Note
~~~~
Commits will be marked with git notes in the namespace
``refs/notes/upstream-merge``.
To list of commit id marked with a note, run
``git notes --ref refs/notes/upstream-merge``.
To show a specific note run
``git notes --ref refs/notes/upstream-merge show <marked commit sha1>``
As ``drop`` uses git notes to mark commits that have to be skipped
during import, notes should be present on the cloned copy of your
repository. Thus, if you are going to create notes on a system and
perform the actual import on a different system, **notes must be present
on the latter**.
You can push notes directly to git repository on the target system or
push them in a different repository and then pull notes from your target
system.
supersede
---------
Description
~~~~~~~~~~~
Mark a commit as superseded by a set of change-ids. Marked commits will
be skipped during the upstream rebasing process **only if all the
specified change-ids are present in ``<upstream-branch>`` during
import**. If you want to unconditionally drop a commit, use the ``drop``
command instead.
See also the "git upstream import" command.
Usage
~~~~~
::
git upstream supersede [-h] [-f] [-u <upstream-branch>]
<commit> <change id> [<change id> ...]
Arguments
~~~~~~~~~
::
positional arguments:
<commit> Commit to be marked as superseded
<change id> Change id which makes <commit> obsolete. The change id
must be present in <upstream-branch> to drop <commit>.
If more than one change id is specified, all must be
present in <upstream-branch> to drop <commit>
optional arguments:
-h, --help show this help message and exit
-f, --force Apply the commit mark even if one or more change ids
could not be found. Use this flag carefully as commits
will not be dropped during import command execution as
long as all associated change ids are present in the
local copy of the upstream branch
-u <upstream-branch>, --upstream-branch <upstream-branch>
Search change ids values in <upstream-branch> branch
(default: upstream/master)
Note
~~~~
*This command doesn't perform the actual drop*. Commits to be dropped
during the next import, will be marked with git notes in the namespace
``refs/notes/upstream-merge``. There is no need to retain notes after an
import dropped the correspondent commits, of course it doesn't harm
keeping them either.
To list of commit id marked with a note, run
``git notes --ref refs/notes/upstream-merge``.
To show a specific note run
``git notes --ref refs/notes/upstream-merge show <marked commit sha1>``.
As ``supersede`` uses git notes to mark commits that have to be skipped
during import, notes should be present on the cloned copy of your
repository. Thus, if you are going to create notes on a system and
perform the actual import on a different system, **notes must be present
on the latter**. You can push notes directly to git repository on the
target system or push them in a different repository and then pull notes
from your target system.
Please see `subcommands <doc/source/subcommands.rst>`_
Authors
=======

456
USAGE.md
View File

@ -1,456 +0,0 @@
# Importing from upstream: using git-upstream
**Note**: this guide assumes that you are using a branch named *master* to
maintain your new features or bug fixes that sit on top of the upstream code of
some project (probably somewhat related to OpenStack).
It is also assumed you are tracking releases, which is only one of the possible
approaches to upstream tracking. Another approach would be tracking the master
tip of a project. Of course even other strategies are possible.
## Install git-upstream on a development workstation
Clone git-upstream from its git repository.
```bash
git clone https://git.openstack.org/openstack/git-upstream.git
cd git-upstream
# Install git-upstream itself
python setup.py install
```
Or
```bash
git clone https://git.openstack.org/openstack/git-upstream.git
cd git-upstream
easy_install .
```
If you want command line completion (using tab), install the argcomplete
package and source the provided "bash completion" file
```bash
mkdir ~/bin && cp ./bash_completion/git-upstream ~/bin
echo ". ~/bin/git-upstream" >> ~/.bash_profile
pip install argcomplete
```
Verify your installation.
```bash
pip show git-upstream
---
Name: git-upstream
Version: unknown-version
Location: ../ve-git-upstream/lib/python2.7/site-packages/git-upstream-unknown_version-py2.7.egg
Requires: GitPython
git-upstream --help
usage: git-upstream [--version] [-h] [-q | -v] <command> ...
[...]
```
## Initial import of an upstream project
To explain the usage of the git-upstream tool we are going to use a real-world
(but trivial) example, by performing some sample operations on a project called
``jenkins-job-builder``.
In this example, we will create a local file based Git repository to host our
mirror of jenkins-job-builder. You could also use an existing internal mirror,
a Github fork, etc.
Start by setting the following environment variables:
```bash
export REPO_NAME="jenkins-job-builder"
export INTERNAL_REMOTE="file:///tmp/jenkins-job-builder.git"
export UPSTREAM_REMOTE="https://github.com/openstack-infra/jenkins-job-builder.git"
export FIRST_IMPORT_REF="0.5.0"
```
1) Create two empty repositories, one to serve as your working copy, and one to
serve as the remote:
```bash
git init --bare /tmp/${REPO_NAME}.git
git init $REPO_NAME
cd $REPO_NAME
```
2) Add your remotes
We will name it *origin* and *upstream* (for the sake of originality).
```bash
git remote add origin $INTERNAL_REMOTE
git remote add upstream $UPSTREAM_REMOTE
```
3) Fetch objects and refs from upstream remote
```bash
git fetch --all
```
4) Push refs
Push refs defined upstream to the `origin` remote (*i.e.*, the internal copy
of the repository with local patches) using the string `upstream` as prefix,
also pushing tags.
```bash
git for-each-ref refs/remotes/upstream --format "%(refname:short)" | \
sed -e 's:\(upstream/\(.*\)\)$:\1\:refs/heads/upstream/\2:' | \
xargs git push --tags origin
```
You may want to repeat the last two commands before starting any new feature
development or a bug fix.
5) Check-out the first import commit (*e.g.*, tag or SHA1)
This will be the starting point for the internal development.
```bash
git checkout -b import/$FIRST_IMPORT_REF $FIRST_IMPORT_REF
```
6) Create and switch to the master branch
```bash
git checkout -b master
```
Now the tips of master, `$FIRST_IMPORT_REF` and `import/$FIRST_IMPORT_REF`
should be pointing to the same commit.
Push local master branch to the remote origin, and make `origin master` the
default when pushing commits.
```bash
git push -u origin master
```
## Writing your patches/features
Now start to develop new feature or fix bugs on master, as usual.
For this example, we are going to change the .gitreview file in order to use a
local Gerrit server.
```bash
sed -i 's/review\.openstack\.org/gerrit\.my\.org/' .gitreview
```
Don’t forget to commit and push (after this step, you may want to use git
review as usual)
```bash
git commit -a -m "Set .gitreview content to use internal gating infra"
git push
```
Our master (local and remote) tip should be now pointing to the last commit.
## Importing single patches from upstream
Before implementing any feature or fixing any bug (in short, before reinventing
the wheel), check if someone has already implemented the required code
upstream.
If not, try not to develop code only for your specific needs, be ambitious and
try to develop something that could be useful for the whole community. This way
you can propose your patch upstream and save yourself a lot of trouble which
arise when there are many local changes to carry on the tip of upstream
releases.
In this example, we tried to use our code and we found out that the job
filtering isn’t working! Fortunately, Antoine Musso has already fixed this bug,
as we can see in the upstream repo.
```bash
git show --summary 2eca0d11669b55d4ab02ba609a15aa242fd80d14
commit 2eca0d11669b55d4ab02ba609a15aa242fd80d14
Author: Antoine Musso <hashar@free.fr>
Date: Mon Jun 24 14:36:52 2013 +0200
job filtering was not working properly
When passing job names as arguments to 'update', the command is supposed
to only retain this jobs. Due to the job being a dict, the filter would
never match and the none of the job would be updated.
This has apparently always been broken since the feature got introduced
in 85cf7a41. Using job.['name'] fix it up.
Change-Id: Icf4d5b0bb68777f7faff91ade04451d4c8501c6a
Reviewed-on: https://review.openstack.org/34197
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
```
We are also interested in the following commit, which adds the Environment File
Plugin (finally!).
```bash
git show --summary bf4524fae25c11640ef839aa422ac81bd926ca20
commit bf4524fae25c11640ef839aa422ac81bd926ca20
Author: zaro0508 <zaro0508@gmail.com>
Date: Mon Jul 1 11:21:24 2013 -0700
add Environment File Plugin
This commit adds the Environment File Plugin to JJB.
https://wiki.jenkins-ci.org/display/JENKINS/Envfile+Plugin
Change-Id: Id35a4d6ab25b0440303da02bb91007b459979243
Reviewed-on: https://review.openstack.org/35170
Reviewed-by: Arnaud Fabre <fabre.arnaud@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
```
Import those changes simply cherry-picking the two commits. Don’t forget to
push (review!) your changes.
```bash
git cherry-pick 2eca0d11669b55d4ab02ba609a15aa242fd80d14
git cherry-pick bf4524fae25c11640ef839aa422ac81bd926ca20
git push
```
## Importing new versions from upstream
Days passes and finally a new releases comes out.
```bash
git fetch --all
git for-each-ref refs/remotes/upstream --format "%(refname:short)" | \
sed -e 's:\(upstream/\(.*\)\)$:\1\:refs/heads/upstream/\2:' | \
xargs git push --tags origin
```
A lot of work has been done upstream and we need to rebase our master onto the
upstream master branch. In this process we must skip all the commits we already
cherry-picked some days ago, of course.
**Note**: the rebasing for this example is trivial but it is just to break the
ice. Later in this guide we will address rebasing conflicts that can occur in
the real world.
Create a new local branch with the new release tag as a starting point
```bash
git branch import/0.6.0 0.6.0
```
## Running git-upstream
Finally, it is time to run git-upstream! Before doing so make sure the current
branch is master
```bash
git checkout master
```
```bash
git-upstream import import/0.6.0
Searching for previous import
Starting import of upstream
Successfully created import branch
Attempting to linearise previous changes
Successfully applied all locally carried changes
Merging import to requested branch 'HEAD'
Successfully finished import:
target branch: 'HEAD'
upstream branch: 'import/0.6.0'
import branch: 'import/0.6.0'
```
***No errors***, we have been lucky!
What has just happened?
git-upstream has created a new branch named `import/0.6.0-base` which tip is
set to the commit pointed by the release tag `0.6.0`, and has rebased all
changes present in our local master which were not already present in the
upstream new release (`import/0.6.0-base`) onto `import/0.6.0-base`.
You can see that running the following command
```bash
git log --graph --oneline --all --decorate
```
For this trivial example, the only commit not present in the upstream release
was about the customisation of the .gitreview file.
The default strategy git-upstream uses to find duplicate entries is the
comparison of Change-id entries in commit messages. Of course, it’s not
possible to compare directly the SHA1 for a commit because the cherry-picking
changes the information used for SHA1 calculation
---
**Note**: A git commit SHA1 is generated from the following information:
* commit message
* author signature (identity + timestamp)
* committer signature (identity + timestamp)
* tree SHA1 (hierarchy of directories and files within the commit)
* list of the SHA1's of the parent commits
---
The local branch `import/0.6.0` now contains our local changes rebased onto the
new upstream release. git-upstream has also merged this branch with the local
master branch (with "ours" strategy) to allow the normal workflow
(committing/merging to master for review).
**Note**: The "final" merging step is not mandatory. Of course you can keep a
separate branch for each new import. On one hand this strategy allows a
"cleaner" history as you will always have your local changes rebased on top of
the exact copy of the upstream repository. On the other hand you will be
creating a new branch every time you want to import upstream code.
You can customise the name of the import branch using the `--import-branch
<branch name>` option.
In principle, you could also replace your master branch (history) with the new
import branch created by git-upstream... Unfortunately there is no way to do
this without requiring ad-hoc intervention on cloned copies of the repository
(aka do-not-do-that(TM))
To disable automatic merging, just use the `--no-merge` flag
```bash
git-upstream import --no-merge import/0.6.0
```
# Handling conflicts
Of course in the real world things are much more complicated. From time to time,
during import, you will get rebasing conflict (for instance due to changes from
both local and upstream repository to the same piece of code).
In case of rebasing conflict, git-upstream will stop allowing the user to fix
the conflict.
```bash
git-upstream import import/0.5.0 --into master
Searching for previous import
Starting import of upstream
Successfully created import branch
Attempting to linearise previous changes
ERROR : Rebase failed, will need user intervention to resolve.
error: could not apply f9b4fca... Fixup for openstack review
When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".
Could not apply f9b4fca... Fixup for openstack review
Import cancelled
```
Let's find out why git-upstream failed and let's try to continue the rebasing
manually.
```bash
git status
# HEAD detached from 8e6b9e9
# You are currently rebasing branch 'import/0.5.0' on '8e6b9e9'.
# (fix conflicts and then run "git rebase --continue")
# (use "git rebase --skip" to skip this patch)
# (use "git rebase --abort" to check out the original branch)
#
# Unmerged paths:
# (use "git reset HEAD <file>..." to unstage)
# (use "git add <file>..." to mark resolution)
#
# both modified: jenkins_jobs/cmd.py
# both modified: jenkins_jobs/modules/hipchat_notif.py
#
no changes added to commit (use "git add" and/or "git commit -a")
```
Depending on the type of conflict, you will could:
* drop the local change
Issuing `git rebase --skip`
* edit conflicting code
Change conflicting code in order to accommodate local changes to the new
upstream code.
You can later resume rebasing process issuing `git rebase --continue`
Currently git-upstream can't resume the rebasing process. So, if needed, the
final "merging" steps have to be performed manually:
```bash
git merge -s ours --no-commit <import-xxxx>
```
Replacing tree contents with those from the import branch
```bash
git read-tree -u --reset <import-xxxx>
```
Committing merge commit
```bash
git commit --no-edit
```
**Note**: git-upstream performs exactly those steps in order to replace the
content of `master` branch with the import branch preserving the history.
# Integration with Gerrit
You may want to use review with Gerrit the output of git-upstream, in order to
perform tests, gating, etc.
You have 2 options for doing that:
## Re-review every new commit
In this case we want to review every new commit (since the last import). In
order to do so, use the `--no-merge` flag of git-upstream import command, and:
```bash
git checkout import-xxxxx
git push gerrit import-xxxxx-base:import-xxxxx
git review import-xxxxx
```
If there is more than one new commit, git-review will ask to confirm the
submission of multiple changes.
## Re-review only the final merge commit
This would be possible by using the `--import-branch` option of import command
and **pushing directly** (*i.e.*: bypassing Gerrit) the new branch to the local
repo. For instance:
```bash
TIMESTAMP=$(date +"%Y%m%d%H%M%s")
git upstream import --import-branch "import/import-$TIMESTAMP" upstream/master
git push gerrit import/import-$TIMESTAMP:import/import-$TIMESTAMP
```
Then, create a valid `Change-Id` for the merge commit
```bash
git commit --amend -C HEAD --no-edit
```
Locally, git-review will still complain about the presence of N+M commits which
would be committed BUT on the remote side all those commits will be recognised
as already present in one of the two branch involved in the merge.
```bash
git review -R -y master
```

2
doc/source/examples.rst Normal file
View File

@ -0,0 +1,2 @@
Examples
========

7
doc/source/index.rst
View File

@ -6,11 +6,16 @@
Welcome to git-upstream's documentation!
========================================
Contents:
Contents:
=========
.. toctree::
:maxdepth: 2
installation
subcommands
workflows
examples
Indices and tables

131
doc/source/installation.rst Normal file
View File

@ -0,0 +1,131 @@
Installation
============
To install git-upstream from pypi_, run:
.. code:: bash
pip install --user git-upstream
Alternatively, the current release can be installed system-wide from
pypi_:
.. code:: bash
sudo pip install git-upstream
Installing directly from source is possible, first clone and then
install using pip:
.. code:: bash
git clone https://git.openstack.org/openstack/git-upstream.git
cd git-upstream
pip install .
Or setup.py:
.. code:: bash
git clone https://git.openstack.org/openstack/git-upstream.git
cd git-upstream
python setup.py install
Or alternatively:
.. code:: bash
git clone https://git.openstack.org/openstack/git-upstream.git
cd git-upstream
easy_install .
If you want command line completion (using tab), install the provided
"bash completion" file
.. code:: bash
mkdir ~/bin && cp ./bash_completion/git-upstream ~/bin
echo ". ~/bin/git-upstream" >> ~/.bash_profile
Verify your installation.
.. code:: bash
pip show git-upstream
---
Name: git-upstream
Version: 0.12.1
Summary: git tool to help manage upstream repositories
Home-page: https://pypi.python.org/pypi/git-upstream
Author: Darragh Bailey
Author-email: dbailey@hpe.com
License: Apache License (2.0)
Location: /home/<username>/.local/lib/python2.7/site-packages
Requires: argcomplete, pbr, six, GitPython
git-upstream --help
usage: git-upstream [--version] [-h] [-q | -v] <command> ...
[...]
.. _pypi: https://pypi.python.org/pypi/git-upstream
Installing for Development
--------------------------
A virtual environment is recommended for development. For example,
git-upstream may be installed from the top level directory:
.. code:: bash
virtualenv .venv
source .venv/bin/activate
pip install -r test-requirements.txt -e .
Generating Documentation
------------------------
Documentation is included in the ``doc`` folder. To generate docs
locally execute the command::
tox -e docs
The generated documentation is then available under
``doc/build/html/index.html``.
* Note: When behind a proxy it is necessary to use ``TOX_TESTENV_PASSENV``
to pass any proxy settings for this test to be able to check links are
valid.
Unit Tests
----------
Unit tests have been included and are in the ``git_upstream/tests``
folder. Many unit tests samples are included as example scenarios in
our documentation to help explain how git-upstream handles various use
cases. To run the unit tests, execute the command:
.. code:: bash
tox -e py34,py27
* Note: View ``tox.ini`` to run tests on other versions of Python,
generating the documentation and additionally for any special notes
on building one of the scenarios to allow direct inspection and
manual execution of ``git-upstream`` with various scenarios.
The unit tests can in many cases be better understood as being closer
to functional tests.
Test Coverage
-------------
To measure test coverage, execute the command::
tox -e cover

177
doc/source/subcommands.rst Normal file
View File

@ -0,0 +1,177 @@
Git-Upstream Subcommands
========================
import
------
Description
~~~~~~~~~~~
Import code from specified upstream branch. Creates an import branch
from the specified upstream branch, and optionally merges additional
branches given as arguments. Current branch, unless overridden by the
``--into`` option, is used as the target branch from which a list of
changes to apply onto the new import is constructed based on the
specified strategy. Once complete it will merge and replace the contents
of the target branch with those from the import branch, unless
``--no-merge`` is specified.
Usage
~~~~~
::
git upstream import [-h] [-d] [-i] [-f] [--merge] [--no-merge]
[-s <strategy>] [--into <branch>]
[--import-branch <import-branch>]
[<upstream-branch>] [<branches> [<branches> ...]]
Arguments
~~~~~~~~~
::
positional arguments:
<upstream-branch> Upstream branch to import. Must be specified if you
wish to provide additional branches.
<branches> Branches to additionally merge into the import branch
using default git merging behaviour
optional arguments:
-h, --help show this help message and exit
-d, --dry-run Only print out the list of commits that would be
applied.
-i, --interactive Let the user edit the list of commits before applying.
-f, --force Force overwrite of existing import branch if it
exists.
--merge Merge the resulting import branch into the target
branch once complete
--no-merge Disable merge of the resulting import branch
-s <strategy>, --strategy <strategy>
Use the given strategy to re-apply locally carried
changes to the import branch. (default: drop)
--into <branch> Branch to take changes from, and replace with imported
branch.
--import-branch <import-branch>
Name of import branch to use
drop
----
Description
~~~~~~~~~~~
Mark a commit as dropped. Marked commits will be skipped during the
upstream rebasing process.
See also the "git upstream import" command.
Usage
~~~~~
::
git upstream drop [-h] [-a <author>] <commit>
Arguments
~~~~~~~~~
::
positional arguments:
<commit> Commit to be marked as dropped
optional arguments:
-h, --help show this help message and exit
-a <author>, --author <author>
Git author for the mark
Note
~~~~
Commits will be marked with git notes in the namespace
``refs/notes/upstream-merge``.
To list of commit id marked with a note, run
``git notes --ref refs/notes/upstream-merge``.
To show a specific note run
``git notes --ref refs/notes/upstream-merge show <marked commit sha1>``
As ``drop`` uses git notes to mark commits that have to be skipped
during import, notes should be present on the cloned copy of your
repository. Thus, if you are going to create notes on a system and
perform the actual import on a different system, **notes must be present
on the latter**.
You can push notes directly to git repository on the target system or
push them in a different repository and then pull notes from your target
system.
supersede
---------
Description
~~~~~~~~~~~
Mark a commit as superseded by a set of change-ids. Marked commits will
be skipped during the upstream rebasing process **only if all the
specified change-ids are present in ``<upstream-branch>`` during
import**. If you want to unconditionally drop a commit, use the ``drop``
command instead.
See also the "git upstream import" command.
Usage
~~~~~
::
git upstream supersede [-h] [-f] [-u <upstream-branch>]
<commit> <change id> [<change id> ...]
Arguments
~~~~~~~~~
::
positional arguments:
<commit> Commit to be marked as superseded
<change id> Change id which makes <commit> obsolete. The change id
must be present in <upstream-branch> to drop <commit>.
If more than one change id is specified, all must be
present in <upstream-branch> to drop <commit>
optional arguments:
-h, --help show this help message and exit
-f, --force Apply the commit mark even if one or more change ids
could not be found. Use this flag carefully as commits
will not be dropped during import command execution as
long as all associated change ids are present in the
local copy of the upstream branch
-u <upstream-branch>, --upstream-branch <upstream-branch>
Search change ids values in <upstream-branch> branch
(default: upstream/master)
Note
~~~~
*This command doesn't perform the actual drop*. Commits to be dropped
during the next import, will be marked with git notes in the namespace
``refs/notes/upstream-merge``. There is no need to retain notes after an
import dropped the correspondent commits, of course it doesn't harm
keeping them either.
To list of commit id marked with a note, run
``git notes --ref refs/notes/upstream-merge``.
To show a specific note run
``git notes --ref refs/notes/upstream-merge show <marked commit sha1>``.
As ``supersede`` uses git notes to mark commits that have to be skipped
during import, notes should be present on the cloned copy of your
repository. Thus, if you are going to create notes on a system and
perform the actual import on a different system, **notes must be present
on the latter**. You can push notes directly to git repository on the
target system or push them in a different repository and then pull notes
from your target system.

438
doc/source/workflows.rst Normal file
View File

@ -0,0 +1,438 @@
Workflows
=========
.. note:: this guide assumes that you are using a branch named *master*
to maintain your new features or bug fixes that sit on top of the
upstream code of some project (probably somewhat related to
OpenStack).
It is also assumed you are tracking releases, which is only one of
the possible approaches to upstream tracking. Another approach would
be tracking the master tip of a project. Of course even other
strategies are possible.
Importing from upstream: using git-upstream
-------------------------------------------
See :doc:`installation instructions </installation>` for details on
installing.
Initial import of an upstream project
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To explain the usage of the git-upstream tool we are going to use a
real-world (but trivial) example, by performing some sample operations
on a project called ``jenkins-job-builder``.
In this example, we will create a local file based Git repository to
host our mirror of jenkins-job-builder. You could also use an existing
internal mirror, a Github fork, etc.
Start by setting the following environment variables:
.. code:: bash
export REPO_NAME="jenkins-job-builder"
export INTERNAL_REMOTE="file:///tmp/jenkins-job-builder.git"
export UPSTREAM_REMOTE="https://github.com/openstack-infra/jenkins-job-builder.git"
export FIRST_IMPORT_REF="0.5.0"
1\) Create two empty repositories, one to serve as your working copy, and
one to serve as the remote:
.. code:: bash
git init --bare /tmp/${REPO_NAME}.git
git init $REPO_NAME
cd $REPO_NAME
2\) Add your remotes
We will name it *origin* and *upstream* (for the sake of originality).
.. code:: bash
git remote add origin $INTERNAL_REMOTE
git remote add upstream $UPSTREAM_REMOTE
3\) Fetch objects and refs from upstream remote
.. code:: bash
git fetch --all
4\) Push refs
Push refs defined upstream to the ``origin`` remote (*i.e.*, the
internal copy of the repository with local patches) using the string
``upstream`` as prefix, also pushing tags.
.. code:: bash
git for-each-ref refs/remotes/upstream --format "%(refname:short)" | \
sed -e 's:\(upstream/\(.*\)\)$:\1\:refs/heads/upstream/\2:' | \
xargs git push --tags origin
You may want to repeat the last two commands before starting any new
feature development or a bug fix.
5\) Check-out the first import commit (*e.g.*, tag or SHA1)
This will be the starting point for the internal development.
.. code:: bash
git checkout -b import/$FIRST_IMPORT_REF $FIRST_IMPORT_REF
6\) Create and switch to the master branch
.. code:: bash
git checkout -b master
Now the tips of master, ``$FIRST_IMPORT_REF`` and
``import/$FIRST_IMPORT_REF`` should be pointing to the same commit.
Push local master branch to the remote origin, and make
``origin master`` the default when pushing commits.
.. code:: bash
git push -u origin master
Writing your patches/features
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now start to develop new feature or fix bugs on master, as usual. For
this example, we are going to change the .gitreview file in order to use
a local Gerrit server.
.. code:: bash
sed -i 's/review\.openstack\.org/gerrit\.my\.org/' .gitreview
Don’t forget to commit and push (after this step, you may want to use
git review as usual)
.. code:: bash
git commit -a -m "Set .gitreview content to use internal gating infra"
git push
Our master (local and remote) tip should be now pointing to the last
commit.
Importing single patches from upstream
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Before implementing any feature or fixing any bug (in short, before
reinventing the wheel), check if someone has already implemented the
required code upstream.
If not, try not to develop code only for your specific needs, be
ambitious and try to develop something that could be useful for the
whole community. This way you can propose your patch upstream and save
yourself a lot of trouble which arise when there are many local changes
to carry on the tip of upstream releases.
In this example, we tried to use our code and we found out that the job
filtering isn’t working! Fortunately, Antoine Musso has already fixed
this bug, as we can see in the upstream repo.
.. code:: bash
git show --summary 2eca0d11669b55d4ab02ba609a15aa242fd80d14
commit 2eca0d11669b55d4ab02ba609a15aa242fd80d14
Author: Antoine Musso <hashar@free.fr>
Date: Mon Jun 24 14:36:52 2013 +0200
job filtering was not working properly
When passing job names as arguments to 'update', the command is supposed
to only retain this jobs. Due to the job being a dict, the filter would
never match and the none of the job would be updated.
This has apparently always been broken since the feature got introduced
in 85cf7a41. Using job.['name'] fix it up.
Change-Id: Icf4d5b0bb68777f7faff91ade04451d4c8501c6a
Reviewed-on: https://review.openstack.org/34197
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
We are also interested in the following commit, which adds the
Environment File Plugin (finally!).
.. code:: bash
git show --summary bf4524fae25c11640ef839aa422ac81bd926ca20
commit bf4524fae25c11640ef839aa422ac81bd926ca20
Author: zaro0508 <zaro0508@gmail.com>
Date: Mon Jul 1 11:21:24 2013 -0700
add Environment File Plugin
This commit adds the Environment File Plugin to JJB.
https://wiki.jenkins-ci.org/display/JENKINS/Envfile+Plugin
Change-Id: Id35a4d6ab25b0440303da02bb91007b459979243
Reviewed-on: https://review.openstack.org/35170
Reviewed-by: Arnaud Fabre <fabre.arnaud@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Import those changes simply cherry-picking the two commits. Don’t forget
to push (review!) your changes.
.. code:: bash
git cherry-pick 2eca0d11669b55d4ab02ba609a15aa242fd80d14
git cherry-pick bf4524fae25c11640ef839aa422ac81bd926ca20
git push
Importing new versions from upstream
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Days passes and finally a new releases comes out.
.. code:: bash
git fetch --all
git for-each-ref refs/remotes/upstream --format "%(refname:short)" | \
sed -e 's:\(upstream/\(.*\)\)$:\1\:refs/heads/upstream/\2:' | \
xargs git push --tags origin
A lot of work has been done upstream and we need to rebase our master
onto the upstream master branch. In this process we must skip all the
commits we already cherry-picked some days ago, of course.
.. note:: the rebasing for this example is trivial but it is just to
break the ice. Later in this guide we will address rebasing
conflicts that can occur in the real world.
Create a new local branch with the new release tag as a starting point
.. code:: bash
git branch import/0.6.0 0.6.0
Running git-upstream
~~~~~~~~~~~~~~~~~~~~
Finally, it is time to run git-upstream! Before doing so make sure the
current branch is master
.. code:: bash
git checkout master
.. code:: bash
git-upstream import import/0.6.0
Searching for previous import
Starting import of upstream
Successfully created import branch
Attempting to linearise previous changes
Successfully applied all locally carried changes
Merging import to requested branch 'HEAD'
Successfully finished import:
target branch: 'HEAD'
upstream branch: 'import/0.6.0'
import branch: 'import/0.6.0'
***No errors***, we have been lucky!
What has just happened?
git-upstream has created a new branch named ``import/0.6.0-base`` which
tip is set to the commit pointed by the release tag ``0.6.0``, and has
rebased all changes present in our local master which were not already
present in the upstream new release (``import/0.6.0-base``) onto
``import/0.6.0-base``.
You can see that running the following command
.. code:: bash
git log --graph --oneline --all --decorate
For this trivial example, the only commit not present in the upstream
release was about the customisation of the .gitreview file.
The default strategy git-upstream uses to find duplicate entries is the
comparison of Change-id entries in commit messages. Of course, it’s not
possible to compare directly the SHA1 for a commit because the
cherry-picking changes the information used for SHA1 calculation
.. note:: A git commit SHA1 is generated from the following information:
- commit message
- author signature (identity + timestamp)
- committer signature (identity + timestamp)
- tree SHA1 (hierarchy of directories and files within the commit)
- list of the SHA1's of the parent commits
--------
The local branch ``import/0.6.0`` now contains our local changes rebased
onto the new upstream release. git-upstream has also merged this branch
with the local master branch (with "ours" strategy) to allow the normal
workflow (committing/merging to master for review).
.. note:: The "final" merging step is not mandatory. Of course you can
keep a separate branch for each new import. On one hand this
strategy allows a "cleaner" history as you will always have your
local changes rebased on top of the exact copy of the upstream
repository. On the other hand you will be creating a new branch
every time you want to import upstream code. You can customise the
name of the import branch using the
``--import-branch <branch name>`` option.
In principle, you could also replace your master branch (history) with
the new import branch created by git-upstream... Unfortunately there is
no way to do this without requiring ad-hoc intervention on cloned copies
of the repository (aka do-not-do-that(TM))
To disable automatic merging, just use the ``--no-merge`` flag
.. code:: bash
git-upstream import --no-merge import/0.6.0
Handling conflicts
------------------
Of course in the real world things are much more complicated. From time
to time, during import, you will get rebasing conflict (for instance due
to changes from both local and upstream repository to the same piece of
code).
In case of rebasing conflict, git-upstream will stop allowing the user
to fix the conflict.
.. code:: bash
git-upstream import import/0.5.0 --into master
Searching for previous import
Starting import of upstream
Successfully created import branch
Attempting to linearise previous changes
ERROR : Rebase failed, will need user intervention to resolve.
error: could not apply f9b4fca... Fixup for openstack review
When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".
Could not apply f9b4fca... Fixup for openstack review
Import cancelled
Let's find out why git-upstream failed and let's try to continue the
rebasing manually.
.. code:: bash
git status
# HEAD detached from 8e6b9e9
# You are currently rebasing branch 'import/0.5.0' on '8e6b9e9'.
# (fix conflicts and then run "git rebase --continue")
# (use "git rebase --skip" to skip this patch)
# (use "git rebase --abort" to check out the original branch)
#
# Unmerged paths:
# (use "git reset HEAD <file>..." to unstage)
# (use "git add <file>..." to mark resolution)
#
# both modified: jenkins_jobs/cmd.py
# both modified: jenkins_jobs/modules/hipchat_notif.py
#
no changes added to commit (use "git add" and/or "git commit -a")
Depending on the type of conflict, you will could:
- drop the local change
Issuing ``git rebase --skip``
- edit conflicting code
Change conflicting code in order to accommodate local changes to the new
upstream code. You can later resume rebasing process issuing
``git rebase --continue``
Currently git-upstream can't resume the rebasing process. So, if needed,
the final "merging" steps have to be performed manually:
.. code:: bash
git merge -s ours --no-commit <import-xxxx>
Replacing tree contents with those from the import branch
.. code:: bash
git read-tree -u --reset <import-xxxx>
Committing merge commit
.. code:: bash
git commit --no-edit
.. note:: git-upstream performs exactly those steps in order to replace
the content of ``master`` branch with the import branch preserving the
history.
Integration with Gerrit
-----------------------
You may want to use review with Gerrit the output of git-upstream, in
order to perform tests, gating, etc.
You have 2 options for doing that:
Re-review every new commit
~~~~~~~~~~~~~~~~~~~~~~~~~~
In this case we want to review every new commit (since the last import).
In order to do so, use the ``--no-merge`` flag of git-upstream import
command, and:
.. code:: bash
git checkout import-xxxxx
git push gerrit import-xxxxx-base:import-xxxxx
git review import-xxxxx
If there is more than one new commit, git-review will ask to confirm the
submission of multiple changes.
Re-review only the final merge commit
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This would be possible by using the ``--import-branch`` option of import
command and **pushing directly** (*i.e.*: bypassing Gerrit) the new
branch to the local repo. For instance:
.. code:: bash
TIMESTAMP=$(date +"%Y%m%d%H%M%s")
git upstream import --import-branch "import/import-$TIMESTAMP" upstream/master
git push gerrit import/import-$TIMESTAMP:import/import-$TIMESTAMP
Then, create a valid ``Change-Id`` for the merge commit
.. code:: bash
git commit --amend -C HEAD --no-edit
Locally, git-review will still complain about the presence of N+M
commits which would be committed BUT on the remote side all those
commits will be recognised as already present in one of the two branch
involved in the merge.
.. code:: bash
git review -R -y master