Into MOS9 Custom ISO included only mos9.0-updates, mos9.0-security
and mos9.0-proposed dists. We should also include mos9.0-holdback
and mos9.0-hotfix, because they are contain packages with fixes for
critical issues.
Change-Id: Ie439f61ec10c45e1ba69767067d7c6716e8df145
Closes-bug: #1657731
(cherry picked from commit 4e4d96c16b)
After MOS 9.0 release main repositories are now freezed, and the latest
merged code lands into -proposed. Thus to keep development and CI for
stable/mitaka branch packages from proposed should be added to the ISO.
Change-Id: I5f63ec2921e96e7efbb383eb4a92a40e2ab900bd
Closes-bug: #1619309
(cherry picked from commit 7c45cbe613)
After MOS 9.0 release main repositories are now freezed, and the latest
merged code lands into -proposed. Thus to keep development and CI for
stable/mitaka branch packages from proposed should be added to the ISO.
Closes-bug: #1619309
Change-Id: I8f7cd2fd221e3db5893f7a369696b13f29f83bd5
All related internal mirrors can be re-defined through
the env variables, so we can freely remove them
Change-Id: Iae8bb0fbba8097e67287406a8aaf378283545c5c
Closes-bug: #1582701
mirror/centos/yum-priorities-plugin.py is a module rather than a
script, there is no need to set +x flag.
Change-Id: I59705c729530fdb90ac59b1855e7bf9ad46a221b
Disable SSL certificates check since we can have self signed
certs for yum repos
Change-Id: I38067ec3cf4e86b0b2623d716059763308ae0c5b
Closes-bug: #1541934
We got rid of using docker containers
on the master node. Now we deploy all
Fuel service directly on the host.
Implements blueprint: get-rid-docker-containers
Change-Id: Ie636d33d091a95fb0ff8fbd71e49cbccf1cb34fa
According to the separate-mos-from-linux spec, packages
with debug symbols are distributed via public mirrors,
so putting them to the product ISO makes no sense.
This change was introduced in commit 7177b8fcb3
but removed in the next one b715f0b397
Implements: separate-mos-from-linux
Closes-bug: #1531476
Change-Id: I6ddf7965d527ba76a85e560356c6faeabb9e6801
During an ISO build, the extra repositories are synced by using
reposync utility. This utility should store cached metadata in
a local temporary directory, in order to prevent issues with
stale yum cache.
Change-Id: I58c326fe2bfd4da4cba9daa3343bfe5123e8e560
Closes-Bug: #1529073
During an ISO build, the MOS repository is synced by using reposync
utility. This utility should store cached metadata in a local temporary
directory, in order to prevent issues with stale yum cache.
Change-Id: I92573191f6c9b4d4a17b14890d0ab88ceb0a75c0
Closes-Bug: #1529073
Existing ISO build scripts can't build CentOS7 based
ISO due to changes in package names, files, and so on.
This review fixes issue have been found, and introduces
several minor improvements.
Blueprint: master-on-centos7
Breaks: centos6
Co-Authored-By: Ivan Suzdal <isuzdal@mirantis.com>
Change-Id: I830ff03aa1051f14f2bdafb16cca1979d38e9ad5
Fix the problem with separate repos to support rebuild docker
1. Make a symlink for mos-centos in /var/www/nailgun in iso/ks.template
2. Add mos-repos to the bootstrap_admin_node.sh
This reverts commit 32b1363015.
Closes-bug: #1521551
Change-Id: I8ef58b581f253b26d158b725a9e366d7f1b3706d
The commit this patch reverts breaks Fuel CI
on docker container rebuilds.
This reverts commit bc9c3117c7.
Change-Id: I18fe16cf8f0be40e30fec6229606d1d69cbfe73d
Closes-Bug: #1521551
Release ISO must contain only packages that are
modified by MOS team, and CentOS upstream packages
for Fuel node deployment.
MOS repo, as well as the ones specified in the
EXTRA_RPM_REPOS variable, are downloaded and
placed on the ISO as-is.
Blueprint: separate-mos-from-centos
Change-Id: I983f37371a70d2901bdf0ebef122ba1a2f2e44b9
Build iso with BUILD_PACKAGES=0:
* Late packages:
- fuel-bootstrap-image,
- fuel-docker-images,
should be always built in spite of BUILD_PACKAGES flag,
since they depend on the first level artifacts (packages)
and are not updated in the mirrors yet;
* Build fuel-bootstrap-image-builder package for astute container
* Update requirements-*.txt file by adding all fuel packages only in case
BUILD_PACKAGES=0
* Update docker-images spec to support fakeroot rpmbuild approach
Related-bug: #1460480
Change-Id: I00ea15c9df9277d48a294f8379d87352c017becb
Sometimes it is necessary to enable extras yum repository while it is
disabled by default. The most convenient way is to set YUM_REPOS
variable for a particular job. This patch makes it possible.
Change-Id: Icb074474fa4602288aa43129dd1dfc058ebe646b
Perestroika builds both rpm and deb fuel repos.
We use perestroika rpm repository to build fwm
repository which includes fuel packages as well
as those packages which are listed in requirements-rpm.txt.
Deb repository can be used directly.
Closes-Bug: #1479761
Change-Id: I236144759eb38e71977d175226f81772d23de9b3
We can talk about several types of repos configurations:
* Ubuntu sandbox (make show-ubuntu-sandbox-repos);
* CentOS:
- for building local mirror (make show-yum-repos-centos);
- for using in sandbox (make show-centos-sandbox-repos);
This patch introduces several changes
1. Rename show-apt-source-list target to show-ubuntu-sandbox-repos
to be closer to centos naming (show-centos-sandbox-repos)
2. Add show-centos-sandbox-repos target, which will show yum
configuration for centos sandbox
3. Add show-yum-repos-centos, which will show yum configuration
used while building local centos mirror
Examples:
- make show-ubuntu-sandbox-repos
- make show-centos-sandbox-repos
- USE_MIRROR=none \
EXTRA_RPM_REPOS=extra,http://my.repo \
make show-yum-repos-centos
Change-Id: I0b1141f8b7acfa682f7d4c4cc6e2fd09363eb3ca
Seems we don't need this any more because
perestroika puts source packages on its own.
They are available online for users.
Besides, perestroika has slightly different
repo format than fwm repo and switching
to perestroika breaks src rpm downloading
for staging mirror job.
Change-Id: Ifef8e37b690dc1d867cad023fa138144637f689e
We need to be able to download packages from
various repositories taking into account repository
priority. We need this for patching story.
By default priorities are
1) upstream 90
2) proprietary 20
3) DENY_RPM_DOWNGRADE=1 (default: 1), we are
not allowed to downgrade rpm packages. In some
cases when we need to be able to use downgraded
versions this flag should be set to 0:
DENY_RPM_DOWNGRADE=0
EXTRA_RPM_REPOS is to have the following format
"name1,url1,pri1 name2,url2,pri2"
In case when pri is not defined it will be set to 10,
for example:
"name1,url1" is equal to "name1,url1,10"
Closes-Bug: #1446293
Change-Id: I4a26dd005852af60104c39e46384a21c39cce587
Apparently yumdownloader does not raise an error if some packages are
missing or broken (dependencies can not be satisfied). As a result some
packages have silently disappered from the mirror (kmod-openvswitch) and
others have been broken (sheepdog).
Patch yumdownloader so it bails out if some packages are missing or have
broken dependencies. As a side effect yumdownloader raises an error if
some of the packages being downloaded conflict (like kernel-headers vs
kernel-headers-lt), so modify the mirroring code to detect and handle
the conflicting packages.
More strict dependencies check yield some unexpected issues. In
particular a few 32 bit packages are broken due to the upgraded base
packages from Fuel repo being x86_64 only. In particular, 32 bit dmraid
package is broken due to device-mapper-libs:
Package: device-mapper-libs-1.02.79-8.el6.i686 (base)
Requires: device-mapper = 1.02.79-8.el6
Available: device-mapper-1.02.79-8.el6.x86_64 (base)
device-mapper = 1.02.79-8.el6
Installing: device-mapper-1.02.90-2.mira1.x86_64 (fuel)
device-mapper = 1.02.90-2.mira1
To solve the problem configure yum to exclude most of i686 packages.
The configuration appeared to be a bit cumbersome since *some* i686
packages are required. In particular syslinux package requires
syslinux-nonlinux (binaries which run in syslinux environment before
booting the OS) which is formally an i686 package.
Closes-Bug: #1457382
Related-Bug: #1456459
Change-Id: I404b662cd3b54ee9d058692bb30177b985871b16
We used to create differential Ubuntu and Centos
repositories for new Fuel versions to make them
smaller. However, this technique contradicts
our current upgrade/patching approach. So, this
patch removes this functionality.
Closes-Bug: #1447135
Related-Bug: #1446293
Change-Id: I6c9288e77c6ccc5872056a8119230ec725846bba
1) unconditional rebuilding due to
never existent file
repocleanup.done used to depend on PHONY target (sources)
which led to unconditional rebuilding.
This didn't influence ISO builds because CI always cleans
environment.
2) fail when try to rebuild second
time without cleaning environment
repo.done used to consume some config files
created by reprepro.done target. After consuming those files
it used to remove them not touching reprepro.done itself.
Without cleaning build environment this fails the build process.
Change-Id: Ib0de2e5ea94caeb7f93f5bc8d0f06a6f806e3ca6
Closes-Bug: #1448206
wget defaults to appending ".1" to filenames when
downloading files that already exists. Switching
to timestamp mode (-N) forces wget to replace
existing file only when there is a newer file
on the server. This applies just to downloading
RPM files currently.
Change-Id: Ifc63abb181d53255105e620b26fd9e81611cdc5e
Closes-Bug: #1423874
This commit is a squashed version of commits that
implement packaging of fuel components. In order to
make our packaging process better I had to introduce
some new macros and do additional cleanup.
The main paradigm that we are following with this commit is
the following:
1 GIT REPO -> 1 PACKAGING SPEC -> N PACKAGES
Though it would be ideal to have only one package per git repo,
but this is left for further releases
This change is very essential to make updates process easier
as we provide only one type of artifacts that a user can fetch -
distro-specific package
This particular commit does not introduce new DEB packages
(such as packages of OCF scripts and other binaries that
we currently put onto the slave nodes with puppet instead of
installing them as packages) which will be introduced as
a part of whole packaging initiative a little bit later
1) Add prepare_git_source macro
This change is important to always
checkout the code from git working
or we can end up with garbage getting
into package source code and add
version.txt file to source archives
2) Remove nailgun-redhat-license package
This package is obsolete - we do not need
it.
3) Package fuel-library into RPM package
4) Delete old packages and misc stuff
This change is a little cleanup of python-tasklib
and other leftovers
5) Change RPM sandbox configuration
Change RPM sandbox configuration to use upstream mirrors
to be able to install nailgun build dependencies
6) Alter Nailgun spec
a) Change nailgun spec to use NPM during package
build
b) Switch nailgun to prepare_git_source
c) Package all packages of fuel-web repo using
only nailgun.spec
7) Alter RPM sandbox preparation to install build-deps
8) Package astute and mcagents using astute.spec
9) Include all built RPMs
Include all RPMS that were built inside the sandbox
10) Package fuel-ostf with git-archive
11) Build python-fuelclient using git_archive macro
12) Move specs to particular git repositories
13) Rename fuel-library and ostf to make macros easier
14)Adjust build scripts to fetch specs from particular
fuel repositories
FUEL DEB-packaging related code
15) Restore part of make system code for DEB packages building
16) Nailgun debian spec files for all nailgun-based packages
17) Clean up SANDBOX for Ubuntu generation
18) Clean up some package building scripts
19) Put PACKAGE_VERSION variable into the top config.mk
20) Add dockerctl and change fuel-library tar.tg path
21) Move deb rules to particula repos
22) Allow to rebuild deb repo with reprepro on-the-fly
24) Fix repocleanup stanza to parse source packages metadata
25) Debmirror whole repos in order not to break reprepro
26) Make docker image build depend on late RPM packages
or it may lead to race conditions
27) Clean up old deb packages with one shot instead
of parallel run
Change-Id: I15ae4d0abe51c72b28793e1a1ef30fec0f668b73
Blueprint: package-fuel-components
1) Change mirroring procedure for Ubuntu:
- USE_MIRROR=none: rsync the whole mirror from build server
- USE_MIRROR=<location>: mirror only required MOS distro
Added support for different mirroring methods.
2) Remove procedure of building Fuel deb packages
3) Remove building of deb-based IBP images
4) Remove deb-based chroots
5) Modify creation of Ubuntu diff mirror artifact
Implements: separate-mos-from-linux
Change-Id: Icd10aea6ebddb27976e88ceb930be9afac0241ad
Apparently yumdownloader processes conflicting packages just fine if asked
for URLs only (as opposed to downloading the actual packages), hence
src.rpms can be processed in batch (which is much faster).
Speeds up the staging mirror job quite a bit.
Change-Id: Id4209cbad3cbefe4e96b12abf4e83ac6bafcfa2e
CentOS base docker image is a dependency for
fuel-centos-build container. It was left out
in commit b975019fab.
Change-Id: I822a6bb8c32f6592adc8971c1d67ca80af354e45
Partial-Bug: #1428060
Pass the USE_MIRROR and MIRROR_UBUNTU_SECURITY variables to submake
so it generates a valid sources.list
Change-Id: I796fa4c8fe0452d975fb57bedbcf332129da0154
* image/ubuntu:
- make sure puppet agent is not started during OS boot
- catch up the new ruby packages naming convention
- don't install Mellanox related packages (should be reverted
after rebuilding those packages for Ubuntu 14.04)
* requirements-deb.txt:
- adapt to the packages' names changes (guestmount -> libguestfs-tools)
- use the default version of ruby (ruby 1.8 is not available any more),
catch up the new ruby packages naming convention.
- dhcp-checker is gone, long live nailgun-net-checker.
- temporarily skip Mellanox related packages (until they get rebuild
for Ubuntu 14.04)
* config.mk:
- use the new OBS APT repository
* packages/deb/specs:
- nailgun-agent, fencing-agent: depend on ohai, so the custom ruby-ohai
package (which actually contains the identical code) is not required
any more
- temporarily remove nailgun-agent and fencing-agent from requiremets-deb.txt
Implements: blueprint support-ubuntu-trusty
Change-Id: I4300b892c1fbb8d1dfa8bdd18815d56bcabe324d
Create an alternative APT config/state instead of making a chroot.
Guess the set of nonconflicting packages and process them in a batch.
Figure out the set of udebs to download using apt-cache instead of
parsing APT repository meta-data with shell code.
blueprint support-ubuntu-trusty
Change-Id: I6f3694c714bbaa488894857ce3a6d5c753197db7
Create an alternative APT config/state instead of making a chroot.
Calculate packages' dependencies in parallel.
blueprint support-ubuntu-trusty
Change-Id: I27545b9f3bc1393803b953919ea7f077c23f3245
Andreas Jaeger