Commit Graph

58 Commits

Author SHA1 Message Date
Jenkins c3c2069e2e Merge "packages_tree: exact match trumps other providers" 2017-01-30 11:27:00 +00:00
Alexandru Avadanii 25b8673f84 packages_tree: exact match trumps other providers
Ubuntu frequently has multiple providers for the same package name,
e.g. "vim" is provided by "vim", "vim-gnome" etc.

We need to adjust the resolving algorithm to:
1. lookup in virtual packages
2. lookup in packages
3. lookup in obsoletes.
Use the package which was found last (version-sorted).

Closes-bug: 1656888

Change-Id: I7279aa6526ff9133829be2e316932c9b052c7814
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-01-26 19:00:00 +01:00
Alexandru Avadanii ce756a309a deb_driver: logger: Fix formatting typo
No functional impact, just fix a recurring traceback when verbose:
"TypeError: %d format: a number is required, not str"

Change-Id: I4a88e7eba81b4a470bd6a442762336d4110ea548
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-01-14 00:02:11 +01:00
Jenkins 3021c00156 Merge "clone: Create metadata for empty components" 2016-12-13 17:52:11 +00:00
Alexandru Avadanii cb261b0994 clone: Create metadata for empty components
In certain scenarios, mirror components (e.g. trusty-security)
are present in both packetary configuration input, and in target
system's apt source definitions, but contain no packages of interest.

For such repository/requirements combinations, packetary currently
skips creating metadata (Release, Packages) for those components,
which leads to the partial mirror missing some critical files for apt,
(an empty file would be enough).

e.g.: Using packetary to create a partial Ubuntu mirror, then trying
to build a bootstrap image from the new mirror leads to:

W: Failed to fetch http://127.0.0.1:8080/mirrors/ubuntu/dists/\
   trusty-security/multiverse/binary-amd64/Packages  404

Closes-bug: 1638631

Change-Id: I850b43d5b4d8742d99e9a5702cc9ad4de881a401
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2016-12-13 16:55:04 +01:00
Jenkins 485618fd5f Merge "AArch64 support: api, cli, controllers, drivers" 2016-12-13 15:33:57 +00:00
Alexandru Avadanii 874d9bd900 deb_driver: Translate repository.architecture
When generating repository metadata, the Release file gets the
architecture list in an untranslated format, ending up with
'x86_64' instead of 'amd64'.

Closes-bug: 1638628

Change-Id: I8471db6d54157175ff275c969ed5195d4616fa0f
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2016-11-02 20:07:06 +01:00
Alexandru Avadanii fd5b8c0359 AArch64 support: api, cli, controllers, drivers
Enable new architecture 'aarch64' ('arm64' on Ubuntu).
Tested with DEB driver.

Closes-bug: 1638635

Change-Id: I9761322020837186c109c18e849128791ab909d8
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2016-11-02 18:13:42 +01:00
Jenkins 1508e35df2 Merge "Allow to search mandatory packages in all forest" 2016-07-05 21:23:39 +00:00
Sergey Kulanov f922090dc6 Allow to search mandatory packages in all forest
Update requirements with filter option:

        "mandatory": {
            "enum": ["exact", "newest"]
        }

1) [RPM] We need to search the newest mandatory packages in all
   forest but not from the exact tree, also repo priority should
   be take into account

2) Add priority to gen_repository in tests, since this is
   mandatory parameter

3) [Tests] Add package10 with different version to check mandatory
   values:
    - None
    - "exact"
    - "newest"
  match for mandatory packages

Change-Id: I36a1463b495075bf6c44a46b557c3ac524de0220
Closes-bug: #1585103
2016-07-05 22:45:28 +03:00
Bulat Gaifullin cae6df70bf Added packaging driver to build RPM by using mock
Option 'cache_dir' uses to specify directory where
will be downloaded remote files
The packaging controller allows to use files which
are available via HTTP as source or spec file
Each driver has its own section in input data,
this allows to use same input data for several drivers.

Change-Id: I1fb3b08fe305c3413e5aa4a9213762208a2479da
2016-07-05 16:49:06 +03:00
Jenkins 5c9d32f234 Merge "Added CLI for building packages" 2016-07-01 12:28:06 +00:00
Jenkins 86aec07ca7 Merge "Fix createrepo workers racing" 2016-06-30 10:12:25 +00:00
Bulat Gaifullin 794256f1bc Added CLI for building packages
Usage of command:
packetary build -t "the name of driver" \
    -i "the path of file with input data" \
    -o "the path to output directory" \
    [-C "the config for driver"]

Change-Id: I71df1b8a36342a9ea238d5b6ebb686e5bce71007
2016-06-27 19:31:16 +03:00
Jenkins c88b21ee95 Merge "Introduced packaging API" 2016-06-27 13:48:27 +00:00
Jenkins 485fb77a88 Merge "Fixed handling http redirects" 2016-06-24 12:32:20 +00:00
Bulat Gaifullin 406bdaa7e2 Introduced packaging API
This change introduces API for building packages.
The commit includes Controller object and interface for PackagingDriver.

Change-Id: I1ce0f746c1cbc5cf8ff29d08d09175acb5d79586
2016-06-22 13:09:23 +03:00
Bulat Gaifullin f86e3216bd Fixed handling http redirects
Override method 'redirect_request' for patching
new request, that has been created on handling
http redirect.

Closes-Bug: 1592464
Change-Id: Id32bc46cce2c0eabb91f645d01378b6da3128469
2016-06-15 18:16:31 +03:00
Bulat Gaifullin f659d0ed63 Moved repositorory drivers to dedicated namespace
The 'packetary.drivers' is very common namespace, so
the repository drivers was moved to 'repository_drivers' namespace.

Change-Id: Ie246e8ba83e1f4cd01ff707c0cd1e77ea5fe3551
2016-06-15 13:34:23 +03:00
Sergey Kulanov e1f7a5975a [priority] Merge repos with the same priority
1) We need to merge trees with the same priority in order
to catch case when we have package with different versions in
repos with the same priority, eg:
   repo2, priority=10, package=foo-5.3
   repo1, priority=5,  package=asd-1.2
   repo3, priority=10, package=foo-7.1

   --> sort(key=priority) can result:
   repo1, priority=5,  package=asd-1.2
   repo2, priority=10, package=foo-5.3
   repo3, priority=10, package=foo-7.1
   so we will return: foo-5.3 instead we must return foo-7.1

2) priority must be a mandatory parameter for Repository

3) Align rpm default priority _DEFAULT_PRIORITY: N, where
   N is an integer from 1 to 99. The default priority for
   repositories is 99. The repositories with the lowest
   numerical priority number have the highest priority.

Closes-bug: #1590525
Change-Id: I8299b1502775cc68bd1c783487ce7dc802a31a05
2016-06-14 13:42:34 +03:00
Sergey Kulanov 54e5912a88 Fix rpm package equality issue
If package depends on exact equality of =name+version
we also need to search candidates in release section:
   =name+version+(>=release)

eg:
  name: foo-1.1-3
  Requires: bar == 5.5

  then the below package must meet the requiremets:
  name: bar-5.5-56.el7

To make this true we need:
* always search for candidates in package.name and package.provides,
  package.obsoletes, [check PackagesTree.find_all()]
* make an assumption that foo-1.1 and foo-1.1-33 has intersection,
  [check RpmPackageVersion.cmp()]

Tests:
======
gen_package() in stubs.generator must always generate Provides
section, which is always: name=version, eg:
1)   Name: my-python-requests
     Version: 2.10.0
     Release: 1.el7~mos1

   will result PROVIDES:
     my-python-requests = 2.10.0-1.el7~mos1

2)   Name: my-python-urllib3
     Version: 1.15.1
     Release: 7777
     Provides: test

   will result PROVIDES:
     test
     my-python-urllib3 = 1.15.1-7777

Change-Id: Ib26918b2e1d98733fbd49f1fc05403ca5665e3fe
Closes-Bug: #1588399
2016-06-06 15:15:05 +03:00
Sergey Kulanov 07079a0b55 [RPM] Use yum library for packages comparison
Closes-bug: #1587032

Change-Id: I8c857edb839a441b71864d915ef4a814463ed81b
2016-06-01 15:18:08 +03:00
Sergey Kulanov 2bd9a9415d [rpm] Handle the tilde separator in versioning
We need correctly handle the tidle separator in RPM versioning [1]:
eg:
   rpmdev-vercmp 1:8.0.0~b3 1:8.0.0
       1:8.0.0~b3 < 1:8.0.0
       1:8.0.1~b3 > 1:8.0.0
       1:8.0.0~b3 < 1:8.0.1~c1
       1:8.0.0~b3 > 1:8.0.0~a1

[1]. http://blog.jasonantman.com/2014/07/how-yum-and-rpm-compare-versions/

Change-Id: I4a3c867a603a595515e76ee6f4415177a407a0b8
Closes-bug: #1586981
2016-05-30 13:03:34 +03:00
Sergey Kulanov ff85847eca Fix rpm package provides/obsoletes relations
Rpm package can provides and obsoletes the same package name
but with different versions, for eg:
  OBSOLETES:
     python-debtcollector < 1.2.0-1.el7~mos1
     python-debtcollector < 0.7.0-3

  PROVIDES:
     python-debtcollector = 1.2.0-1.el7~mos1
     python2-debtcollector = 1.2.0-1.el7~mos1

This comes from %python_provide macros, eg:
    # rpm --eval '%python_provide python2-foo'
       Provides: python-foo(x86-64) = %{version}-%{release}
       Provides: python-foo = %{version}-%{release}
       Obsoletes: python-foo(x86-64) < %{version}-%{release}
       Obsoletes: python-foo < %{version}-%{release}

    # rpm --eval '%python_provide python-foo'
       Provides: python2-foo = %{version}-%{release}

So we need to align flow

Change-Id: I07b3d8b9a25c5c98e97e697c26e6ae9d1f4960ff
Closes-bug: #1578224
2016-05-26 16:45:31 +03:00
Bulat Gaifullin d661055322 Introduced new scheme to declare requirements
the requirements can contain the following sections:
packages - the list of packages that are needed
repositories - the list of repositories, packages from that are needed
mandatory - boolean flag that uses to automatically copy mandatory packages

Change-Id: Ic26f991c0bf1e9819005cd4bbe7ed40228b2ce1b
2016-05-19 15:26:16 +03:00
Vladimir Kozhukalov c193aa6c60 Fix createrepo workers racing
Sometimes packetary hangs forever when spawning
mutiple createrepo workers. This patch sets
number of workers to 1.

Change-Id: Ica52245b20343428e663e9397ffa8b52d6f46ffc
Closes-Bug: #1552138
2016-04-15 20:13:29 +03:00
Jenkins 6e94aee106 Merge "Fixed error-handling for input data validation" 2016-03-10 07:37:40 +00:00
Bulat Gaifullin 6c5d681b51 Fixed error-handling for input data validation
The property absolute_path was 2.5.1 and not available
in previous versions

Change-Id: Ib4b7e9c9a7f0113275915799624aeef5f6bf8281
Closes-Bug: 1554249
2016-03-09 09:53:27 +03:00
Bulat Gaifullin 161b80907c RPM dirver: Extended provides with list of primary files
Change-Id: I7795bf1259d16ed69d83fe79cbf958624e11d51d
Closes-Bug: 1552148
2016-03-02 13:44:17 +03:00
Bulat Gaifullin ee7193c277 Made release argment of PackageVersion optional
The reletion may ommit the release component of version,
that means any release
Aslo the strignify for PackageVersion was changed,
now ':' is used to separate epoch

Change-Id: I8d74c9cc82720103a5b6d9189f85276bda4ec082
2016-03-01 16:42:05 +03:00
Jenkins d84ae63612 Merge "Fixed retrieve groups.xml on merge repositories" 2016-03-01 13:30:38 +00:00
Bulat Gaifullin d644789905 Fixed retrieve groups.xml on merge repositories
Also add package name when warn about unresolved

Closes-Bug: 1551614
Change-Id: I9545b5c419f03fd9095a6a0abb5d5f21fb3a1ac0
2016-03-01 15:51:08 +03:00
Jenkins 06645584de Merge "Copied groups index upon fork of repository" 2016-02-25 08:47:37 +00:00
Bulat Gaifullin f27bed00a5 Copied groups index upon fork of repository
Also keep original group index durring rebuild repository index.
Build only meta-files without package information durring create and fork
repository

Change-Id: I613ce2084750f02035a13c301aa5812e08a90482
Closes-Bug: 1549413
2016-02-24 22:12:17 +03:00
Jenkins 0188265c39 Merge "Fixed building destination path on forking repository" 2016-02-20 13:47:40 +00:00
Bulat Gaifullin c0f04443b5 Fixed building destination path on forking repository
Also fixed handling 'path' argument for centos repositories

Change-Id: I3292e64d47e0ea58a44d3ce625649d4bb934f3c6
Closes-Bug: 1547820
2016-02-20 13:20:14 +03:00
Vladimir Kozhukalov 1238b33ebd Add package filtering feature
Change-Id: Ia8bf716e9902d250c9fb26bbad325fbe2a204b54
2016-02-20 07:00:52 +03:00
Bulat Gaifullin 34b2ba01b2 Fixed invalid call signature for clone-command
Also make versions as optional field in packages_schema.
Fixed ValidationError formatting when path contains int indexes.

Closes-Bug: 1543503
Change-Id: Idf070b4431bdac2c49340e9e09902a5b39d2a548
2016-02-09 13:36:17 +03:00
Bulat Gaifullin 9e107cde9a Implemented input data validation
Change-Id: I8407cf4cbb69e60b7def891625522f3ca3c822fe
Implements:  blueprint unify-input-data
2016-02-05 17:07:48 +03:00
Bulat Gaifullin d99d1228ef Ignore 404 error on dowloading release file
Some debian repositories does not contain Release file.

Change-Id: Iad69241856e62b74cdee3a68481957ef38c3ab59
Closes-Bug: #1539703
2016-02-02 10:33:54 +03:00
Jenkins ec8a9c8c22 Merge "Fixed retries for HTTP Errors" 2016-02-01 17:30:06 +00:00
Bulat Gaifullin fb3f8ffe02 Fixed retries for HTTP Errors
The code does not properly handle situation when server returns http error.
Also added option 'retries_delay' to specify timeout between retries

Change-Id: I2908f5d774d6d388085e48a965e5888773578530
Closes-Bug: #1539025
2016-02-01 17:40:49 +03:00
Bulat Gaifullin cacf6b75e3 Added command-line interface for create_repository
Change-Id: I22b39adc9a40be8e9d500a81dfef7cbf49f8b651
Implements: blueprint build-repository
2016-01-30 21:42:47 +03:00
Bulat Gaifullin aff85a919e Introduced API to build DEB and RPM repository
Change-Id: I7abc2705f66228e81b36febe146d06988efb7b00
Implements: blueprint build-repository
2016-01-28 18:22:39 +00:00
Bulat Gaifullin 1ce69b4fef Unify input data format
The input data is described in YAML or JSON format
Also implemented support of priorities for repositories

Change-Id: I02f11714ba8880dd06c3ceeadf230c1d812ff0be
Implements:  blueprint unify-input-data
2016-01-28 17:27:15 +00:00
Bulat Gaifullin 0cb734c002 Updated file-headers 2015-12-29 12:51:14 +03:00
Bulat Gaifullin 15cbcf5e0a Update file-headers 2015-12-29 12:08:39 +03:00
Bulat Gaifullin 2873c724bf Drop files are not related to packetary 2015-12-24 13:00:48 +03:00
Bulat Gaifullin af0a66d31c [packetary] Introduce command-line interface
Available commands:
- clone - Clones specified repository(es) to local folder.
  packetary clone -o "http://archive.ubuntu.com/ubuntu trusty main restricted universe multiverse" -t deb -d /var/www/mirror/ubuntu

- packages - Gets the list of packages from repository(es).
  packetary packages -o http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos6-fuel/os -t yum

- unresolved - Get the list of external dependencies for repository.
  packetary unresolved -o "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0 mos8.0 main"

Change-Id: I4b3f3fb7b7f2967cf596ed3c7758cfbbf76dfe73
Partial-Bug: #1487077
2015-12-03 09:45:38 +00:00
Bulat Gaifullin d0b144b0b2 Introduced fuel-mirror
Avaliable commands:
- create - Creates a new local mirrors.
  fuel-mirror create --input ubuntu.yaml

- apply - Applies local mirrors for Fuel-environments.
  fuel-mirror apply --input centos.yaml -e "ENV1"

Change-Id: I6cbf4e581a93b47cf75fb3c89aa1020a352784c8
Closes-Bug: #1487077
Implements: blueprint refactor-local-mirror-scripts
2015-11-26 16:35:47 +03:00