Commit Graph

43 Commits

Author SHA1 Message Date
Jenkins 1508e35df2 Merge "Allow to search mandatory packages in all forest" 2016-07-05 21:23:39 +00:00
Sergey Kulanov f922090dc6 Allow to search mandatory packages in all forest
Update requirements with filter option:

        "mandatory": {
            "enum": ["exact", "newest"]
        }

1) [RPM] We need to search the newest mandatory packages in all
   forest but not from the exact tree, also repo priority should
   be take into account

2) Add priority to gen_repository in tests, since this is
   mandatory parameter

3) [Tests] Add package10 with different version to check mandatory
   values:
    - None
    - "exact"
    - "newest"
  match for mandatory packages

Change-Id: I36a1463b495075bf6c44a46b557c3ac524de0220
Closes-bug: #1585103
2016-07-05 22:45:28 +03:00
Bulat Gaifullin cae6df70bf Added packaging driver to build RPM by using mock
Option 'cache_dir' uses to specify directory where
will be downloaded remote files
The packaging controller allows to use files which
are available via HTTP as source or spec file
Each driver has its own section in input data,
this allows to use same input data for several drivers.

Change-Id: I1fb3b08fe305c3413e5aa4a9213762208a2479da
2016-07-05 16:49:06 +03:00
Bulat Gaifullin 794256f1bc Added CLI for building packages
Usage of command:
packetary build -t "the name of driver" \
    -i "the path of file with input data" \
    -o "the path to output directory" \
    [-C "the config for driver"]

Change-Id: I71df1b8a36342a9ea238d5b6ebb686e5bce71007
2016-06-27 19:31:16 +03:00
Jenkins c88b21ee95 Merge "Introduced packaging API" 2016-06-27 13:48:27 +00:00
Bulat Gaifullin 406bdaa7e2 Introduced packaging API
This change introduces API for building packages.
The commit includes Controller object and interface for PackagingDriver.

Change-Id: I1ce0f746c1cbc5cf8ff29d08d09175acb5d79586
2016-06-22 13:09:23 +03:00
Bulat Gaifullin f86e3216bd Fixed handling http redirects
Override method 'redirect_request' for patching
new request, that has been created on handling
http redirect.

Closes-Bug: 1592464
Change-Id: Id32bc46cce2c0eabb91f645d01378b6da3128469
2016-06-15 18:16:31 +03:00
Sergey Kulanov e1f7a5975a [priority] Merge repos with the same priority
1) We need to merge trees with the same priority in order
to catch case when we have package with different versions in
repos with the same priority, eg:
   repo2, priority=10, package=foo-5.3
   repo1, priority=5,  package=asd-1.2
   repo3, priority=10, package=foo-7.1

   --> sort(key=priority) can result:
   repo1, priority=5,  package=asd-1.2
   repo2, priority=10, package=foo-5.3
   repo3, priority=10, package=foo-7.1
   so we will return: foo-5.3 instead we must return foo-7.1

2) priority must be a mandatory parameter for Repository

3) Align rpm default priority _DEFAULT_PRIORITY: N, where
   N is an integer from 1 to 99. The default priority for
   repositories is 99. The repositories with the lowest
   numerical priority number have the highest priority.

Closes-bug: #1590525
Change-Id: I8299b1502775cc68bd1c783487ce7dc802a31a05
2016-06-14 13:42:34 +03:00
Sergey Kulanov 54e5912a88 Fix rpm package equality issue
If package depends on exact equality of =name+version
we also need to search candidates in release section:
   =name+version+(>=release)

eg:
  name: foo-1.1-3
  Requires: bar == 5.5

  then the below package must meet the requiremets:
  name: bar-5.5-56.el7

To make this true we need:
* always search for candidates in package.name and package.provides,
  package.obsoletes, [check PackagesTree.find_all()]
* make an assumption that foo-1.1 and foo-1.1-33 has intersection,
  [check RpmPackageVersion.cmp()]

Tests:
======
gen_package() in stubs.generator must always generate Provides
section, which is always: name=version, eg:
1)   Name: my-python-requests
     Version: 2.10.0
     Release: 1.el7~mos1

   will result PROVIDES:
     my-python-requests = 2.10.0-1.el7~mos1

2)   Name: my-python-urllib3
     Version: 1.15.1
     Release: 7777
     Provides: test

   will result PROVIDES:
     test
     my-python-urllib3 = 1.15.1-7777

Change-Id: Ib26918b2e1d98733fbd49f1fc05403ca5665e3fe
Closes-Bug: #1588399
2016-06-06 15:15:05 +03:00
Sergey Kulanov 07079a0b55 [RPM] Use yum library for packages comparison
Closes-bug: #1587032

Change-Id: I8c857edb839a441b71864d915ef4a814463ed81b
2016-06-01 15:18:08 +03:00
Sergey Kulanov 2bd9a9415d [rpm] Handle the tilde separator in versioning
We need correctly handle the tidle separator in RPM versioning [1]:
eg:
   rpmdev-vercmp 1:8.0.0~b3 1:8.0.0
       1:8.0.0~b3 < 1:8.0.0
       1:8.0.1~b3 > 1:8.0.0
       1:8.0.0~b3 < 1:8.0.1~c1
       1:8.0.0~b3 > 1:8.0.0~a1

[1]. http://blog.jasonantman.com/2014/07/how-yum-and-rpm-compare-versions/

Change-Id: I4a3c867a603a595515e76ee6f4415177a407a0b8
Closes-bug: #1586981
2016-05-30 13:03:34 +03:00
Bulat Gaifullin d661055322 Introduced new scheme to declare requirements
the requirements can contain the following sections:
packages - the list of packages that are needed
repositories - the list of repositories, packages from that are needed
mandatory - boolean flag that uses to automatically copy mandatory packages

Change-Id: Ic26f991c0bf1e9819005cd4bbe7ed40228b2ce1b
2016-05-19 15:26:16 +03:00
Jenkins 6e94aee106 Merge "Fixed error-handling for input data validation" 2016-03-10 07:37:40 +00:00
Bulat Gaifullin 6c5d681b51 Fixed error-handling for input data validation
The property absolute_path was 2.5.1 and not available
in previous versions

Change-Id: Ib4b7e9c9a7f0113275915799624aeef5f6bf8281
Closes-Bug: 1554249
2016-03-09 09:53:27 +03:00
Bulat Gaifullin 161b80907c RPM dirver: Extended provides with list of primary files
Change-Id: I7795bf1259d16ed69d83fe79cbf958624e11d51d
Closes-Bug: 1552148
2016-03-02 13:44:17 +03:00
Bulat Gaifullin ee7193c277 Made release argment of PackageVersion optional
The reletion may ommit the release component of version,
that means any release
Aslo the strignify for PackageVersion was changed,
now ':' is used to separate epoch

Change-Id: I8d74c9cc82720103a5b6d9189f85276bda4ec082
2016-03-01 16:42:05 +03:00
Jenkins d84ae63612 Merge "Fixed retrieve groups.xml on merge repositories" 2016-03-01 13:30:38 +00:00
Bulat Gaifullin d644789905 Fixed retrieve groups.xml on merge repositories
Also add package name when warn about unresolved

Closes-Bug: 1551614
Change-Id: I9545b5c419f03fd9095a6a0abb5d5f21fb3a1ac0
2016-03-01 15:51:08 +03:00
Jenkins 06645584de Merge "Copied groups index upon fork of repository" 2016-02-25 08:47:37 +00:00
Bulat Gaifullin f27bed00a5 Copied groups index upon fork of repository
Also keep original group index durring rebuild repository index.
Build only meta-files without package information durring create and fork
repository

Change-Id: I613ce2084750f02035a13c301aa5812e08a90482
Closes-Bug: 1549413
2016-02-24 22:12:17 +03:00
Jenkins 0188265c39 Merge "Fixed building destination path on forking repository" 2016-02-20 13:47:40 +00:00
Bulat Gaifullin c0f04443b5 Fixed building destination path on forking repository
Also fixed handling 'path' argument for centos repositories

Change-Id: I3292e64d47e0ea58a44d3ce625649d4bb934f3c6
Closes-Bug: 1547820
2016-02-20 13:20:14 +03:00
Vladimir Kozhukalov 1238b33ebd Add package filtering feature
Change-Id: Ia8bf716e9902d250c9fb26bbad325fbe2a204b54
2016-02-20 07:00:52 +03:00
Bulat Gaifullin 34b2ba01b2 Fixed invalid call signature for clone-command
Also make versions as optional field in packages_schema.
Fixed ValidationError formatting when path contains int indexes.

Closes-Bug: 1543503
Change-Id: Idf070b4431bdac2c49340e9e09902a5b39d2a548
2016-02-09 13:36:17 +03:00
Bulat Gaifullin 9e107cde9a Implemented input data validation
Change-Id: I8407cf4cbb69e60b7def891625522f3ca3c822fe
Implements:  blueprint unify-input-data
2016-02-05 17:07:48 +03:00
Bulat Gaifullin d99d1228ef Ignore 404 error on dowloading release file
Some debian repositories does not contain Release file.

Change-Id: Iad69241856e62b74cdee3a68481957ef38c3ab59
Closes-Bug: #1539703
2016-02-02 10:33:54 +03:00
Jenkins ec8a9c8c22 Merge "Fixed retries for HTTP Errors" 2016-02-01 17:30:06 +00:00
Bulat Gaifullin fb3f8ffe02 Fixed retries for HTTP Errors
The code does not properly handle situation when server returns http error.
Also added option 'retries_delay' to specify timeout between retries

Change-Id: I2908f5d774d6d388085e48a965e5888773578530
Closes-Bug: #1539025
2016-02-01 17:40:49 +03:00
Bulat Gaifullin cacf6b75e3 Added command-line interface for create_repository
Change-Id: I22b39adc9a40be8e9d500a81dfef7cbf49f8b651
Implements: blueprint build-repository
2016-01-30 21:42:47 +03:00
Bulat Gaifullin aff85a919e Introduced API to build DEB and RPM repository
Change-Id: I7abc2705f66228e81b36febe146d06988efb7b00
Implements: blueprint build-repository
2016-01-28 18:22:39 +00:00
Bulat Gaifullin 1ce69b4fef Unify input data format
The input data is described in YAML or JSON format
Also implemented support of priorities for repositories

Change-Id: I02f11714ba8880dd06c3ceeadf230c1d812ff0be
Implements:  blueprint unify-input-data
2016-01-28 17:27:15 +00:00
Bulat Gaifullin 15cbcf5e0a Update file-headers 2015-12-29 12:08:39 +03:00
Bulat Gaifullin 2873c724bf Drop files are not related to packetary 2015-12-24 13:00:48 +03:00
Bulat Gaifullin af0a66d31c [packetary] Introduce command-line interface
Available commands:
- clone - Clones specified repository(es) to local folder.
  packetary clone -o "http://archive.ubuntu.com/ubuntu trusty main restricted universe multiverse" -t deb -d /var/www/mirror/ubuntu

- packages - Gets the list of packages from repository(es).
  packetary packages -o http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos6-fuel/os -t yum

- unresolved - Get the list of external dependencies for repository.
  packetary unresolved -o "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0 mos8.0 main"

Change-Id: I4b3f3fb7b7f2967cf596ed3c7758cfbbf76dfe73
Partial-Bug: #1487077
2015-12-03 09:45:38 +00:00
Bulat Gaifullin d0b144b0b2 Introduced fuel-mirror
Avaliable commands:
- create - Creates a new local mirrors.
  fuel-mirror create --input ubuntu.yaml

- apply - Applies local mirrors for Fuel-environments.
  fuel-mirror apply --input centos.yaml -e "ENV1"

Change-Id: I6cbf4e581a93b47cf75fb3c89aa1020a352784c8
Closes-Bug: #1487077
Implements: blueprint refactor-local-mirror-scripts
2015-11-26 16:35:47 +03:00
Bulat Gaifullin 994fed9b1e [packetary] Introduce driver for rpm-repository
Change-Id: Ib9b9208964dba25d585fb9345a6ef95e7782a9ff
Implements: blueprint refactor-local-mirror-scripts
Partial-Bug: #1487077
2015-11-19 16:32:31 +03:00
Bulat Gaifullin aa570b1f98 [packetary] Introduce driver for debian-repository
Change-Id: Icb8f6964bd0a9080ace50b4786ac76ae5dbb9976
Implements: blueprint refactor-local-mirror-scripts
Partial-Bug: #1487077
2015-11-16 15:00:55 +03:00
Bulat Gaifullin 671af8e611 [packetary] Repository class
class Repository composes from:
* RepositryDriver - low-level support for physical repository. deb, yum, etc.
* RepositoryController - infrastcuture method to communicate with driver
* RepositoryApi - high-level class, that provides methods to work with repository

Change-Id: Iaf868fca982d91089e369d13a6fb381ff879ea73
Implements: blueprint refactor-local-mirror-scripts
Partial-Bug: #1487077
2015-11-09 16:22:18 +03:00
Bulat Gaifullin 60ba0e6e62 [packetary] Infrastructure improvments.
* Added retry for 404 (HTTP Status).
* Added method for downloading the file to the ConnectionManager.
* AsynchronousSection collects all exceptions with traceback and
     log it after all tasks has been completed.
* generator.gen_relation returns object instead of list
* GzipDecompress: fixed case when there is not enough data to decompress next chunk

Change-Id: Ie2014aeee0c4776ed411ef6f28a996ef2d029c95
2015-11-06 12:24:34 +03:00
Bulat Gaifullin 6a9c463042 [packetary] Packages indexer
Change-Id: I981fe7ae8f0aba8314475c3633f325e18d5e3bb5
Implements: blueprint refactor-local-mirror-scripts
Partial-Bug: #1487077
2015-11-03 17:29:10 +03:00
Bulat Gaifullin 56689ae7cf [packetary] Infrastructure
Introduce infrastructure:
* Executor (The Asynchronous task executor)
* ConnectionsPool (The pool of network connections)
* ResumeableStream - allows resume streaming when error occures

Change-Id: Ia68d60c2b9d685820a4c1916c8f1aa724f3a3f91
Implements: blueprint refactor-local-mirror-scripts
Partial-Bug: #1487077
2015-10-27 17:41:57 +03:00
Bulat Gaifullin 5b9488899f CI fix issues
* Fix module import in packetary tests
* Fix PEP8 in perestroika code

Change-Id: I0a9fc3cbbe5a59152f3733db235c23a242c58b04
Implements: blueprint refactor-local-mirror-scripts
Partial-Bug: #1487077
2015-10-27 14:02:02 +00:00
Bulat Gaifullin b6d09c7ead [packetary] initial commit
the project structure was created by cookiecutter.

Implements: blueprint refactor-local-mirror-scripts
Partial-Bug: #1487077

Change-Id: Ib33955ba1d4013bc1491a503610885367924b0e9
2015-10-20 19:33:48 +00:00