Fuel-mirror project is deprecated

Instead of fuel-mirror it is recommended to use
Packetary to create partial repository mirrors
and use Python-fuelclient to manage Fuel metadata.
Fuel-mirror is going to be deprecated and
master branch won't be supported any more.

However we still need test jobs to support stable
branches.

Change-Id: I9fb76f3606c0888dd1c730a40e1bbb3b8c0f02ed
This commit is contained in:
Vladimir Kozhukalov 2016-06-30 12:12:22 +03:00
parent 24b17cce5c
commit afaa4ca6c7
129 changed files with 5 additions and 9756 deletions

View File

@ -1,17 +0,0 @@
If you would like to contribute to the development of OpenStack, you must
follow the steps in this page:
http://docs.openstack.org/infra/manual/developers.html
If you already have a good understanding of how the system works and your
OpenStack accounts are set up, you can skip to the development workflow
section of this documentation to learn how changes to OpenStack should be
submitted for review via the Gerrit tool:
http://docs.openstack.org/infra/manual/developers.html#development-workflow
Pull requests submitted through GitHub will be ignored.
Bugs should be filed on Launchpad, not GitHub:
https://bugs.launchpad.net/fuel

View File

@ -1,4 +0,0 @@
fuel-mirror Style Commandments
==============================
Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/

340
LICENSE
View File

@ -1,340 +0,0 @@
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc., <http://fsf.org/>
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
{description}
Copyright (C) {year} {fullname}
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
{signature of Ty Coon}, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License.

View File

@ -26,49 +26,6 @@ description:
maintainers:
- contrib/:
- name: Bulat Gaifullin
email: bgaifullin@mirantis.com
IRC: bgaifullin
- name: Vladimir Kozhukalov
email: vkozhukalov@mirantis.com
IRC: kozhukalov
- debian/: &packaging_team
- name: Mikhail Ivanov
email: mivanov@mirantis.com
IRC: mivanov
- name: Artem Silenkov
email: asilenkov@mirantis.com
IRC: asilenkov
- name: Alexander Tsamutali
email: atsamutali@mirantis.com
IRC: astsmtl
- name: Daniil Trishkin
email: dtrishkin@mirantis.com
IRC: dtrishkin
- name: Ivan Udovichenko
email: iudovichenko@mirantis.com
IRC: tlbr
- name: Igor Yozhikov
email: iyozhikov@mirantis.com
IRC: IgorYozhikov
- packetary/:
- name: Bulat Gaifullin
email: bgaifullin@mirantis.com
IRC: bgaifullin
- name: Vladimir Kozhukalov
email: vkozhukalov@mirantis.com
IRC: kozhukalov
- perestroika/: &build_team
- name: Dmitry Burmistrov
email: dburmistrov@mirantis.com
@ -77,5 +34,3 @@ maintainers:
- name: Sergey Kulanov
email: skulanov@mirantis.com
IRC: SergK
- specs/: *packaging_team

View File

@ -1,5 +0,0 @@
include AUTHORS
include ChangeLog
include *requirements.txt
global-exclude *.pyc

View File

@ -2,38 +2,6 @@
Repository structure
====================
* contrib/fuel_mirror
It is a command line utility that provides the same functionality
and user interface as deprecated fuel-createmirror. It provides
two major features:
* clone/build mirror (full or partial)
* update repository configuration in nailgun
First one is a matter of packetary while second one should be left
totally up to fuelclient. So this module is to be deprecated soon
in favor of packetary and fuelclient.
WARNING: It is not designed to be used on 'live' repositories
that are available to clients during synchronization. That means
repositories will be inconsistent during the update. Please use these
scripts in conjunction with snapshots, on inactive repos, etc.
* debian
Specs for DEB packages.
* doc
Documentation for packetary module.
* packetary
It is a Python library and command line utilty that allows
one to clone and build rpm/deb repositories.
Features:
* Common interface for different package-managers.
* Utility to build dependency graph for package(s).
* Utility to create mirror of repository according to dependency graph.
* perestroika
It is a set shell/python script that are used to build DEB/RPM
packages. These scripts are widely used by Fuel Packaging CI.
* specs
Specs for RPM packages.

View File

@ -1,2 +0,0 @@
[python: **.py]

View File

@ -1,8 +0,0 @@
include AUTHORS
include ChangeLog
recursive-include etc *
exclude .gitignore
exclude .gitreview
global-exclude *.pyc

View File

@ -1,16 +0,0 @@
===========
fuel_mirror
===========
The fuel-mirror is utility, that allows to create local repositories
with packages are required for the OpenStack deployment.
* Free software: Apache license
* Documentation: http://docs.openstack.org/developer/fuel-mirror
* Source: http://git.openstack.org/cgit/openstack/fuel-mirror/
* Bugs: http://bugs.launchpad.net/fuel
Features
--------
* TODO

View File

@ -1,2 +0,0 @@
[python: **.py]

View File

@ -1,55 +0,0 @@
fuel_release_match:
version: $openstack_version
operating_system: CentOS
repos:
- &centos
name: "centos"
uri: "http://vault.centos.org/7.1.1503/os/x86_64/"
type: "rpm"
priority: null
- &centos_updates
name: "centos-updates"
uri: "http://vault.centos.org/7.1.1503/updates/x86_64/"
type: "rpm"
priority: null
- &mos
name: "mos"
uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/os/x86_64/"
type: "rpm"
priority: null
- &mos_updates
name: "mos-updates"
uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/updates/x86_64/"
type: "rpm"
priority: null
- &mos_security
name: "mos-security"
uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/security/x86_64/"
type: "rpm"
priority: null
- &mos_holdback
name: "mos-holdback"
uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/holdback/x86_64/"
type: "rpm"
priority: null
groups:
mos:
- *mos
- *mos_updates
- *mos_security
- *mos_holdback
centos:
- *centos
- *centos_updates
inheritance:
centos: mos

View File

@ -1,158 +0,0 @@
# GLOBAL variables
ubuntu_baseurl: &ubuntu_baseurl http://archive.ubuntu.com/ubuntu
mos_baseurl: &mos_baseurl http://mirror.fuel-infra.org/mos-repos/ubuntu/$mos_version
fuel_release_match:
version: $openstack_version
operating_system: Ubuntu
# Main is a required parameter which defines what repository will be used
# for images creation and that mirror should contain all packages for minimal
# system creation.
repos:
- &ubuntu
main: true
name: "ubuntu"
uri: *ubuntu_baseurl
suite: "trusty"
section: "main multiverse restricted universe"
type: "deb"
priority: null
- &ubuntu_updates
name: "ubuntu-updates"
uri: *ubuntu_baseurl
suite: "trusty-updates"
section: "main multiverse restricted universe"
type: "deb"
priority: null
- &ubuntu_security
name: "ubuntu-security"
uri: *ubuntu_baseurl
suite: "trusty-security"
section: "main multiverse restricted universe"
type: "deb"
priority: null
- &mos
name: "mos"
uri: *mos_baseurl
suite: "mos$mos_version"
section: "main restricted"
type: "deb"
priority: 1000
- &mos_updates
name: "mos-updates"
uri: *mos_baseurl
suite: "mos$mos_version-updates"
section: "main restricted"
type: "deb"
priority: 1000
- &mos_security
name: "mos-security"
uri: *mos_baseurl
suite: "mos$mos_version-security"
section: "main restricted"
type: "deb"
priority: 1000
- &mos_holdback
name: "mos-holdback"
uri: *mos_baseurl
suite: "mos$mos_version-holdback"
section: "main restricted"
type: "deb"
priority: 1000
# Packages are required to build bootstrap images for a system.
# The mirror should contiain such packages in addition to local mirror.
packages: &packages
- "acpi-support"
- "anacron"
- "aptitude"
- "atop"
- "acct"
- "bash-completion"
- "bc"
- "build-essential"
- "cloud-init"
- "conntrackd"
- "cpu-checker"
- "cpufrequtils"
- "debconf-utils"
- "devscripts"
- "fping"
- "git"
- "grub-pc"
- "htop"
- "hwloc"
- "ifenslave"
- "iperf"
- "iptables-persistent"
- "irqbalance"
- "language-pack-en"
- "libapache2-mod-fastcgi"
- "libnss3-tools"
- "linux-firmware-nonfree"
- "linux-headers-generic-lts-trusty"
- "linux-image-generic-lts-trusty"
- "live-boot"
- "livecd-rootfs"
- "mc"
- "memcached"
- "monit"
- "msmtp-mta"
- "multipath-tools"
- "multipath-tools-boot"
- "nginx"
- "ntp"
- "openssh-server"
- "percona-toolkit"
- "percona-xtrabackup"
- "pm-utils"
- "puppet"
- "python-lesscpy"
- "python-pip"
- "rsyslog-gnutls"
- "rsyslog-relp"
- "screen"
- "squashfs-tools"
- "swift-plugin-s3"
- "sysfsutils"
- "sysstat"
- "telnet"
- "tmux"
- "traceroute"
- "ubuntu-standard"
- "vim"
- "virt-what"
- "xinetd"
- "xmlstarlet"
- "tftpd-hpa"
- "syslinux"
groups:
mos:
- *mos
- *mos_updates
- *mos_security
- *mos_holdback
ubuntu:
- *ubuntu
- *ubuntu_updates
- *ubuntu_security
inheritance:
ubuntu: mos
osnames:
mos: ubuntu
requirements:
ubuntu: *packages

View File

@ -1,79 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
sys.path.insert(0, os.path.abspath('../..'))
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
#'sphinx.ext.intersphinx',
'oslosphinx'
]
# autodoc generation is a bit aggressive and a nuisance when doing heavy
# text edit cycles.
# execute "export SPHINX_DEBUG=1" in your terminal to disable
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'fuel_mirror'
copyright = u'2015, Mirantis, Inc'
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = True
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output --------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
# html_theme = '_theme'
# html_static_path = ['static']
# Output file base name for HTML help builder.
htmlhelp_basename = '%sdoc' % project
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto/manual]).
latex_documents = [
('index',
'%s.tex' % project,
u'%s Documentation' % project,
u'OpenStack Foundation', 'manual'),
]
# Example configuration for intersphinx: refer to the Python standard library.
#intersphinx_mapping = {'http://docs.python.org/': None}

View File

@ -1,4 +0,0 @@
============
Contributing
============
.. include:: ../../../../CONTRIBUTING.rst

View File

@ -1,25 +0,0 @@
.. fuel_mirror documentation master file, created by
sphinx-quickstart on Tue Jul 9 22:26:36 2013.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to fuel_mirror's documentation!
========================================================
Contents:
.. toctree::
:maxdepth: 2
readme
installation
usage
contributing
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

View File

@ -1,12 +0,0 @@
============
Installation
============
At the command line::
$ pip install fuel_mirror
Or, if you have virtualenvwrapper installed::
$ mkvirtualenv fuel_mirror
$ pip install fuel_mirror

View File

@ -1 +0,0 @@
.. include:: ../../README.rst

View File

@ -1,7 +0,0 @@
========
Usage
========
To use fuel_mirror in a project::
import fuel_createmirror

View File

@ -1,11 +0,0 @@
threads_num: 10
ignore_errors_num: 2
retries_num: 3
target_dir: "/var/www/nailgun/mirrors"
pattern_dir: "/usr/share/fuel-mirror"
base_url: "http://{FUEL_SERVER_IP}:8080/mirrors/"
# uncomment if need
# http_proxy: null
# https_proxy: null
# fuel_server: 10.20.0.2

View File

@ -1,26 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import pbr.version
try:
__version__ = pbr.version.VersionInfo(
'fuel_mirror').version_string()
except Exception as e:
print("ERROR", e)
__version__ = "0.0.0-test"

View File

@ -1,157 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from cliff import app
from cliff.commandmanager import CommandManager
import yaml
import fuel_mirror
from fuel_mirror.common import accessors
from fuel_mirror.common import utils
class Application(app.App):
"""Main cliff application class.
Performs initialization of the command manager and
configuration of basic engines.
"""
config = None
fuel = None
repo_manager_accessor = None
sources = None
versions = None
def build_option_parser(self, description, version, argparse_kwargs=None):
"""Specifies common cmdline arguments."""
p_inst = super(Application, self)
parser = p_inst.build_option_parser(description=description,
version=version,
argparse_kwargs=argparse_kwargs)
parser.add_argument(
"--config",
default="/etc/fuel-mirror/config.yaml",
metavar="PATH",
help="Path to config file."
)
parser.add_argument(
"-S", "--fuel-server",
metavar="FUEL-SERVER",
help="The public address of Fuel Master."
)
parser.add_argument(
"--fuel-user",
help="Fuel Master admin login."
" Alternatively, use env var KEYSTONE_USER)."
)
parser.add_argument(
"--fuel-password",
help="Fuel Master admin password."
" Alternatively, use env var KEYSTONE_PASSWORD)."
)
return parser
def initialize_app(self, argv):
"""Initialises common options."""
with open(self.options.config, "r") as stream:
self.config = yaml.load(stream)
self._initialize_fuel_accessor()
self._initialize_repo_manager()
def _initialize_repo_manager(self):
self.repo_manager_accessor = accessors.get_packetary_accessor(
threads_num=int(self.config.get('threads_num', 0)),
retries_num=int(self.config.get('retries_num', 0)),
ignore_errors_num=int(self.config.get('ignore_errors_num', 0)),
http_proxy=self.config.get('http_proxy'),
https_proxy=self.config.get('https_proxy'),
)
def _initialize_fuel_accessor(self):
fuel_default = utils.get_fuel_settings()
fuel_server = utils.first(
self.options.fuel_server,
self.config.get("fuel_server"),
fuel_default.get("server")
)
fuel_user = utils.first(
self.options.fuel_user,
fuel_default.get("user")
)
fuel_password = utils.first(
self.options.fuel_password,
fuel_default.get("password")
)
if not fuel_server:
for option in ("mos_version", "openstack_version"):
if not self.config.setdefault(option, ''):
self.LOG.warning(
"The option '{0}' is not defined."
"Please specify the option 'fuel-server' or '{0}'."
.format(option)
)
return
self.config["base_url"] = self.config["base_url"].format(
FUEL_SERVER_IP=fuel_server.split(":", 1)[0]
)
self.fuel = accessors.get_fuel_api_accessor(
fuel_server,
fuel_user,
fuel_password
)
fuel_ver = self.fuel.FuelVersion.get_all_data()
self.config.setdefault(
'mos_version', fuel_ver['release']
)
self.config.setdefault(
'openstack_version', fuel_ver['openstack_version']
)
def main(argv=None):
"""Entry point."""
return Application(
description="The utility to create local mirrors.",
version=fuel_mirror.__version__,
command_manager=CommandManager("fuel_mirror", convert_underscores=True)
).run(argv)
def debug(name, cmd_class, argv=None):
"""Helps to debug command."""
import sys
if argv is None:
argv = sys.argv[1:]
argv = [name] + argv + ["-v", "-v", "--debug"]
cmd_mgr = CommandManager("test_fuel_mirror", convert_underscores=True)
cmd_mgr.add_command(name, cmd_class)
return Application(
description="The fuel mirror utility test.",
version="0.0.1",
command_manager=cmd_mgr
).run(argv)

View File

@ -1,203 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import six
from packetary.library.utils import localize_repo_url
from fuel_mirror.commands.base import BaseCommand
from fuel_mirror.common.utils import is_subdict
from fuel_mirror.common.utils import lists_merge
class ApplyCommand(BaseCommand):
"""Applies local mirrors for Fuel-environments."""
def get_parser(self, prog_name):
parser = super(ApplyCommand, self).get_parser(prog_name)
parser.add_argument(
"--default",
dest="set_default",
action="store_true",
default=False,
help="Set as default repository."
)
parser.add_argument(
"--replace",
dest="replace",
action="store_true",
default=False,
help="Replace default repos with generated mirrors."
)
parser.add_argument(
"-e", "--env",
dest="env", nargs="+",
help="Fuel environment ID to update, "
"by default applies for all environments."
)
return parser
def take_action(self, parsed_args):
if self.app.fuel is None:
raise ValueError("Please specify the fuel-server option.")
data = self.load_data(parsed_args)
base_url = self.app.config["base_url"]
release_match = data["fuel_release_match"]
replace_repos = parsed_args.replace
localized_repos = []
for _, repos in self.get_groups(parsed_args, data):
for repo_data in repos:
new_data = repo_data.copy()
new_data['uri'] = localize_repo_url(
base_url, repo_data['uri']
)
localized_repos.append(new_data)
localized_repos.sort(key=lambda x: not x.pop('main', False))
self.update_clusters(
parsed_args.env,
localized_repos,
release_match,
replace_repos=replace_repos)
if parsed_args.set_default:
self.update_release_repos(
localized_repos,
release_match,
replace_repos=replace_repos)
self.app.stdout.write(
"Operations have been completed successfully.\n"
)
def update_clusters(self,
ids,
repositories,
release_match,
replace_repos=False):
"""Applies repositories for existing clusters.
:param ids: the cluster ids.
:param repositories: the meta information of repositories
:param release_match: The pattern to check Fuel Release
"""
self.app.stdout.write("Updating the Cluster repositories...\n")
if ids:
clusters = self.app.fuel.Environment.get_by_ids(ids)
else:
clusters = self.app.fuel.Environment.get_all()
for cluster in clusters:
releases = six.moves.filter(
lambda x: is_subdict(release_match, x.data),
self.app.fuel.Release.get_by_ids([cluster.data["release_id"]])
)
if next(releases, None) is None:
continue
modified = self._update_repository_settings(
cluster.get_settings_data(),
repositories,
replace_repos=replace_repos)
if modified:
self.app.LOG.info(
"Try to update the Cluster '%s'",
cluster.data['name']
)
self.app.LOG.debug(
"The modified cluster attributes: %s",
modified
)
cluster.set_settings_data(modified)
def update_release_repos(self,
repositories,
release_match,
replace_repos=False):
"""Applies repositories for existing default settings.
:param repositories: the meta information of repositories
:param release_match: The pattern to check Fuel Release
"""
self.app.stdout.write("Updating the release repositories...\n")
releases = six.moves.filter(
lambda x: is_subdict(release_match, x.data),
self.app.fuel.Release.get_all()
)
for release in releases:
modified = self._update_repository_settings(
release.data["attributes_metadata"],
repositories,
replace_repos=replace_repos)
if modified:
release.data["attributes_metadata"] = modified
self.app.LOG.info(
"Try to update the Release '%s'",
release.data['name']
)
self.app.LOG.debug(
"The modified release attributes: %s",
release.data
)
# TODO(need to add method for release object)
release.connection.put_request(
release.instance_api_path.format(release.id),
release.data
)
def _update_repository_settings(self,
settings,
repositories,
replace_repos=False):
"""Updates repository settings.
:param settings: the target settings
:param repositories: the meta of repositories
"""
editable = settings["editable"]
if 'repo_setup' not in editable:
self.app.LOG.info('Attributes are read-only.')
return
repos_attr = editable["repo_setup"]["repos"]
if replace_repos:
repos_attr['value'] = repositories
else:
lists_merge(repos_attr['value'], repositories, "name")
# NOTE(akostrikov) That assignment is only for informational purpose.
settings["editable"]["repo_setup"]["repos"] = repos_attr
return settings
def debug(argv=None):
"""Helper for debugging Apply command."""
from fuel_mirror.app import debug
return debug("apply", ApplyCommand, argv)
if __name__ == "__main__":
debug()

View File

@ -1,125 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os.path
from cliff import command
from jsonschema import validate
from jsonschema import ValidationError
import six
from fuel_mirror.common.utils import load_input_data
from fuel_mirror.schemas.input_data_schema import SCHEMA
class BaseCommand(command.Command):
"""The Base command for fuel-mirror."""
REPO_ARCH = "x86_64"
@property
def stdout(self):
"""Shortcut for self.app.stdout."""
return self.app.stdout
def get_parser(self, prog_name):
"""Specifies common options."""
parser = super(BaseCommand, self).get_parser(prog_name)
input_group = parser.add_mutually_exclusive_group(required=True)
input_group.add_argument(
'-I', '--input-file',
metavar='PATH',
help='The path to file with input data.')
input_group.add_argument(
'-P', '--pattern',
metavar='NAME',
help='The builtin input file name.'
)
parser.add_argument(
"-G", "--group",
dest="groups",
required=True,
nargs='+',
help="The name of repository groups."
)
return parser
def resolve_input_pattern(self, pattern):
"""Gets the full path to input file by pattern.
:param pattern: the config file name without ext
:return: the full path
"""
return os.path.join(
self.app.config['pattern_dir'], pattern + ".yaml"
)
@staticmethod
def validate_data(data, schema):
"""Validate the input data using jsonschema validation.
:param data: a data to validate represented as a dict
:param schema: a schema to validate represented as a dict;
must be in JSON Schema Draft 4 format.
"""
try:
validate(data, schema)
except ValidationError as ex:
if len(ex.path) > 0:
join_ex_path = '.'.join(six.text_type(x) for x in ex.path)
detail = ("Invalid input for field/attribute {0}."
" Value: {1}. {2}").format(join_ex_path,
ex.instance, ex.message)
else:
detail = ex.message
raise ValidationError(detail)
def load_data(self, parsed_args):
"""Load the input data.
:param parsed_args: the command-line arguments
:return: the input data
"""
if parsed_args.pattern:
input_file = self.resolve_input_pattern(parsed_args.pattern)
else:
input_file = parsed_args.input_file
data = load_input_data(
input_file,
mos_version=self.app.config["mos_version"],
openstack_version=self.app.config["openstack_version"]
)
self.validate_data(data, SCHEMA)
return data
@classmethod
def get_groups(cls, parsed_args, data):
"""Gets repository groups from input data.
:param parsed_args: the command-line arguments
:param data: the input data
:return: the sequence of pairs (group_name, repositories)
"""
all_groups = data['groups']
return (
(x, all_groups[x]) for x in parsed_args.groups if x in all_groups
)

View File

@ -1,79 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from fuel_mirror.commands.base import BaseCommand
from fuel_mirror.common.url_builder import get_url_builder
class CreateCommand(BaseCommand):
"""Creates a new local mirrors."""
def take_action(self, parsed_args):
"""See the Command.take_action."""
data = self.load_data(parsed_args)
repos_reqs = data.get('requirements', {})
inheritance = data.get('inheritance', {})
target_dir = self.app.config["target_dir"]
total_stats = None
for group_name, repos in self.get_groups(parsed_args, data):
url_builder = get_url_builder(repos[0]["type"])
repo_manager = self.app.repo_manager_accessor(
repos[0]["type"], self.REPO_ARCH
)
if group_name in inheritance:
child_group = inheritance[group_name]
dependencies = [
url_builder.get_repo_url(x)
for x in data['groups'][child_group]
]
else:
dependencies = None
stat = repo_manager.clone_repositories(
[url_builder.get_repo_url(x) for x in repos],
target_dir,
dependencies,
repos_reqs.get(group_name)
)
if total_stats is None:
total_stats = stat
else:
total_stats += stat
if total_stats is not None:
self.stdout.write(
"Packages processed: {0.copied}/{0.total}\n"
.format(total_stats)
)
else:
self.stdout.write(
"No packages.\n"
)
def debug(argv=None):
"""Helper for debugging Create command."""
from fuel_mirror.app import debug
return debug("create", CreateCommand, argv)
if __name__ == "__main__":
debug()

View File

@ -1,64 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import functools
import os
def get_packetary_accessor(**kwargs):
"""Gets the configured repository manager.
:param kwargs: The packetary configuration parameters.
"""
import packetary
return functools.partial(
packetary.RepositoryApi.create,
packetary.Context(packetary.Configuration(**kwargs))
)
def get_fuel_api_accessor(address=None, user=None, password=None):
"""Gets the fuel client api accessor.
:param address: The address of Fuel Master node.
:param user: The username to access to the Fuel Master node.
:param user: The password to access to the Fuel Master node.
"""
if address:
host_and_port = address.split(":")
os.environ["SERVER_ADDRESS"] = host_and_port[0]
if len(host_and_port) > 1:
os.environ["LISTEN_PORT"] = host_and_port[1]
if user is not None:
os.environ["KEYSTONE_USER"] = user
if password is not None:
os.environ["KEYSTONE_PASS"] = password
# import fuelclient.ClientAPI after configuring
# environment variables
try:
from fuelclient import objects
except ImportError:
raise RuntimeError(
"fuelclient module seems not installed. "
"This action requires it to be available."
)
return objects

View File

@ -1,59 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
def get_url_builder(repotype):
"""Gets the instance of RepoUrlBuilder.
:param repotype: the type of repository: rpm|deb
:return: the RepoBuilder implementation
"""
return {
"deb": AptRepoUrlBuilder,
"rpm": YumRepoUrlBuilder
}[repotype]
class RepoUrlBuilder(object):
REPO_FOLDER = "mirror"
@classmethod
def get_repo_url(cls, repo_data):
"""Gets the url with replaced variable holders.
:param repo_data: the repositories`s meta data
:return: the full repository`s url
"""
class AptRepoUrlBuilder(RepoUrlBuilder):
"""URL builder for apt-repository(es)."""
@classmethod
def get_repo_url(cls, repo_data):
return " ".join(
repo_data[x] for x in ("uri", "suite", "section")
)
class YumRepoUrlBuilder(RepoUrlBuilder):
"""URL builder for Yum repository(es)."""
@classmethod
def get_repo_url(cls, repo_data):
return repo_data["uri"]

View File

@ -1,106 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from string import Template
import six
import yaml
def lists_merge(main, patch, key):
"""Merges the list of dicts with same keys.
>>> lists_merge([{"a": 1, "c": 2}], [{"a": 1, "c": 3}], key="a")
[{'a': 1, 'c': 3}]
:param main: the main list
:type main: list
:param patch: the list of additional elements
:type patch: list
:param key: the key for compare
"""
main_idx = dict(
(x[key], i) for i, x in enumerate(main)
)
patch_idx = dict(
(x[key], i) for i, x in enumerate(patch)
)
for k in sorted(patch_idx):
if k in main_idx:
main[main_idx[k]].update(patch[patch_idx[k]])
else:
main.append(patch[patch_idx[k]])
return main
def is_subdict(dict1, dict2):
"""Checks that dict1 is subdict of dict2.
>>> is_subdict({"a": 1}, {'a': 1, 'b': 1})
True
:param dict1: the candidate
:param dict2: the super dict
:return: True if all keys from dict1 are present
and has same value in dict2 otherwise False
"""
for k, v in six.iteritems(dict1):
if k not in dict2 or dict2[k] != v:
return False
return True
def first(*args):
"""Get first not empty value.
>>> first(0, 1) == next(iter(filter(None, [0, 1])))
True
:param args: the list of arguments
:return first value that bool(v) is True, None if not found.
"""
for arg in args:
if arg:
return arg
def get_fuel_settings():
"""Gets the fuel settings from astute container, if it is available."""
try:
with open("/etc/fuel/astute.yaml", "r") as fd:
settings = yaml.load(fd)
return {
"server": settings.get("ADMIN_NETWORK", {}).get("ipaddress"),
"user": settings.get("FUEL_ACCESS", {}).get("user"),
"password": settings.get("FUEL_ACCESS", {}).get("password")
}
except (OSError, IOError):
return {}
def load_input_data(input_file, **kwargs):
"""Load yaml file and parse it to dict with replacement by kwargs.
:param input_file: name of file to parse fuel mirror template
:param kwargs: arguments to substitute template
:return: processed from yaml file dict.
"""
with open(input_file, "r") as fd:
return yaml.load(Template(fd.read()).safe_substitute(**kwargs))

View File

@ -1,120 +0,0 @@
SCHEMA = {
"$schema": "http://json-schema.org/draft-04/schema#",
"definitions": {
"DEB_REPO_SCHEMA": {
"type": "object",
"required": [
"name",
"uri",
"suite",
"section"
],
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string",
"enum": ["deb"]
},
"uri": {
"type": "string"
},
"priority": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
]
},
"suite": {
"type": "string"
},
"section": {
"type": "string"
},
}
},
"RPM_REPO_SCHEMA": {
"type": "object",
"required": [
"name",
"uri",
],
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string",
"enum": ["rpm"]
},
"uri": {
"type": "string"
},
"priority": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
]
},
}
},
"REPO_SCHEMA": {
"anyOf":
[
{"$ref": "#/definitions/DEB_REPO_SCHEMA"},
{"$ref": "#/definitions/RPM_REPO_SCHEMA"}
]
},
"REPOS_SCHEMA": {
"type": "array", "items": {"$ref": "#/definitions/REPO_SCHEMA"}
}
},
"type": "object",
"required": [
"groups",
],
"properties": {
"fuel_release_match": {
"type": "object",
"properties": {
"operating_system": {
"type": "string"
}
},
"required": [
"operating_system"
]
},
"requirements": {
"type": "object",
"patternProperties": {
"^[0-9a-z_-]+$": {"type": "array"}
},
"additionalProperties": False,
},
"groups": {
"type": "object",
"patternProperties": {
"^[0-9a-z_-]+$": {"$ref": "#/definitions/REPOS_SCHEMA"}
},
"additionalProperties": False,
},
"inheritance": {
"type": "object",
"patternProperties": {
"^[0-9a-z_-]+$": {"type": "string"}
},
"additionalProperties": False,
}
}
}

View File

@ -1,33 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
try:
import unittest2 as unittest
except ImportError:
import unittest
class TestCase(unittest.TestCase):
"""Test case base class for all unit tests."""
def assertNotRaises(self, exception, method, *args, **kwargs):
try:
method(*args, **kwargs)
except exception as e:
self.fail("Unexpected error: {0}".format(e))

View File

@ -1,22 +0,0 @@
fuel_release_match:
operating_system: CentOS
inheritance:
centos: mos
groups:
mos:
- name: "mos"
type: "rpm"
uri: "http://localhost/mos$mos_version/x86_64"
priority: 10
centos:
- name: "centos"
type: "rpm"
uri: "http://localhost/centos/os/x86_64"
priority: 5
requirements:
centos:
- "package_rpm"

View File

@ -1,7 +0,0 @@
threads_num: 1
ignore_errors_num: 2
retries_num: 3
http_proxy: "http://localhost"
https_proxy: "https://localhost"
target_dir: "/var/www/"
base_url: "http://{FUEL_SERVER_IP}:8080/"

View File

@ -1,9 +0,0 @@
fuel_release_match:
operating_system: Ubuntu
inheritance:
ubuntu: mos
requirements:
ubuntu:
- "package_deb"

View File

@ -1,27 +0,0 @@
fuel_release_match:
operating_system: Ubuntu
inheritance:
ubuntu: mos
groups:
mos:
- name: "mos"
type: "deb"
uri: "http://localhost/mos"
suite: "mos$mos_version"
section: "main restricted"
priority: 1000
ubuntu:
- name: "ubuntu"
type: "deb"
main: true
uri: "http://localhost/ubuntu"
suite: "trusty"
section: "main multiverse restricted universe"
priority: 500
requirements:
ubuntu:
- "package_deb"

View File

@ -1,92 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import mock
from fuel_mirror.common import accessors
from fuel_mirror.tests import base
class TestAccessors(base.TestCase):
def test_get_packetary_accessor(self):
packetary = mock.MagicMock()
with mock.patch.dict("sys.modules", packetary=packetary):
accessor = accessors.get_packetary_accessor(
http_proxy="http://localhost",
https_proxy="https://localhost",
retries_num=1,
threads_num=2,
ignore_errors_num=3
)
accessor("deb")
accessor("yum")
packetary.Configuration.assert_called_once_with(
http_proxy="http://localhost",
https_proxy="https://localhost",
retries_num=1,
threads_num=2,
ignore_errors_num=3
)
packetary.Context.assert_called_once_with(
packetary.Configuration()
)
self.assertEqual(2, packetary.RepositoryApi.create.call_count)
packetary.RepositoryApi.create.assert_any_call(
packetary.Context(), "deb"
)
packetary.RepositoryApi.create.assert_any_call(
packetary.Context(), "yum"
)
@mock.patch("fuel_mirror.common.accessors.os")
def test_get_fuel_api_accessor(self, os):
fuelclient = mock.MagicMock()
patch = {
"fuelclient": fuelclient,
"fuelclient.objects": fuelclient.objects
}
with mock.patch.dict("sys.modules", patch):
accessor = accessors.get_fuel_api_accessor(
"localhost:8080", "guest", "123"
)
accessor.Environment.get_all()
os.environ.__setitem__.asseert_any_call(
"SERVER_ADDRESS", "localhost"
)
os.environ.__setitem__.asseert_any_call(
"LISTEN_PORT", "8080"
)
os.environ.__setitem__.asseert_any_call(
"KEYSTONE_USER", "guest"
)
os.environ.__setitem__.asseert_any_call(
"KEYSTONE_PASS", "123"
)
fuelclient.objects.Environment.get_all.assert_called_once_with()
@mock.patch("fuel_mirror.common.accessors.os")
def test_get_fuel_api_accessor_with_default_parameters(self, os):
fuelclient = mock.MagicMock()
patch = {
"fuelclient": fuelclient,
"fuelclient.objects": fuelclient.objects
}
with mock.patch.dict("sys.modules", patch):
accessors.get_fuel_api_accessor()
os.environ.__setitem__.assert_not_called()

View File

@ -1,499 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import mock
import os.path
import subprocess
from jsonschema import ValidationError
# The cmd2 does not work with python3.5
# because it tries to get access to the property mswindows,
# that was removed in 3.5
subprocess.mswindows = False
from fuel_mirror.commands import apply
from fuel_mirror.commands import create
from fuel_mirror.common.utils import load_input_data
from fuel_mirror.tests import base
CONFIG_PATH = os.path.join(
os.path.dirname(__file__), "data", "test_config.yaml"
)
UBUNTU_PATH = os.path.join(
os.path.dirname(__file__), "data", "test_ubuntu.yaml"
)
CENTOS_PATH = os.path.join(
os.path.dirname(__file__), "data", "test_centos.yaml"
)
INVALID_DATA_PATH = os.path.join(
os.path.dirname(__file__), "data", "test_invalid_ubuntu.yaml"
)
# TODO(akostrikov) lists_merge we are using is not stable so we have to use
# different local repos in cases with merge and without it.
# We pass sorted by priority list, but in lists_merge we sort it by key.
# As we are aiming to use existing repos as primary source - it is not issue.
def local_repos(mirror_host='10.25.0.10:8080', name_postfix='', reverse=True):
mirror_lists = load_input_data(UBUNTU_PATH, mos_version=1)
sorted_repos = reduce(lambda x, y: x + y, mirror_lists['groups'].values())
sorted_repos.sort(key=lambda x: x['priority'], reverse=reverse)
for repo in sorted_repos:
repo.pop('main', None)
repo['name'] = repo['name'] + name_postfix
repo['uri'] = repo['uri'].replace('localhost', mirror_host)
return sorted_repos
def mirror_repos():
return local_repos(mirror_host='mirror.com:8080', name_postfix='-mirror')
@mock.patch.multiple(
"fuel_mirror.app",
accessors=mock.DEFAULT
)
class TestCliCommands(base.TestCase):
common_argv = [
"--config", CONFIG_PATH,
"--fuel-server=10.25.0.10",
"--fuel-user=test",
"--fuel-password=test1"
]
def start_cmd(self, cmd, argv, data_file):
cmd.debug(
argv + self.common_argv + ["--input-file", data_file]
)
def _setup_fuel_versions(self, fuel_mock):
fuel_mock.FuelVersion.get_all_data.return_value = {
"release": "1",
"openstack_version": "2"
}
def _create_fuel_release(self, fuel_mock, osname, repos=None):
if repos is None:
repos = []
release = mock.MagicMock(data={
"name": "test release",
"operating_system": osname,
"attributes_metadata": {
"editable": {"repo_setup": {"repos": {"value": repos}}}
}
})
fuel_mock.Release.get_by_ids.return_value = [release]
fuel_mock.Release.get_all.return_value = [release]
return release
def _create_fuel_env(self, fuel_mock, repos=None):
if repos is None:
repos = []
env = mock.MagicMock(data={
"name": "test",
"release_id": 1
})
env.get_settings_data.return_value = {
"editable": {"repo_setup": {"repos": {"value": repos}}}
}
fuel_mock.Environment.get_by_ids.return_value = [env]
fuel_mock.Environment.get_all.return_value = [env]
return env
def test_create_mos_ubuntu(self, accessors):
self._setup_fuel_versions(accessors.get_fuel_api_accessor())
packetary = accessors.get_packetary_accessor()
self.start_cmd(create, ["--group", "mos"], UBUNTU_PATH)
accessors.get_packetary_accessor.assert_called_with(
threads_num=1,
ignore_errors_num=2,
retries_num=3,
http_proxy="http://localhost",
https_proxy="https://localhost",
)
packetary.assert_called_with("deb", "x86_64")
api = packetary()
api.clone_repositories.assert_called_once_with(
['http://localhost/mos mos1 main restricted'],
'/var/www/',
None, None
)
def test_create_partial_ubuntu(self, accessors):
self._setup_fuel_versions(accessors.get_fuel_api_accessor())
packetary = accessors.get_packetary_accessor()
self.start_cmd(create, ["--group", "ubuntu"], UBUNTU_PATH)
accessors.get_packetary_accessor.assert_called_with(
threads_num=1,
ignore_errors_num=2,
retries_num=3,
http_proxy="http://localhost",
https_proxy="https://localhost",
)
packetary.assert_called_with("deb", "x86_64")
api = packetary()
api.clone_repositories.assert_called_once_with(
['http://localhost/ubuntu trusty '
'main multiverse restricted universe'],
'/var/www/',
['http://localhost/mos mos1 main restricted'],
['package_deb']
)
def test_create_mos_centos(self, accessors):
self._setup_fuel_versions(accessors.get_fuel_api_accessor())
packetary = accessors.get_packetary_accessor()
self.start_cmd(create, ["--group", "mos"], CENTOS_PATH)
accessors.get_packetary_accessor.assert_called_with(
threads_num=1,
ignore_errors_num=2,
retries_num=3,
http_proxy="http://localhost",
https_proxy="https://localhost",
)
packetary.assert_called_with("rpm", "x86_64")
api = packetary()
api.clone_repositories.assert_called_once_with(
['http://localhost/mos1/x86_64'],
'/var/www/',
None, None
)
def test_create_partial_centos(self, accessors):
self._setup_fuel_versions(accessors.get_fuel_api_accessor())
packetary = accessors.get_packetary_accessor()
self.start_cmd(create, ["--group", "centos"], CENTOS_PATH)
accessors.get_packetary_accessor.assert_called_with(
threads_num=1,
ignore_errors_num=2,
retries_num=3,
http_proxy="http://localhost",
https_proxy="https://localhost",
)
packetary.assert_called_with("rpm", "x86_64")
api = packetary()
api.clone_repositories.assert_called_once_with(
['http://localhost/centos/os/x86_64'],
'/var/www/',
['http://localhost/mos1/x86_64'],
["package_rpm"]
)
def test_apply_for_ubuntu_based_env(self, accessors):
fuel = accessors.get_fuel_api_accessor()
self._setup_fuel_versions(fuel)
env = self._create_fuel_env(fuel)
self._create_fuel_release(fuel, "Ubuntu")
self.start_cmd(
apply, ['--group', 'mos', 'ubuntu', '--env', '1'],
UBUNTU_PATH
)
accessors.get_fuel_api_accessor.assert_called_with(
"10.25.0.10", "test", "test1"
)
fuel.FuelVersion.get_all_data.assert_called_once_with()
env.set_settings_data.assert_called_with(
{
'editable': {
'repo_setup': {
'repos': {'value': local_repos()}
}
}
}
)
def test_with_existing_mirrors(self, accessors):
fuel = accessors.get_fuel_api_accessor()
self._setup_fuel_versions(fuel)
env = self._create_fuel_env(fuel, repos=mirror_repos())
self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos())
self.start_cmd(
apply, ['--group', 'mos', 'ubuntu', '--env', '1'],
UBUNTU_PATH
)
accessors.get_fuel_api_accessor.assert_called_with(
"10.25.0.10", "test", "test1"
)
fuel.FuelVersion.get_all_data.assert_called_once_with()
env.set_settings_data.assert_called_with(
{
'editable': {
'repo_setup': {
'repos': {'value': mirror_repos() + local_repos()}
}
}
}
)
def test_replace_existing_mirrors_with_local(self, accessors):
fuel = accessors.get_fuel_api_accessor()
self._setup_fuel_versions(fuel)
env = self._create_fuel_env(fuel, repos=mirror_repos())
self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos())
self.start_cmd(
apply, ['--group', 'mos', 'ubuntu', '--env', '1', '--replace'],
UBUNTU_PATH
)
accessors.get_fuel_api_accessor.assert_called_with(
"10.25.0.10", "test", "test1"
)
fuel.FuelVersion.get_all_data.assert_called_once_with()
env.set_settings_data.assert_called_with(
{
'editable': {
'repo_setup': {
'repos': {'value': local_repos(reverse=False)}
}
}
}
)
def test_apply_for_centos_based_env(self, accessors):
fuel = accessors.get_fuel_api_accessor()
self._setup_fuel_versions(fuel)
env = self._create_fuel_env(fuel)
self._create_fuel_release(fuel, "CentOS")
self.start_cmd(
apply, ['--group', 'mos', 'centos', '--env', '1'],
CENTOS_PATH
)
accessors.get_fuel_api_accessor.assert_called_with(
"10.25.0.10", "test", "test1"
)
fuel.FuelVersion.get_all_data.assert_called_once_with()
env.set_settings_data.assert_called_with(
{'editable': {'repo_setup': {'repos': {'value': [
{
'priority': 5,
'name': 'centos',
'type': 'rpm',
'uri': 'http://10.25.0.10:8080/centos/os/x86_64'
},
{
'priority': 10,
'name': 'mos',
'type': 'rpm',
'uri': 'http://10.25.0.10:8080/mos1/x86_64'
}]
}}}}
)
def test_apply_for_ubuntu_release(self, accessors):
fuel = accessors.get_fuel_api_accessor()
self._setup_fuel_versions(fuel)
env = self._create_fuel_env(fuel)
release = self._create_fuel_release(fuel, "Ubuntu")
self.start_cmd(
apply, ['--group', 'mos', 'ubuntu', '--default'],
UBUNTU_PATH
)
accessors.get_fuel_api_accessor.assert_called_with(
"10.25.0.10", "test", "test1"
)
fuel.FuelVersion.get_all_data.assert_called_once_with()
self.assertEqual(1, env.set_settings_data.call_count)
release.connection.put_request.assert_called_once_with(
release.instance_api_path.format(),
{
'name': "test release",
'operating_system': 'Ubuntu',
'attributes_metadata': {
'editable': {'repo_setup': {'repos': {'value': [
{
'name': 'mos',
'priority': 1000,
'suite': 'mos1',
'section': 'main restricted',
'type': 'deb',
'uri': 'http://10.25.0.10:8080/mos'
},
{
'name': 'ubuntu',
'priority': 500,
'suite': 'trusty',
'section': 'main multiverse restricted universe',
'type': 'deb',
'uri': 'http://10.25.0.10:8080/ubuntu'
}
]}}}
}
}
)
def test_update_release_and_cluster_repositories(self, accessors):
fuel = accessors.get_fuel_api_accessor()
self._setup_fuel_versions(fuel)
env = self._create_fuel_env(fuel, repos=mirror_repos())
release = \
self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos())
self.start_cmd(
apply, ['--group', 'mos', 'ubuntu', '--default'],
UBUNTU_PATH
)
accessors.get_fuel_api_accessor.assert_called_with(
"10.25.0.10", "test", "test1"
)
fuel.FuelVersion.get_all_data.assert_called_once_with()
expected_repos = mirror_repos() + local_repos()
env.set_settings_data.assert_called_with(
{
'editable': {
'repo_setup': {
'repos': {'value': expected_repos}
}
}
}
)
release.connection.put_request.assert_called_once_with(
release.instance_api_path.format(),
{
'name': "test release",
'operating_system': 'Ubuntu',
'attributes_metadata': {
'editable': {'repo_setup': {'repos': {
'value': expected_repos
}}}
}
}
)
def test_replace_release_and_cluster_repositories(self, accessors):
fuel = accessors.get_fuel_api_accessor()
self._setup_fuel_versions(fuel)
env = self._create_fuel_env(fuel, repos=mirror_repos())
release = \
self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos())
self.start_cmd(
apply, ['--group', 'mos', 'ubuntu', '--default', '--replace'],
UBUNTU_PATH
)
accessors.get_fuel_api_accessor.assert_called_with(
"10.25.0.10", "test", "test1"
)
fuel.FuelVersion.get_all_data.assert_called_once_with()
expected_repos = local_repos(reverse=False)
env.set_settings_data.assert_called_with(
{
'editable': {
'repo_setup': {
'repos': {'value': expected_repos}
}
}
}
)
release.connection.put_request.assert_called_once_with(
release.instance_api_path.format(),
{
'name': "test release",
'operating_system': 'Ubuntu',
'attributes_metadata': {
'editable': {'repo_setup': {'repos': {
'value': expected_repos
}}}
}
}
)
def test_apply_for_centos_release(self, accessors):
fuel = accessors.get_fuel_api_accessor()
self._setup_fuel_versions(fuel)
env = self._create_fuel_env(fuel)
release = self._create_fuel_release(fuel, "CentOS")
self.start_cmd(
apply, ['--group', 'mos', 'centos', '--default'],
CENTOS_PATH
)
accessors.get_fuel_api_accessor.assert_called_with(
"10.25.0.10", "test", "test1"
)
fuel.FuelVersion.get_all_data.assert_called_once_with()
self.assertEqual(1, env.set_settings_data.call_count)
release.connection.put_request.assert_called_once_with(
release.instance_api_path.format(),
{
'name': "test release",
'operating_system': 'CentOS',
'attributes_metadata': {
'editable': {'repo_setup': {'repos': {'value': [
{
'name': 'centos',
'priority': 5,
'type': 'rpm',
'uri': 'http://10.25.0.10:8080/centos/os/x86_64'
},
{
'name': 'mos',
'priority': 10,
'type': 'rpm',
'uri': 'http://10.25.0.10:8080/mos1/x86_64'
},
]}}}
}
}
)
@mock.patch("fuel_mirror.app.utils.get_fuel_settings")
def test_apply_fail_if_no_fuel_address(self, m_get_settings, accessors):
m_get_settings.return_value = {}
with self.assertRaisesRegexp(
ValueError, "Please specify the fuel-server option"):
apply.debug(
["--config", CONFIG_PATH, "-G", "mos", "-I", UBUNTU_PATH]
)
self.assertFalse(accessors.get_fuel_api_accessor.called)
@mock.patch("fuel_mirror.app.utils.get_fuel_settings")
def test_create_without_fuel_address(self, m_get_settings, accessors):
m_get_settings.return_value = {}
packetary = accessors.get_packetary_accessor()
create.debug(
["--config", CONFIG_PATH, "-G", "mos", "-I", UBUNTU_PATH]
)
self.assertFalse(accessors.get_fuel_api_accessor.called)
accessors.get_packetary_accessor.assert_called_with(
threads_num=1,
ignore_errors_num=2,
retries_num=3,
http_proxy="http://localhost",
https_proxy="https://localhost",
)
packetary.assert_called_with("deb", "x86_64")
api = packetary()
api.clone_repositories.assert_called_once_with(
['http://localhost/mos mos main restricted'],
'/var/www/',
None,
None
)
@mock.patch("fuel_mirror.app.utils.get_fuel_settings")
def test_create_with_invalid_data(self, m_get_settings, accessors):
self.assertRaises(
ValidationError, create.debug, ["--config", CONFIG_PATH, "-G",
"mos", "-I", INVALID_DATA_PATH]
)

View File

@ -1,70 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from fuel_mirror.common import url_builder
from fuel_mirror.tests import base
class TestUrlBuilder(base.TestCase):
def test_get_url_builder(self):
self.assertTrue(issubclass(
url_builder.get_url_builder("deb"),
url_builder.AptRepoUrlBuilder
))
self.assertTrue(issubclass(
url_builder.get_url_builder("rpm"),
url_builder.YumRepoUrlBuilder
))
with self.assertRaises(KeyError):
url_builder.get_url_builder("unknown")
class TestAptUrlBuilder(base.TestCase):
@classmethod
def setUpClass(cls):
cls.builder = url_builder.get_url_builder("deb")
cls.repo_data = {
"name": "ubuntu",
"suite": "trusty",
"section": "main restricted",
"type": "deb",
"uri": "http://localhost/ubuntu"
}
def test_get_repo_url(self):
self.assertEqual(
"http://localhost/ubuntu trusty main restricted",
self.builder.get_repo_url(self.repo_data)
)
class TestYumUrlBuilder(base.TestCase):
@classmethod
def setUpClass(cls):
cls.builder = url_builder.get_url_builder("rpm")
cls.repo_data = {
"name": "centos",
"type": "rpm",
"uri": "http://localhost/os/x86_64"
}
def test_get_repo_url(self):
self.assertEqual(
"http://localhost/os/x86_64",
self.builder.get_repo_url(self.repo_data)
)

View File

@ -1,114 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import mock
import six
from fuel_mirror.common import utils
from fuel_mirror.tests import base
class DictAsObj(object):
def __init__(self, d):
self.__dict__.update(d)
def __eq__(self, other):
return self.__dict__ == other.__dict__
class TestUtils(base.TestCase):
def test_lists_merge(self):
main = [{"a": 1, "b": 2, "c": 0}, {"a": 2, "b": 3, "c": 1}]
patch = [{"a": 2, "b": 4}, {"a": 3, "b": 5}]
utils.lists_merge(
main,
patch,
key="a"
)
self.assertItemsEqual(
[{"a": 1, "b": 2, "c": 0},
{"a": 2, "b": 4, "c": 1},
{"a": 3, "b": 5}],
main
)
def test_first(self):
self.assertEqual(
1,
utils.first(0, 1, 0),
)
self.assertEqual(
1,
utils.first(None, [], '', 1),
)
self.assertIsNone(
utils.first(None, [], 0, ''),
)
self.assertIsNone(
utils.first(),
)
def test_is_subdict(self):
self.assertFalse(utils.is_subdict({"c": 1}, {"a": 1, "b": 1}))
self.assertFalse(utils.is_subdict({"a": 1, "b": 2}, {"a": 1, "b": 1}))
self.assertFalse(
utils.is_subdict({"a": 1, "b": 1, "c": 2}, {"a": 1, "b": 1})
)
self.assertFalse(
utils.is_subdict({"a": 1, "b": None}, {"a": 1})
)
self.assertTrue(utils.is_subdict({}, {"a": 1}))
self.assertTrue(utils.is_subdict({"a": 1}, {"a": 1, "b": 1}))
self.assertTrue(utils.is_subdict({"a": 1, "b": 1}, {"a": 1, "b": 1}))
@mock.patch("fuel_mirror.common.utils.open")
def test_get_fuel_settings(self, m_open):
m_open().__enter__.side_effect = [
six.StringIO(
'ADMIN_NETWORK:\n'
' ipaddress: "10.20.0.4"\n'
'FUEL_ACCESS:\n'
' user: "test"\n'
' password: "test_pwd"\n',
),
OSError
]
self.assertEqual(
{
"server": "10.20.0.4",
"user": "test",
"password": "test_pwd",
},
utils.get_fuel_settings()
)
self.assertEqual(
{},
utils.get_fuel_settings()
)
@mock.patch("fuel_mirror.common.utils.yaml")
@mock.patch("fuel_mirror.common.utils.open")
def test_load_input_data(self, open_mock, yaml_mock):
data = "$param1: $param2"
open_mock().__enter__().read.return_value = data
v = utils.load_input_data("data.yaml", param1="key", param2="value")
open_mock.assert_called_with("data.yaml", "r")
yaml_mock.load.assert_called_once_with("key: value")
self.assertIs(yaml_mock.load(), v)

View File

@ -1,158 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os.path
from jsonschema import validate
from jsonschema import ValidationError
import yaml
from fuel_mirror.schemas.input_data_schema import SCHEMA
from fuel_mirror.tests import base
DATA_DIR = os.path.join(os.path.dirname(__file__), "..", "..", "data")
class TestValidateConfigs(base.TestCase):
def test_validate_data_files(self):
for f in os.listdir(DATA_DIR):
with open(os.path.join(DATA_DIR, f), "r") as fd:
data = yaml.load(fd)
self.assertNotRaises(ValidationError, validate, data, SCHEMA)
self.assertIn("groups", data)
self.assertIn("fuel_release_match", data)
def test_validate_fail_with_empty_data(self):
self.assertRaises(ValidationError, validate, {}, SCHEMA)
def test_validate_fail_without_groups(self):
invalid_data = {
"requirements": {
"ubuntu": ["package_deb"]
}
}
self.assertRaisesRegexp(
ValidationError, "'groups' is a required property", validate,
invalid_data, SCHEMA)
def test_invalid_requirements_in_pattern_properies(self):
invalid_data = {
"requirements": {
"ubun.tu": ["package_deb"]
},
"groups": {
}
}
self.assertRaisesRegexp(
ValidationError, "'ubun.tu' was unexpected", validate,
invalid_data, SCHEMA)
def test_invalid_requirements_type_array(self):
invalid_data = {
"requirements": {
"ubuntu": "package_deb"
},
"groups": {
}
}
self.assertRaisesRegexp(
ValidationError, "'package_deb' is not of type 'array'", validate,
invalid_data, SCHEMA)
def test_invalid_inheritens_in_pattern_properies(self):
invalid_data = {
"inheritance": {
"ubun.tu": "mos"
},
"groups": {
}
}
self.assertRaisesRegexp(
ValidationError, "'ubun.tu' was unexpected", validate,
invalid_data, SCHEMA)
def test_invalid_inheritens_type_string(self):
invalid_data = {
"inheritance": {
"ubuntu": 123
},
"groups": {
}
}
self.assertRaisesRegexp(
ValidationError, "123 is not of type 'string'", validate,
invalid_data, SCHEMA)
def test_invalid_groups_in_pattern_properies(self):
invalid_data = {
"groups": {
"mo.s": []
}
}
self.assertRaisesRegexp(
ValidationError, "'mo.s' was unexpected", validate,
invalid_data, SCHEMA)
def test_invalid_groups_type_array(self):
invalid_data = {
"groups": {
"mos": "string"
}
}
self.assertRaisesRegexp(
ValidationError, "'string' is not of type 'array'", validate,
invalid_data, SCHEMA)
def test_without_name_in_groups_array(self):
invalid_data = {
"groups": {
"mos": [
{
'type': 'deb',
'uri': 'http://localhost/mos',
'priority': None,
'suite': 'mos$mos_version',
'section': 'main restricted'
}
]
}
}
self.assertRaisesRegexp(
ValidationError, "is not valid under any of the given schemas",
validate, invalid_data, SCHEMA)
def test_with_invalid_type_in_groups_array(self):
invalid_data = {
"groups": {
"mos": [
{
'name': 'mos',
'type': 'adf',
'uri': 'http://localhost/mos',
'priority': None,
'suite': 'mos$mos_version',
'section': 'main restricted'
}
]
}
}
self.assertRaisesRegexp(
ValidationError, "is not valid under any of the given schemas",
validate, invalid_data, SCHEMA)

View File

@ -1,12 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
pbr>=0.8
Babel>=1.3
cliff>=1.7.0
six>=1.5.2
PyYAML>=3.10
packetary>=0.1.0
python-fuelclient>=7.0.0
jsonschema>=2.3.0

View File

@ -1,80 +0,0 @@
#!/bin/bash
echo "This script is DEPRECATED. Please use fuel-mirror utility!"
# This shell script was wraps the fuel-mirror utility to provide backward compatibility
# with previous version of tool.
usage() {
cat <<EOF
Usage: `basename $0` [options]
Create and update local mirrors of MOS and/or Ubuntu.
IMPORTANT!
If NO parameters specified, this script will:
- Create/Update both MOS and Ubuntu local mirrors
- Set them as repositories for existing NEW environments in Fuel UI
- Set them as DEFAULT repositories for new environments
Options:
-h| --help This help screen.
-d| --no-default Don't change default repositories for new environments
-a| --no-apply Don't apply changes to Fuel environments
-M| --mos Create/Update MOS local mirror only
-U| --ubuntu Create/Update Ubuntu local mirror only
-p| --password Fuel Master admin password (defaults to admin)
-v| --verbose Add detailed output
EOF
}
# Parse options
OPTS=`getopt -o hdaMUNp: -l help,no-default,no-apply,mos,ubuntu,password:,dry-run,verbose -- "$@"`
if [ $? != 0 ]; then
usage
exit 1
fi
eval set -- "$OPTS"
CMD_OPTS="--pattern=ubuntu"
REPO_GROUPS=""
while true ; do
case "$1" in
-h| --help ) usage ; exit 0;;
-d | --no-default ) OPT_NO_DEFAULT=1; shift;;
-a | --no-apply ) OPT_NO_APPLY=1; shift;;
-N | --dry-run ) EXEC_PREFIX="echo EXEC "; shift;;
-M | --mos ) REPO_GROUPS="$REPO_GROUPS mos"; shift;;
-U | --ubuntu ) REPO_GROUPS="$REPO_GROUPS ubuntu"; shift;;
-p | --password ) CMD_OPTS="$CMD_OPTS --fuel-password=$2"; shift; shift;;
-v | --verbose ) CMD_OPTS="$CMD_OPTS --verbose"; shift;;
-- ) shift; break;;
* ) break;;
esac
done
if [[ "$@" != "" ]]; then
echo "Invalid option -- $@"
usage
exit 1
fi
if [[ "$REPO_GROUPS" == "" ]]; then
REPO_GROUPS="mos ubuntu"
fi
CMD_OPTS="$CMD_OPTS --group $REPO_GROUPS"
$EXEC_PREFIX fuel-mirror create ${CMD_OPTS}
if [[ "$OPT_NO_DEFAULT" == "" ]]; then
CMD_OPTS="$CMD_OPTS --default"
fi
if [[ "$OPT_NO_APPLY" == "" ]]; then
$EXEC_PREFIX fuel-mirror apply ${CMD_OPTS}
fi

View File

@ -1,67 +0,0 @@
[metadata]
name = fuel_mirror
version = 10.0.0
summary = The Utility to create local repositories with packages is
required for openstack deployment.
description-file =
README.rst
author = Mirantis Inc.
author_email = product@mirantis.com
url = http://mirantis.com
home-page = http://mirantis.com
classifier =
Development Status :: 4 - Beta
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
License :: OSI Approved :: GNU General Public License v2 (GPLv2)
Operating System :: POSIX :: Linux
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.3
Programming Language :: Python :: 3.4
Topic :: Utilities
[files]
packages =
fuel_mirror
data_files =
etc/fuel-mirror = etc/*
share/fuel-mirror = data/*
[build_sphinx]
source-dir = doc/source
build-dir = doc/build
all_files = 1
[entry_points]
console_scripts =
fuel-mirror=fuel_mirror.app:main
fuel_mirror =
apply=fuel_mirror.commands.apply:ApplyCommand
create=fuel_mirror.commands.create:CreateCommand
[upload_sphinx]
upload-dir = doc/build/html
[compile_catalog]
directory = locale
domain = fuel_mirror
[update_catalog]
domain = fuel_mirror
output_dir = locale
input_file = locale/fuel_mirror.pot
[extract_messages]
keywords = _ gettext ngettext l_ lazy_gettext
mapping_file = babel.cfg
output_file = locale/fuel_mirror.pot
[global]
setup-hooks =
pbr.hooks.setup_hook
setup_hooks.setup_hook

View File

@ -1,32 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr'],
pbr=True)

View File

@ -1,25 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
def setup_hook(config):
import pbr
import pbr.packaging
# this monkey patch is to avoid appending git version to version
pbr.packaging._get_version_from_git = lambda pre_version: pre_version

View File

@ -1,17 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
hacking<0.11,>=0.10.0
coverage>=3.6
discover
python-subunit>=0.0.18
sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2
oslosphinx>=2.5.0 # Apache-2.0
oslotest>=1.10.0 # Apache-2.0
testrepository>=0.0.18
testscenarios>=0.4
testtools>=1.4.0
cliff>=1.7.0
six>=1.5.2

View File

@ -1,35 +0,0 @@
[tox]
minversion = 1.6
envlist = py34,py27,py26,pep8
skipsdist = True
[testenv]
usedevelop = True
install_command = pip install -U {opts} {packages}
setenv =
VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/test-requirements.txt
commands = python setup.py test --slowest --testr-args='{posargs:fuel_mirror}'
[testenv:pep8]
commands = flake8
[testenv:venv]
commands = {posargs}
[testenv:cover]
commands = python setup.py test --coverage --testr-args='{posargs:fuel_mirror}'
[testenv:docs]
commands = python setup.py build_sphinx
[testenv:debug]
commands = oslo_debug_helper {posargs}
[flake8]
# E123, E125 skipped as they are invalid PEP-8.
show-source = True
ignore = E123,E125
builtins = _
exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build

17
debian/changelog vendored
View File

@ -1,17 +0,0 @@
fuel-mirror (10.0.0-1) experimental; urgency=medium
* Bump version to 10.0
-- Sergey Kulanov <skulanov@mirantis.com> Mon, 21 Mar 2016 13:29:18 +0200
fuel-mirror (9.0.0-1) experimental; urgency=low
* Bump version to 9.0
-- Sergey Kulanov <skulanov@mirantis.com> Thu, 17 Dec 2015 16:39:55 +0200
fuel-mirror (8.0.0-1) experimental; urgency=low
* Initial release.
-- bgaifullin <bgaifullin@mirantis.com> Fri, 27 Nov 2015 00:28:26 +0300

1
debian/compat vendored
View File

@ -1 +0,0 @@
9

54
debian/control vendored
View File

@ -1,54 +0,0 @@
Source: fuel-mirror
Section: Utilities
Priority: extra
Maintainer: Mirantis Product <product@mirantis.com>
Build-Depends: debhelper (>= 9),
dh-python,
openstack-pkg-tools (>= 23~),
python-all,
python-pbr (>= 0.8),
python-setuptools
Standards-Version: 3.9.6
Homepage: mirantis.com
Package: fuel-mirror
Architecture: all
Section: python
Depends: python-babel,
python-cliff (>= 1.7.0),
python-packetary (= ${binary:Version}),
python-pbr (>= 0.8),
python-six,
python-yaml,
python-tz,
${python:Depends}
Recommends: python-fuelclient (>= 7.0.0)
Description: Utility to create RPM and DEB mirror
Provides two commands fuel-mirror and fuel-createmirror.
Second one is for backward compatibility with the previous
generation of the utility. These commands could be used
to create local copies of MOS and upstream deb and rpm
repositories.
Package: python-packetary
Architecture: all
Depends: createrepo,
python-babel,
python-bintrees (>= 2.0.2),
python-chardet,
python-cliff (>= 1.7.0),
python-debian (>= 0.1.21),
python-eventlet (>= 0.15),
python-lxml,
python-pbr (>= 0.8),
python-six,
python-stevedore (>= 1.1.0),
python-tz,
${python:Depends}
Description: Library allows to build and clone deb and rpm repos
Provides object model and API for dealing with deb
and rpm repositories. One can use this framework to
implement operations like building repository
from a set of packages, clone repository, find package
dependencies, mix repositories, pull out a subset of
packages into a separate repository, etc.

43
debian/copyright vendored
View File

@ -1,43 +0,0 @@
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: fuel-mirror
Source: git://github.com/openstack/fuel-mirror.git
Files: debian/*
Copyright: (c) 2014, Mirantis
License: GPL-2
Files: *
Copyright: (c) 2014, Mirantis
License: Apache-2
License: Apache-2
Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at:
.
http://www.apache.org/licenses/LICENSE-2.0
.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
.
On Debian-based systems the full text of the Apache version 2.0 license can be
found in /usr/share/common-licenses/Apache-2.0.
License: GPL-2
Licensed under the GPL License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at:
.
http://www.opensource.org/licenses/GPL-2.0
.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
.
On Debian-based systems the full text of the GPL version 2.0 license can be
found in /usr/share/common-licenses/GPL-2.

View File

@ -1,2 +0,0 @@
contrib/fuel_mirror/etc/config.yaml /etc/fuel-mirror
contrib/fuel_mirror/scripts/fuel-createmirror /usr/bin/

39
debian/rules vendored
View File

@ -1,39 +0,0 @@
#!/usr/bin/make -f
PYTHONS:=$(shell pyversions -vr)
include /usr/share/openstack-pkg-tools/pkgos.make
export OSLO_PACKAGE_VERSION=$(shell dpkg-parsechangelog | grep Version: | cut -d' ' -f2 | sed -e 's/^[[:digit:]]*://' -e 's/[-].*//' -e 's/~/.0/' | head -n 1)
%:
dh $@ --buildsystem=python_distutils --with python2
override_dh_clean:
rm -rf build
dh_clean -O--buildsystem=python_distutils
override_dh_auto_install:
set -e ; for pyvers in $(PYTHONS); do \
python$$pyvers setup.py install --install-layout=deb \
--root $(CURDIR)/debian/python-packetary; \
done
set -e ; cd contrib/fuel_mirror/; \
for pyvers in $(PYTHONS); do \
python$$pyvers ./setup.py install --install-layout=deb \
--root $(CURDIR)/debian/fuel-mirror; \
done
override_dh_fixperms:
set -e; chmod 755 $(CURDIR)/debian/fuel-mirror/usr/bin/fuel-createmirror
override_dh_python2:
dh_python2 --no-guessing-deps
override_dh_installcatalogs:
override_dh_installemacsen override_dh_installifupdown:
override_dh_installinfo override_dh_installmenu override_dh_installmime:
override_dh_installmodules override_dh_installlogcheck:
override_dh_installpam override_dh_installppp override_dh_installudev override_dh_installwm:
override_dh_installxfonts override_dh_gconf override_dh_icons override_dh_perl override_dh_usrlocal:
override_dh_installgsettings:

View File

@ -1 +0,0 @@
3.0 (quilt)

View File

View File

View File

@ -1,79 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
sys.path.insert(0, os.path.abspath('../..'))
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
#'sphinx.ext.intersphinx',
'oslosphinx'
]
# autodoc generation is a bit aggressive and a nuisance when doing heavy
# text edit cycles.
# execute "export SPHINX_DEBUG=1" in your terminal to disable
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'packetary'
copyright = u'2015, Mirantis Inc.'
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = True
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output --------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
# html_theme = '_theme'
# html_static_path = ['static']
# Output file base name for HTML help builder.
htmlhelp_basename = '%sdoc' % project
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto/manual]).
latex_documents = [
('index',
'%s.tex' % project,
u'%s Documentation' % project,
u'OpenStack Foundation', 'manual'),
]
# Example configuration for intersphinx: refer to the Python standard library.
#intersphinx_mapping = {'http://docs.python.org/': None}

View File

@ -1,4 +0,0 @@
============
Contributing
============
.. include:: ../../CONTRIBUTING.rst

View File

@ -1,25 +0,0 @@
.. packetary documentation master file, created by
sphinx-quickstart on Tue Jul 9 22:26:36 2013.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to packetary's documentation!
========================================================
Contents:
.. toctree::
:maxdepth: 2
readme
installation
usage
contributing
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

View File

@ -1,12 +0,0 @@
============
Installation
============
At the command line::
$ pip install packetary
Or, if you have virtualenvwrapper installed::
$ mkvirtualenv packetary
$ pip install packetary

View File

@ -1 +0,0 @@
.. include:: ../../README.rst

View File

@ -1,7 +0,0 @@
========
Usage
========
To use packetary in a project::
import packetary

View File

@ -1,6 +0,0 @@
[DEFAULT]
# The list of modules to copy from oslo-incubator.git
# The base module to hold the copy of openstack.common
base=packetary

View File

@ -1,33 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import pbr.version
from packetary.api import Configuration
from packetary.api import Context
from packetary.api import RepositoryApi
__all__ = [
"Configuration",
"Context",
"RepositoryApi",
]
__version__ = pbr.version.VersionInfo(
'packetary').version_string()

View File

@ -1,233 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import logging
import six
from packetary.controllers import RepositoryController
from packetary.library.connections import ConnectionsManager
from packetary.library.executor import AsynchronousSection
from packetary.objects import Index
from packetary.objects import PackageRelation
from packetary.objects import PackagesTree
from packetary.objects.statistics import CopyStatistics
logger = logging.getLogger(__package__)
class Configuration(object):
"""The configuration holder."""
def __init__(self, http_proxy=None, https_proxy=None,
retries_num=0, threads_num=0,
ignore_errors_num=0, retry_interval=0):
"""Initialises.
:param http_proxy: the url of proxy for connections over http,
no-proxy will be used if it is not specified
:param https_proxy: the url of proxy for connections over https,
no-proxy will be used if it is not specified
:param retries_num: the number of retries on errors
:param retry_interval: the time between retries (in seconds)
:param threads_num: the max number of active threads
:param ignore_errors_num: the number of errors that may occurs
before stop processing
"""
self.http_proxy = http_proxy
self.https_proxy = https_proxy
self.ignore_errors_num = ignore_errors_num
self.retries_num = retries_num
self.retry_interval = retry_interval
self.threads_num = threads_num
class Context(object):
"""The infra-objects holder."""
def __init__(self, config):
"""Initialises.
:param config: the configuration
"""
self._connection = ConnectionsManager(
proxy=config.http_proxy,
secure_proxy=config.https_proxy,
retries_num=config.retries_num,
retry_interval=config.retry_interval
)
self._threads_num = config.threads_num
self._ignore_errors_num = config.ignore_errors_num
@property
def connection(self):
"""Gets the connection."""
return self._connection
def async_section(self, ignore_errors_num=None):
"""Gets the execution scope.
:param ignore_errors_num: custom value for ignore_errors_num,
the class value is used if omitted.
"""
if ignore_errors_num is None:
ignore_errors_num = self._ignore_errors_num
return AsynchronousSection(self._threads_num, ignore_errors_num)
class RepositoryApi(object):
"""Provides high-level API to operate with repositories."""
def __init__(self, controller):
"""Initialises.
:param controller: the repository controller.
"""
self.controller = controller
@classmethod
def create(cls, config, repotype, repoarch):
"""Creates the repository API instance.
:param config: the configuration
:param repotype: the kind of repository(deb, yum, etc)
:param repoarch: the architecture of repository (x86_64 or i386)
"""
context = config if isinstance(config, Context) else Context(config)
return cls(RepositoryController.load(context, repotype, repoarch))
def get_packages(self, origin, debs=None, requirements=None):
"""Gets the list of packages from repository(es).
:param origin: The list of repository`s URLs
:param debs: the list of repository`s URL to calculate list of
dependencies, that will be used to filter packages.
:param requirements: the list of package relations,
to resolve the list of mandatory packages.
:return: the set of packages
"""
repositories = self._get_repositories(origin)
return self._get_packages(repositories, debs, requirements)
def clone_repositories(self, origin, destination, debs=None,
requirements=None, keep_existing=True,
include_source=False, include_locale=False):
"""Creates the clones of specified repositories in local folder.
:param origin: The list of repository`s URLs
:param destination: the destination folder path
:param debs: the list of repository`s URL to calculate list of
dependencies, that will be used to filter packages.
:param requirements: the list of package relations,
to resolve the list of mandatory packages.
:param keep_existing: If False - local packages that does not exist
in original repo will be removed.
:param include_source: if True, the source packages
will be copied as well.
:param include_locale: if True, the locales
will be copied as well.
:return: count of copied and total packages.
"""
repositories = self._get_repositories(origin)
packages = self._get_packages(repositories, debs, requirements)
mirrors = self.controller.clone_repositories(
repositories, destination, include_source, include_locale
)
package_groups = dict((x, set()) for x in repositories)
for pkg in packages:
package_groups[pkg.repository].add(pkg)
stat = CopyStatistics()
for repo, packages in six.iteritems(package_groups):
mirror = mirrors[repo]
logger.info("copy packages from - %s", repo)
self.controller.copy_packages(
mirror, packages, keep_existing, stat.on_package_copied
)
return stat
def get_unresolved_dependencies(self, origin, main=None):
"""Gets list of unresolved dependencies for repository(es).
:param origin: The list of repository`s URLs
:param main: The main repository(es) URL
:return: list of unresolved dependencies
"""
packages = PackagesTree()
self.controller.load_packages(
self._get_repositories(origin),
packages.add
)
if main is not None:
base = Index()
self.controller.load_packages(
self._get_repositories(main),
base.add
)
else:
base = None
return packages.get_unresolved_dependencies(base)
def _get_repositories(self, urls):
"""Gets the set of repositories by url."""
repositories = set()
self.controller.load_repositories(urls, repositories.add)
return repositories
def _get_packages(self, repositories, master, requirements):
"""Gets the list of packages according to master and requirements."""
if master is None and requirements is None:
packages = set()
self.controller.load_packages(repositories, packages.add)
return packages
packages = PackagesTree()
self.controller.load_packages(repositories, packages.add)
if master is not None:
main_index = Index()
self.controller.load_packages(
self._get_repositories(master),
main_index.add
)
else:
main_index = None
return packages.get_minimal_subset(
main_index,
self._parse_requirements(requirements)
)
@staticmethod
def _parse_requirements(requirements):
"""Gets the list of relations from requirements.
:param requirements: the list of requirement in next format:
'name [cmp version]|[alt [cmp version]]'
"""
if requirements is not None:
return set(
PackageRelation.from_args(
*(x.split() for x in r.split("|"))) for r in requirements
)
return set()

View File

@ -1,104 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from cliff import app
from cliff.commandmanager import CommandManager
import packetary
class Application(app.App):
"""Main cliff application class.
Performs initialization of the command manager and
configuration of basic engines.
"""
def build_option_parser(self, description, version, argparse_kwargs=None):
"""Specifies global options."""
p_inst = super(Application, self)
parser = p_inst.build_option_parser(description=description,
version=version,
argparse_kwargs=argparse_kwargs)
parser.add_argument(
"--ignore-errors-num",
type=int,
default=2,
metavar="NUMBER",
help="The number of errors that can be ignored."
)
parser.add_argument(
"--retries-num",
type=int,
default=5,
metavar="NUMBER",
help="The number of retries."
)
parser.add_argument(
"--retry-interval",
type=int,
default=2,
metavar="SECONDS",
help="The minimal time between retries in seconds."
)
parser.add_argument(
"--threads-num",
default=3,
type=int,
metavar="NUMBER",
help="The number of threads."
)
parser.add_argument(
"--http-proxy",
default=None,
metavar="http://username:password@proxy_host:proxy_port",
help="The URL of http proxy."
)
parser.add_argument(
"--https-proxy",
default=None,
metavar="https://username:password@proxy_host:proxy_port",
help="The URL of https proxy."
)
return parser
def main(argv=None):
return Application(
description="The utility manages packages and repositories.",
version=packetary.__version__,
command_manager=CommandManager("packetary", convert_underscores=True)
).run(argv)
def debug(name, cmd_class, argv=None):
"""Helper for debugging single command without package installation."""
import sys
if argv is None:
argv = sys.argv[1:]
argv = [name] + argv + ["-v", "-v", "--debug"]
cmd_mgr = CommandManager("test_packetary", convert_underscores=True)
cmd_mgr.add_command(name, cmd_class)
return Application(
description="The utility manages packages and repositories.",
version="0.0.1",
command_manager=cmd_mgr
).run(argv)

View File

@ -1,185 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import abc
from cliff import command
import six
from packetary.cli.commands.utils import make_display_attr_getter
from packetary.cli.commands.utils import read_lines_from_file
from packetary import RepositoryApi
@six.add_metaclass(abc.ABCMeta)
class BaseRepoCommand(command.Command):
"""Super class for packetary commands."""
@property
def stdout(self):
"""Shortcut for self.app.stdout."""
return self.app.stdout
def get_parser(self, prog_name):
"""Specifies common options."""
parser = super(BaseRepoCommand, self).get_parser(prog_name)
parser.add_argument(
'-t',
'--type',
type=str,
choices=['deb', 'rpm'],
metavar='TYPE',
default='deb',
help='The type of repository.')
parser.add_argument(
'-a',
'--arch',
type=str,
choices=["x86_64", "i386"],
metavar='ARCHITECTURE',
default="x86_64",
help='The target architecture.')
origin_gr = parser.add_mutually_exclusive_group(required=True)
origin_gr.add_argument(
'-o', '--origin-url',
nargs="+",
dest='origins',
type=six.text_type,
metavar='URL',
help='Space separated list of URLs of origin repositories.')
origin_gr.add_argument(
'-O', '--origin-file',
type=read_lines_from_file,
dest='origins',
metavar='FILENAME',
help='The path to file with URLs of origin repositories.')
return parser
def take_action(self, parsed_args):
"""See the Command.take_action.
:param parsed_args: the command-line arguments
:return: the result of take_repo_action
:rtype: object
"""
return self.take_repo_action(
RepositoryApi.create(
self.app_args, parsed_args.type, parsed_args.arch
),
parsed_args
)
@abc.abstractmethod
def take_repo_action(self, api, parsed_args):
"""Takes action on repository.
:param api: the RepositoryApi instance
:param parsed_args: the command-line arguments
:return: the action result
"""
class BaseProduceOutputCommand(BaseRepoCommand):
columns = None
def get_parser(self, prog_name):
parser = super(BaseProduceOutputCommand, self).get_parser(prog_name)
group = parser.add_argument_group(
title='output formatter',
description='output formatter options',
)
group.add_argument(
'-c', '--column',
nargs='+',
choices=self.columns,
dest='columns',
metavar='COLUMN',
default=[],
help='Space separated list of columns to include.',
)
group.add_argument(
'-s',
'--sort-columns',
type=str,
nargs='+',
choices=self.columns,
metavar='SORT_COLUMN',
default=[self.columns[0]],
help='Space separated list of keys for sorting '
'the data.'
)
group.add_argument(
'--sep',
type=six.text_type,
metavar='ROW SEPARATOR',
default=six.text_type('; '),
help='The row separator.'
)
return parser
def produce_output(self, parsed_args, data):
indexes = dict(
(c, i) for i, c in enumerate(self.columns)
)
sort_index = [indexes[c] for c in parsed_args.sort_columns]
if isinstance(data, list):
data.sort(key=lambda x: [x[i] for i in sort_index])
else:
data = sorted(data, key=lambda x: [x[i] for i in sort_index])
if parsed_args.columns:
include_index = [
indexes[c] for c in parsed_args.columns
]
data = ((row[i] for i in include_index) for row in data)
columns = parsed_args.columns
else:
columns = self.columns
stdout = self.stdout
sep = parsed_args.sep
# header
stdout.write("# ")
stdout.write(sep.join(columns))
stdout.write("\n")
for row in data:
stdout.write(sep.join(row))
stdout.write("\n")
def run(self, parsed_args):
# Use custom output producer.
# cliff.lister with default formatters does not work
# with large arrays of data, because it does not support streaming
# TODO(implement custom formatter)
formatter = make_display_attr_getter(self.columns)
data = six.moves.map(formatter, self.take_action(parsed_args))
self.produce_output(parsed_args, data)
return 0
@abc.abstractmethod
def take_repo_action(self, driver, parsed_args):
"""See Command.take_repo_action."""

View File

@ -1,115 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from packetary.cli.commands.base import BaseRepoCommand
from packetary.cli.commands.utils import read_lines_from_file
class CloneCommand(BaseRepoCommand):
"""Clones the specified repository to local folder."""
def get_parser(self, prog_name):
parser = super(CloneCommand, self).get_parser(prog_name)
parser.add_argument(
"-d", "--destination",
required=True,
help="The path to the destination folder."
)
parser.add_argument(
"--clean",
dest="keep_existing",
action='store_false',
default=True,
help="Remove packages that does not exist in origin repo."
)
parser.add_argument(
"--sources",
action='store_true',
default=False,
help="Also copy source packages."
)
parser.add_argument(
"--locales",
action='store_true',
default=False,
help="Also copy localisation files."
)
bootstrap_group = parser.add_mutually_exclusive_group(required=False)
bootstrap_group.add_argument(
"-b", "--bootstrap",
nargs='+',
dest='bootstrap',
metavar='PACKAGE [OP VERSION]',
help="Space separated list of package relations, "
"to resolve the list of mandatory packages."
)
bootstrap_group.add_argument(
"-B", "--bootstrap-file",
type=read_lines_from_file,
dest='bootstrap',
metavar='FILENAME',
help="Path to the file with list of package relations, "
"to resolve the list of mandatory packages."
)
requires_group = parser.add_mutually_exclusive_group(required=False)
requires_group.add_argument(
'-r', '--requires-url',
nargs="+",
dest='requires',
metavar='URL',
help="Space separated list of repository`s URL to calculate list "
"of dependencies, that will be used to filter packages")
requires_group.add_argument(
'-R', '--requires-file',
type=read_lines_from_file,
dest='requires',
metavar='FILENAME',
help="The path to the file with list of repository`s URL "
"to calculate list of dependencies, "
"that will be used to filter packages")
return parser
def take_repo_action(self, api, parsed_args):
stat = api.clone_repositories(
parsed_args.origins,
parsed_args.destination,
parsed_args.requires,
parsed_args.bootstrap,
parsed_args.keep_existing,
parsed_args.sources,
parsed_args.locales
)
self.stdout.write(
"Packages copied: {0.copied}/{0.total}.\n".format(stat)
)
def debug(argv=None):
"""Helper to debug the Clone command."""
from packetary.cli.app import debug
debug("clone", CloneCommand, argv)
if __name__ == "__main__":
debug()

View File

@ -1,93 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from packetary.cli.commands.base import BaseProduceOutputCommand
from packetary.cli.commands.utils import read_lines_from_file
class ListOfPackages(BaseProduceOutputCommand):
"""Gets the list of packages from repository(es)."""
columns = (
"name",
"repository",
"version",
"filename",
"filesize",
"checksum",
"obsoletes",
"provides",
"requires",
)
def get_parser(self, prog_name):
parser = super(ListOfPackages, self).get_parser(prog_name)
bootstrap_group = parser.add_mutually_exclusive_group(required=False)
bootstrap_group.add_argument(
"-b", "--bootstrap",
nargs='+',
dest='bootstrap',
metavar='PACKAGE [OP VERSION]',
help="Space separated list of package relations, "
"to resolve the list of mandatory packages."
)
bootstrap_group.add_argument(
"-B", "--bootstrap-file",
type=read_lines_from_file,
dest='bootstrap',
metavar='FILENAME',
help="Path to the file with list of package relations, "
"to resolve the list of mandatory packages."
)
requires_group = parser.add_mutually_exclusive_group(required=False)
requires_group.add_argument(
'-r', '--requires-url',
nargs="+",
dest='requires',
metavar='URL',
help="Space separated list of repository`s URL to calculate list "
"of dependencies, that will be used to filter packages")
requires_group.add_argument(
'-R', '--requires-file',
type=read_lines_from_file,
dest='requires',
metavar='FILENAME',
help="The path to the file with list of repository`s URL "
"to calculate list of dependencies, "
"that will be used to filter packages")
return parser
def take_repo_action(self, api, parsed_args):
return api.get_packages(
parsed_args.origins,
parsed_args.requires,
parsed_args.bootstrap,
)
def debug(argv=None):
"""Helper to debug the ListOfPackages command."""
from packetary.cli.app import debug
debug("packages", ListOfPackages, argv)
if __name__ == "__main__":
debug()

View File

@ -1,68 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from packetary.cli.commands.base import BaseProduceOutputCommand
from packetary.cli.commands.utils import read_lines_from_file
class ListOfUnresolved(BaseProduceOutputCommand):
"""Gets the list of external dependencies for repository(es)."""
columns = (
"name",
"version",
"alternative",
)
def get_parser(self, prog_name):
parser = super(ListOfUnresolved, self).get_parser(prog_name)
main_group = parser.add_mutually_exclusive_group(required=False)
main_group.add_argument(
'-m', '--main-url',
nargs="+",
dest='main',
metavar='URL',
help='Space separated list of URLs of repository(es) '
' that are used to resolve dependencies.')
main_group.add_argument(
'-M', '--main-file',
type=read_lines_from_file,
dest='main',
metavar='FILENAME',
help='The path to the file, that contains '
'list of URLs of repository(es) '
' that are used to resolve dependencies.')
return parser
def take_repo_action(self, api, parsed_args):
return api.get_unresolved_dependencies(
parsed_args.origins,
parsed_args.main,
)
def debug(argv=None):
"""Helper to debug the ListOfUnresolved command."""
from packetary.cli.app import debug
debug("unresolved", ListOfUnresolved, argv)
if __name__ == "__main__":
debug()

View File

@ -1,72 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import operator
import six
def read_lines_from_file(filename):
"""Reads lines from file.
Note: the line starts with '#' will be skipped.
:param filename: the path of target file
:return: the list of lines from file
"""
with open(filename, 'r') as f:
return [
x
for x in six.moves.map(operator.methodcaller("strip"), f)
if x and not x.startswith("#")
]
def get_object_attrs(obj, attrs):
"""Gets object attributes as list.
:param obj: the target object
:param attrs: the list of attributes
:return: list of values from specified attributes.
"""
return [getattr(obj, f) for f in attrs]
def get_display_value(value):
"""Get the displayable string for value.
:param value: the target value
:return: the displayable string for value
"""
if value is None:
return u"-"
if isinstance(value, list):
return u", ".join(six.text_type(x) for x in value)
return six.text_type(value)
def make_display_attr_getter(attrs):
"""Gets formatter to convert attributes of object in displayable format.
:param attrs: the list of attributes
:return: the formatter (callable object)
"""
return lambda x: [
get_display_value(v) for v in get_object_attrs(x, attrs)
]

View File

@ -1,23 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from packetary.controllers.repository import RepositoryController
__all__ = [
"RepositoryController"
]

View File

@ -1,171 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import logging
import os
import six
import stevedore
logger = logging.getLogger(__package__)
urljoin = six.moves.urllib.parse.urljoin
class RepositoryController(object):
"""Implements low-level functionality to communicate with drivers."""
_drivers = None
def __init__(self, context, driver, arch):
self.context = context
self.driver = driver
self.arch = arch
@classmethod
def load(cls, context, driver_name, repoarch):
"""Creates the repository manager.
:param context: the context
:param driver_name: the name of required driver
:param repoarch: the architecture of repository (x86_64 or i386)
"""
if cls._drivers is None:
cls._drivers = stevedore.ExtensionManager(
"packetary.drivers", invoke_on_load=True
)
try:
driver = cls._drivers[driver_name].obj
except KeyError:
raise NotImplementedError(
"The driver {0} is not supported yet.".format(driver_name)
)
return cls(context, driver, repoarch)
def load_repositories(self, urls, consumer):
"""Loads the repository objects from url.
:param urls: the list of repository urls.
:param consumer: the callback to consume objects
"""
if isinstance(urls, six.string_types):
urls = [urls]
connection = self.context.connection
for parsed_url in self.driver.parse_urls(urls):
self.driver.get_repository(
connection, parsed_url, self.arch, consumer
)
def load_packages(self, repositories, consumer):
"""Loads packages from repository.
:param repositories: the repository object
:param consumer: the callback to consume objects
"""
connection = self.context.connection
for r in repositories:
self.driver.get_packages(connection, r, consumer)
def assign_packages(self, repository, packages, keep_existing=True):
"""Assigns new packages to the repository.
It replaces the current repository`s packages.
:param repository: the target repository
:param packages: the set of new packages
:param keep_existing:
if True, all existing packages will be kept as is.
if False, all existing packages, that are not included
to new packages will be removed.
"""
if not isinstance(packages, set):
packages = set(packages)
else:
packages = packages.copy()
if keep_existing:
consume_exist = packages.add
else:
def consume_exist(package):
if package not in packages:
filepath = os.path.join(
package.repository.url, package.filename
)
logger.info("remove package - %s.", filepath)
os.remove(filepath)
self.driver.get_packages(
self.context.connection, repository, consume_exist
)
self.driver.rebuild_repository(repository, packages)
def copy_packages(self, repository, packages, keep_existing, observer):
"""Copies packages to repository.
:param repository: the target repository
:param packages: the set of packages
:param keep_existing: see assign_packages for more details
:param observer: the package copying process observer
"""
with self.context.async_section() as section:
for package in packages:
section.execute(
self._copy_package, repository, package, observer
)
self.assign_packages(repository, packages, keep_existing)
def clone_repositories(self, repositories, destination,
source=False, locale=False):
"""Creates copy of repositories.
:param repositories: the origin repositories
:param destination: the target folder
:param source: If True, the source packages will be copied too.
:param locale: If True, the localisation will be copied too.
:return: the mapping origin to cloned repository.
"""
mirros = dict()
destination = os.path.abspath(destination)
with self.context.async_section(0) as section:
for r in repositories:
section.execute(
self._fork_repository,
r, destination, source, locale, mirros
)
return mirros
def _fork_repository(self, r, destination, source, locale, mirrors):
"""Creates clone of repository and stores it in mirrors."""
new_repository = self.driver.fork_repository(
self.context.connection, r, destination, source, locale
)
mirrors[r] = new_repository
def _copy_package(self, target, package, observer):
"""Synchronises remote file to local fs."""
dst_path = os.path.join(target.url, package.filename)
src_path = urljoin(package.repository.url, package.filename)
bytes_copied = self.context.connection.retrieve(
src_path, dst_path, size=package.filesize
)
if package.filesize < 0:
package.filesize = bytes_copied
observer(bytes_copied)

View File

@ -1,82 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import abc
import logging
import six
@six.add_metaclass(abc.ABCMeta)
class RepositoryDriverBase(object):
"""The super class for Repository Drivers.
For implementing support of new type of repository:
- inherit this class
- implement all abstract methods
- register implementation in 'packetary.drivers' namespace
"""
def __init__(self):
self.logger = logging.getLogger(__package__)
@abc.abstractmethod
def parse_urls(self, urls):
"""Parses the repository url.
:return: the sequence of parsed urls
"""
@abc.abstractmethod
def get_repository(self, connection, url, arch, consumer):
"""Loads the repository meta information from URL.
:param connection: the connection manager instance
:param url: the repository`s url
:param arch: the repository`s architecture
:param consumer: the callback to consume result
"""
@abc.abstractmethod
def get_packages(self, connection, repository, consumer):
"""Loads packages from repository.
:param connection: the connection manager instance
:param repository: the repository object
:param consumer: the callback to consume result
"""
@abc.abstractmethod
def fork_repository(self, connection, repository, destination,
source=False, locale=False):
"""Creates the new repository with same metadata.
:param connection: the connection manager instance
:param repository: the source repository
:param destination: the destination folder
:param source: copy source files
:param locale: copy localisation
:return: The copy of repository
"""
@abc.abstractmethod
def rebuild_repository(self, repository, packages):
"""Re-builds the repository.
:param repository: the target repository
:param packages: the set of packages
"""

View File

@ -1,374 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from contextlib import closing
import copy
import datetime
import fcntl
import gzip
import os
from debian import deb822
from debian import debfile
from debian.debian_support import Version
import six
from packetary.drivers.base import RepositoryDriverBase
from packetary.library.checksum import composite as checksum_composite
from packetary.library.streams import GzipDecompress
from packetary.library import utils
from packetary.objects import FileChecksum
from packetary.objects import Package
from packetary.objects import PackageRelation
from packetary.objects import Repository
_OPERATORS_MAPPING = {
'>>': 'gt',
'<<': 'lt',
'=': 'eq',
'>=': 'ge',
'<=': 'le',
}
_ARCHITECTURES = {
"x86_64": "amd64",
"i386": "i386",
"source": "Source",
"amd64": "x86_64",
}
_PRIORITIES = {
"required": 1,
"important": 2,
"standard": 3,
"optional": 4,
"extra": 5
}
# Order is important
_REPOSITORY_FILES = [
"Packages",
"Release",
"Packages.gz"
]
# TODO(should be configurable)
_MANDATORY_PRIORITY = 3
_CHECKSUM_METHODS = (
"MD5Sum",
"SHA1",
"SHA256"
)
_checksum_collector = checksum_composite('md5', 'sha1', 'sha256')
class DebRepositoryDriver(RepositoryDriverBase):
def parse_urls(self, urls):
"""Overrides method of superclass."""
for url in urls:
try:
tokens = iter(x for x in url.split(" ") if x)
base, suite = next(tokens), next(tokens)
components = list(tokens)
except StopIteration:
raise ValueError("Invalid url: {0}".format(url))
base = base.rstrip("/")
if base.endswith("/dists"):
base = base[:-6]
# TODO(Flat Repository Format[1])
# [1] https://wiki.debian.org/RepositoryFormat
for component in components:
yield (base, suite, component)
def get_repository(self, connection, url, arch, consumer):
"""Overrides method of superclass."""
base, suite, component = url
release = self._get_url_of_metafile(
(base, suite, component, arch), "Release"
)
deb_release = deb822.Release(connection.open_stream(release))
consumer(Repository(
name=(deb_release["Archive"], deb_release["Component"]),
architecture=arch,
origin=deb_release["origin"],
url=base + "/"
))
def get_packages(self, connection, repository, consumer):
"""Overrides method of superclass."""
index = self._get_url_of_metafile(repository, "Packages.gz")
stream = GzipDecompress(connection.open_stream(index))
self.logger.info("loading packages from %s ...", repository)
pkg_iter = deb822.Packages.iter_paragraphs(stream)
counter = 0
for dpkg in pkg_iter:
try:
consumer(Package(
repository=repository,
name=dpkg["package"],
version=Version(dpkg['version']),
filesize=int(dpkg.get('size', -1)),
filename=dpkg["filename"],
checksum=FileChecksum(
md5=dpkg.get("md5sum"),
sha1=dpkg.get("sha1"),
sha256=dpkg.get("sha256"),
),
mandatory=self._is_mandatory(dpkg),
# Recommends are installed by default (since Lucid)
requires=self._get_relations(
dpkg, "depends", "pre-depends", "recommends"
),
# The deb does not have obsoletes section
obsoletes=[],
provides=self._get_relations(dpkg, "provides"),
))
except KeyError as e:
self.logger.error(
"Malformed index %s - %s: %s",
repository, six.text_type(dpkg), six.text_type(e)
)
raise
counter += 1
self.logger.info("loaded: %d packages from %s.", counter, repository)
def rebuild_repository(self, repository, packages):
"""Overrides method of superclass."""
basedir = utils.get_path_from_url(repository.url)
index_file = utils.get_path_from_url(
self._get_url_of_metafile(repository, "Packages")
)
utils.ensure_dir_exist(os.path.dirname(index_file))
index_gz = index_file + ".gz"
count = 0
with open(index_file, "wb") as fd1:
with closing(gzip.open(index_gz, "wb")) as fd2:
writer = utils.composite_writer(fd1, fd2)
for pkg in packages:
filename = os.path.join(basedir, pkg.filename)
with closing(debfile.DebFile(filename)) as deb:
debcontrol = deb.debcontrol()
debcontrol.setdefault("Origin", repository.origin)
debcontrol["Size"] = str(pkg.filesize)
debcontrol["Filename"] = pkg.filename
for k, v in six.moves.zip(_CHECKSUM_METHODS, pkg.checksum):
debcontrol[k] = v
writer(debcontrol.dump())
writer("\n")
count += 1
self.logger.info("saved %d packages in %s", count, repository)
self._update_suite_index(repository)
def fork_repository(self, connection, repository, destination,
source=False, locale=False):
# TODO(download gpk)
# TODO(sources and locales)
new_repo = copy.copy(repository)
new_repo.url = utils.localize_repo_url(destination, repository.url)
packages_file = utils.get_path_from_url(
self._get_url_of_metafile(new_repo, "Packages")
)
release_file = utils.get_path_from_url(
self._get_url_of_metafile(new_repo, "Release")
)
self.logger.info(
"clone repository %s to %s", repository, new_repo.url
)
utils.ensure_dir_exist(os.path.dirname(release_file))
release = deb822.Release()
release["Origin"] = repository.origin
release["Label"] = repository.origin
release["Archive"] = repository.name[0]
release["Component"] = repository.name[1]
release["Architecture"] = _ARCHITECTURES[repository.architecture]
with open(release_file, "wb") as fd:
release.dump(fd)
open(packages_file, "ab").close()
gzip.open(packages_file + ".gz", "ab").close()
return new_repo
def _update_suite_index(self, repository):
"""Updates the Release file in the suite."""
path = os.path.join(
utils.get_path_from_url(repository.url),
"dists", repository.name[0]
)
release_path = os.path.join(path, "Release")
self.logger.info(
"added repository suite release file: %s", release_path
)
with open(release_path, "a+b") as fd:
fcntl.flock(fd.fileno(), fcntl.LOCK_EX)
try:
fd.seek(0)
release = deb822.Release(fd)
self._add_to_release(release, repository)
for m in _CHECKSUM_METHODS:
release.setdefault(m, [])
self._add_files_to_release(
release, path, self._get_metafiles(repository)
)
fd.truncate(0)
release.dump(fd)
finally:
fcntl.flock(fd.fileno(), fcntl.LOCK_UN)
def _get_relations(self, dpkg, *names):
"""Gets the package relations.
:param dpkg: the debian-package object
:type dpkg: deb822.Packages
:param names: the relation names
:return: the list of PackageRelation objects
"""
relations = list()
for name in names:
for variants in dpkg.relations[name]:
relation = PackageRelation.from_args(
*(self._unparse_relation(v) for v in variants)
)
if relation is not None:
relations.append(relation)
return relations
def _get_metafiles(self, repository):
"""Gets the sequence of metafiles for repository."""
return (
utils.get_path_from_url(
self._get_url_of_metafile(repository, filename)
)
for filename in _REPOSITORY_FILES
)
@staticmethod
def _unparse_relation(relation):
"""Gets the relation parameters.
:param relation: the deb822.Releation object
:return: tuple(name, version_compare, version_edge)
"""
name = relation['name']
version = relation.get("version")
if version is None:
return name, None
else:
return name, _OPERATORS_MAPPING[version[0]], version[1]
@staticmethod
def _is_mandatory(dpkg):
"""Checks that package is mandatory.
:param dpkg: the debian-package object
:type dpkg: deb822.Packages
"""
if dpkg.get("essential") == "yes":
return True
return _PRIORITIES.get(
dpkg.get("priority"), _MANDATORY_PRIORITY + 1
) < _MANDATORY_PRIORITY
@staticmethod
def _get_url_of_metafile(repo_or_comps, filename):
"""Gets the URL of meta-file.
:param repo_or_comps: the repository object or
tuple(baseurl, suite, component, architecture)
:param filename: the name of meta-file
"""
if isinstance(repo_or_comps, Repository):
baseurl = repo_or_comps.url
suite, component = repo_or_comps.name
arch = repo_or_comps.architecture
else:
baseurl, suite, component, arch = repo_or_comps
return "/".join((
baseurl.rstrip("/"), "dists", suite, component,
"binary-" + _ARCHITECTURES[arch],
filename
))
@staticmethod
def _add_to_release(release, repository):
"""Adds repository information to debian release.
:param release: the deb822.Release instance
:param repository: the repository object
"""
# reset the date
release["Date"] = datetime.datetime.now().strftime(
"%a, %d %b %Y %H:%M:%S %Z"
)
release.setdefault("Origin", repository.origin)
release.setdefault("Label", repository.origin)
release.setdefault("Suite", repository.name[0])
release.setdefault("Codename", repository.name[0].split("-", 1)[0])
release.setdefault("Description", "The packages repository.")
keys = ("Architectures", "Components")
values = (repository.architecture, repository.name[1])
for key, value in six.moves.zip(keys, values):
if key in release:
release[key] = utils.append_token_to_string(
release[key],
value
)
else:
release[key] = value
@staticmethod
def _add_files_to_release(release, basepath, files):
"""Adds information about meta files to debian release.
:param release: the deb822.Release instance
:param basepath: the suite folder path
:param files: the sequence of files
"""
files_info = utils.get_size_and_checksum_for_files(
files, _checksum_collector
)
for filepath, size, cs in files_info:
fname = filepath[len(basepath) + 1:]
size = six.text_type(size)
for m, checksum in six.moves.zip(_CHECKSUM_METHODS, cs):
for v in release[m]:
if v["name"] == fname:
v[m] = checksum
v["size"] = size
break
else:
release[m].append(deb822.Deb822Dict({
m: checksum,
"size": size,
"name": fname
}))

View File

@ -1,283 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import copy
import multiprocessing
import os
import shutil
import createrepo
import lxml.etree as etree
import six
from packetary.drivers.base import RepositoryDriverBase
from packetary.library.streams import GzipDecompress
from packetary.library import utils
from packetary.objects import FileChecksum
from packetary.objects import Package
from packetary.objects import PackageRelation
from packetary.objects import PackageVersion
from packetary.objects import Repository
urljoin = six.moves.urllib.parse.urljoin
# TODO(configurable option for drivers)
_CORE_GROUPS = ("core", "base")
_MANDATORY_TYPES = ("mandatory", "default")
# The namespaces are used in metadata xml of repository
_NAMESPACES = {
"main": "http://linux.duke.edu/metadata/common",
"md": "http://linux.duke.edu/metadata/repo",
"rpm": "http://linux.duke.edu/metadata/rpm"
}
class CreaterepoCallBack(object):
"""Callback object for createrepo"""
def __init__(self, logger):
self.logger = logger
def errorlog(self, msg):
"""Error log output."""
self.logger.error(msg)
def log(self, msg):
"""Logs message."""
self.logger.info(msg)
def progress(self, item, current, total):
""""Progress bar."""
pass
class RpmRepositoryDriver(RepositoryDriverBase):
def parse_urls(self, urls):
"""Overrides method of superclass."""
return (url.rstrip("/") for url in urls)
def get_repository(self, connection, url, arch, consumer):
name = utils.get_path_from_url(url, False)
consumer(Repository(
name=name,
url=url + "/",
architecture=arch,
origin=""
))
def get_packages(self, connection, repository, consumer):
"""Overrides method of superclass."""
baseurl = repository.url
repomd = urljoin(baseurl, "repodata/repomd.xml")
self.logger.debug("repomd: %s", repomd)
repomd_tree = etree.parse(connection.open_stream(repomd))
mandatory = self._get_mandatory_packages(
self._load_db(
connection, baseurl, repomd_tree, "group_gz", "group"
)
)
primary_db = self._load_db(connection, baseurl, repomd_tree, "primary")
if primary_db is None:
raise ValueError("Malformed repository: {0}".format(repository))
counter = 0
for tag in primary_db.iterfind("./main:package", _NAMESPACES):
try:
name = tag.find("./main:name", _NAMESPACES).text
consumer(Package(
repository=repository,
name=tag.find("./main:name", _NAMESPACES).text,
version=self._unparse_version_attrs(
tag.find("./main:version", _NAMESPACES).attrib
),
filesize=int(
tag.find("./main:size", _NAMESPACES)
.attrib.get("package", -1)
),
filename=tag.find(
"./main:location", _NAMESPACES
).attrib["href"],
checksum=self._get_checksum(tag),
mandatory=name in mandatory,
requires=self._get_relations(tag, "requires"),
obsoletes=self._get_relations(tag, "obsoletes"),
provides=self._get_relations(tag, "provides")
))
except (ValueError, KeyError) as e:
self.logger.error(
"Malformed tag %s - %s: %s",
repository, etree.tostring(tag), six.text_type(e)
)
raise
counter += 1
self.logger.info("loaded: %d packages from %s.", counter, repository)
def rebuild_repository(self, repository, packages):
"""Overrides method of superclass."""
basepath = utils.get_path_from_url(repository.url)
self.logger.info("rebuild repository in %s", basepath)
md_config = createrepo.MetaDataConfig()
try:
md_config.workers = multiprocessing.cpu_count()
md_config.directory = str(basepath)
md_config.update = True
mdgen = createrepo.MetaDataGenerator(
config_obj=md_config, callback=CreaterepoCallBack(self.logger)
)
mdgen.doPkgMetadata()
mdgen.doRepoMetadata()
mdgen.doFinalMove()
except createrepo.MDError as e:
err_msg = six.text_type(e)
self.logger.exception(
"failed to create yum repository in %s: %s",
basepath,
err_msg
)
shutil.rmtree(
os.path.join(md_config.outputdir, md_config.tempdir),
ignore_errors=True
)
raise RuntimeError(
"Failed to create yum repository in {0}."
.format(err_msg))
def fork_repository(self, connection, repository, destination,
source=False, locale=False):
# TODO(download gpk)
# TODO(sources and locales)
new_repo = copy.copy(repository)
new_repo.url = utils.localize_repo_url(destination, repository.url)
self.logger.info(
"clone repository %s to %s", repository, new_repo.url
)
utils.ensure_dir_exist(new_repo.url)
self.rebuild_repository(new_repo, set())
return new_repo
def _load_db(self, connection, baseurl, repomd, *aliases):
"""Loads database.
:param connection: the connection object
:param baseurl: the base repository URL
:param repomd: the parsed metadata of repository
:param aliases: the aliases of database name
:return: parsed database file or None if db does not exist
"""
for dbname in aliases:
self.logger.debug("loading %s database...", dbname)
node = repomd.find(
"./md:data[@type='{0}']".format(dbname), _NAMESPACES
)
if node is not None:
break
else:
return
url = urljoin(
baseurl,
node.find("./md:location", _NAMESPACES).attrib["href"]
)
self.logger.debug("loading %s - %s...", dbname, url)
stream = connection.open_stream(url)
if url.endswith(".gz"):
stream = GzipDecompress(stream)
return etree.parse(stream)
def _get_mandatory_packages(self, groups_db):
"""Get the set of mandatory package names.
:param groups_db: the parsed groups database
"""
package_names = set()
if groups_db is None:
return package_names
count = 0
for name in _CORE_GROUPS:
result = groups_db.xpath("./group/id[text()='{0}']".format(name))
if len(result) == 0:
self.logger.warning("the group '%s' is not found.", name)
continue
group = result[0].getparent()
for t in _MANDATORY_TYPES:
xpath = "./packagelist/packagereq[@type='{0}']".format(t)
for tag in group.iterfind(xpath):
package_names.add(tag.text)
count += 1
self.logger.info("detected %d mandatory packages.", count)
return package_names
def _get_relations(self, pkg_tag, name):
"""Gets package relations by name from package tag.
:param pkg_tag: the xml-tag with package description
:param name: the relations name
:return: list of PackageRelation objects
"""
relations = list()
append = relations.append
tags_iter = pkg_tag.iterfind(
"./main:format/rpm:%s/rpm:entry" % name,
_NAMESPACES
)
for elem in tags_iter:
append(PackageRelation.from_args(
self._unparse_relation_attrs(elem.attrib)
))
return relations
def _get_checksum(self, pkg_tag):
"""Gets checksum from package tag."""
checksum = dict.fromkeys(("md5", "sha1", "sha256"), None)
checksum_tag = pkg_tag.find("./main:checksum", _NAMESPACES)
checksum[checksum_tag.attrib["type"]] = checksum_tag.text
return FileChecksum(**checksum)
def _unparse_relation_attrs(self, attrs):
"""Gets the package relation from attributes.
:param attrs: the relation tag attributes
:return tuple(name, version_op, version_edge)
"""
if "flags" not in attrs:
return attrs['name'], None
return (
attrs['name'],
attrs["flags"].lower(),
self._unparse_version_attrs(attrs)
)
@staticmethod
def _unparse_version_attrs(attrs):
"""Gets the package version from attributes.
:param attrs: the relation tag attributes
:return: the PackageVersion object
"""
return PackageVersion(
int(attrs.get("epoch", 0)),
attrs.get("ver", "0.0").split("."),
attrs.get("rel", "0").split(".")
)

View File

@ -1,21 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import eventlet
eventlet.monkey_patch()

View File

@ -1,84 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import functools
import hashlib
class _HashComposite(object):
"""Combines several hash methods."""
def __init__(self, hash_objects):
self.hash_objects = hash_objects
def update(self, data):
"""Updates the hash objects with the string arg.
For more details see doc of hashlib.update.
"""
for o in self.hash_objects:
o.update(data)
def hexdigest(self):
"""Returns the list of appropriate hexdigests of hash_objects.
For more details see doc of hashlib.hexdigest.
"""
return [o.hexdigest() for o in self.hash_objects]
def _new_composite(methods):
"""Creates new composite method."""
def wrapper():
return _HashComposite([x() for x in methods])
return wrapper
def _checksum(method):
"""Makes function to calculate checksum for stream."""
@functools.wraps(method)
def calculate(stream, chunksize=16 * 1024):
"""Calculates checksum for binary stream.
:param stream: file-like object opened in binary mode.
:return: the checksum of content in terms of method.
"""
s = method()
while True:
chunk = stream.read(chunksize)
if not chunk:
break
s.update(chunk)
return s.hexdigest()
return calculate
md5 = _checksum(hashlib.md5)
sha1 = _checksum(hashlib.sha1)
sha256 = _checksum(hashlib.sha256)
def composite(*methods):
"""Calculate several checksum at one time."""
return _checksum(_new_composite(
[getattr(hashlib, x) for x in methods]
))

View File

@ -1,307 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import errno
import logging
import os
import six
import six.moves.http_client as http_client
import six.moves.urllib.request as urllib
import six.moves.urllib_error as urlerror
import time
from packetary.library.streams import StreamWrapper
from packetary.library.utils import ensure_dir_exist
logger = logging.getLogger(__package__)
RETRYABLE_ERRORS = (http_client.HTTPException, IOError)
class RangeError(urlerror.URLError):
pass
class RetryableRequest(urllib.Request):
MAX_TIMEOUT = 5
offset = 0
retries_left = 1
retry_interval = 0
start_time = 0
def get_retry_interval(self):
"""Calculates progressive retry interval in seconds.
:return: the time to wait before start retry
"""
# we uses progressive timeout between retries,
# the greatest number of retry will have greatest timeout
# but limited with max_delay
coef = max(self.MAX_TIMEOUT - self.retries_left, 1)
timeout = self.retry_interval * coef
return min(timeout, self.MAX_TIMEOUT)
class ResumableResponse(StreamWrapper):
"""The http-response wrapper to add resume ability.
Allows to resume read from same position if connection is lost.
"""
def __init__(self, request, response, opener):
"""Initialises.
:param request: the original http request
:param response: the original http response
:param opener: the instance of urllib.OpenerDirector
"""
super(ResumableResponse, self).__init__(response)
self.request = request
self.opener = opener
def read_chunk(self, chunksize):
"""Overrides super class method."""
while 1:
try:
chunk = self.stream.read(chunksize)
self.request.offset += len(chunk)
return chunk
except RETRYABLE_ERRORS as e:
# TODO(check hashsums)
response = self.opener.error(
self.request.get_type(), self.request,
self.stream, 502, six.text_type(e), self.stream.info()
)
self.stream = response.stream
class RetryHandler(urllib.HTTPRedirectHandler):
"""urllib Handler to add ability for retrying on server errors."""
def redirect_request(self, req, fp, code, msg, headers, newurl):
new_req = urllib.HTTPRedirectHandler.redirect_request(
self, req, fp, code, msg, headers, newurl
)
if new_req is not None:
# We use class assignment for casting new request to type
# RetryableRequest
new_req.__class__ = RetryableRequest
new_req.retries_left = req.retries_left
new_req.offset = req.offset
new_req.start_time = req.start_time
new_req.retry_interval = req.retry_interval
return new_req
@staticmethod
def http_request(request):
"""Initialises http request.
:param request: the instance of RetryableRequest
:return: the request
"""
logger.debug("start request: %s", request.get_full_url())
if request.offset > 0:
request.add_header('Range', 'bytes=%d-' % request.offset)
request.start_time = time.time()
return request
def http_response(self, request, response):
"""Wraps response in a ResumableResponse.
Checks that partial request completed successfully.
:param request: the instance of RetryableRequest
:param response: the response object
:return: ResumableResponse if success otherwise same response
"""
code, msg = response.getcode(), response.msg
if 300 <= code < 400:
# the redirect group, pass to next handler as is
return response
# the server should response partial content if range is specified
if request.offset > 0 and code != 206:
raise RangeError(msg)
if code >= 400:
logger.error(
"request failed: %s - %d(%s), retries left - %d.",
request.get_full_url(), code, msg, request.retries_left - 1
)
if is_retryable_http_error(code) and request.retries_left > 0:
time.sleep(request.get_retry_interval())
request.retries_left -= 1
response = self.parent.open(request)
# pass response to next handler as is.
return response
logger.debug(
"request completed: %s - %d (%s), duration - %d ms.",
request.get_full_url(), response.getcode(), response.msg,
int((time.time() - request.start_time) * 1000)
)
return ResumableResponse(request, response, self.parent)
https_request = http_request
https_response = http_response
def is_retryable_http_error(code):
"""Checks that http error can be retried.
:param code: the HTTP_CODE
:return: True if request can be retried otherwise False
"""
return code >= http_client.INTERNAL_SERVER_ERROR
class ConnectionsManager(object):
"""The connections manager."""
def __init__(self, proxy=None, secure_proxy=None,
retries_num=0, retry_interval=0):
"""Initialises.
:param proxy: the url of proxy for http-connections
:param secure_proxy: the url of proxy for https-connections
:param retries_num: the number of allowed retries
:param retry_interval: the time between retries (in seconds)
"""
if proxy:
proxies = {
"http": proxy,
"https": secure_proxy or proxy,
}
else:
proxies = None
self.retries_num = retries_num
self.retry_interval = retry_interval
self.opener = urllib.build_opener(
RetryHandler(),
urllib.ProxyHandler(proxies)
)
def make_request(self, url, offset=0):
"""Makes new http request.
:param url: the remote file`s url
:param offset: the number of bytes from the beginning,
that will be skipped
:return: The new http request
"""
if url.startswith("/"):
url = "file://" + url
request = RetryableRequest(url)
request.retries_left = self.retries_num
request.retry_interval = self.retry_interval
request.offset = offset
return request
def open_stream(self, url, offset=0):
"""Opens remote file for streaming.
:param url: the remote file`s url
:param offset: the number of bytes from the beginning,
that will be skipped
"""
request = self.make_request(url, offset)
while 1:
try:
return self.opener.open(request)
except (RangeError, urlerror.HTTPError):
raise
except RETRYABLE_ERRORS as e:
if request.retries_left <= 0:
raise
request.retries_left -= 1
logger.exception(
"Failed to open url - %s: %s. retries left - %d.",
url, six.text_type(e), request.retries_left
)
time.sleep(request.get_retry_interval())
def retrieve(self, url, filename, **attributes):
"""Downloads remote file.
:param url: the remote file`s url
:param filename: the target filename on local filesystem
:param attributes: the file attributes, like size, hashsum, etc.
:return: the count of actually copied bytes
"""
offset = 0
try:
stats = os.stat(filename)
expected_size = attributes.get('size', -1)
if expected_size == stats.st_size:
# TODO(check hashsum)
return 0
if stats.st_size < expected_size:
offset = stats.st_size
except OSError as e:
if e.errno != errno.ENOENT:
raise
ensure_dir_exist(os.path.dirname(filename))
logger.info("download: %s from the offset: %d", url, offset)
fd = os.open(filename, os.O_CREAT | os.O_WRONLY)
try:
return self._copy_stream(fd, url, offset)
except RangeError:
if offset == 0:
raise
logger.warning(
"Failed to resume download, starts from the beginning: %s",
url
)
return self._copy_stream(fd, url, 0)
finally:
os.fsync(fd)
os.close(fd)
def _copy_stream(self, fd, url, offset):
"""Copies remote file to local.
:param fd: the file`s descriptor
:param url: the remote file`s url
:param offset: the number of bytes from the beginning,
that will be skipped
:return: the count of actually copied bytes
"""
source = self.open_stream(url, offset)
os.ftruncate(fd, offset)
os.lseek(fd, offset, os.SEEK_SET)
chunk_size = 16 * 1024
size = 0
while 1:
chunk = source.read(chunk_size)
if not chunk:
break
os.write(fd, chunk)
size += len(chunk)
return size

View File

@ -1,93 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from __future__ import with_statement
import logging
import sys
import six
from eventlet.greenpool import GreenPool
logger = logging.getLogger(__package__)
class AsynchronousSection(object):
"""Allows calling function asynchronously with waiting on exit."""
MIN_POOL_SIZE = 1
def __init__(self, size=0, ignore_errors_num=0):
"""Initialises.
:param size: the max number of parallel tasks
:param ignore_errors_num:
number of errors which does not stop the execution
"""
self.executor = GreenPool(max(size, self.MIN_POOL_SIZE))
self.ignore_errors_num = ignore_errors_num
self.errors = []
self.tasks = set()
def __enter__(self):
self.errors[:] = []
return self
def __exit__(self, etype, *_):
self.wait(etype is not None)
def execute(self, func, *args, **kwargs):
"""Calls function asynchronously."""
if 0 <= self.ignore_errors_num < len(self.errors):
raise RuntimeError("Too many errors.")
gt = self.executor.spawn(func, *args, **kwargs)
self.tasks.add(gt)
gt.link(self.on_complete)
return gt
def on_complete(self, gt):
"""Callback to handle task completion."""
try:
gt.wait()
except Exception as e:
logger.error("Task failed: %s", six.text_type(e))
self.errors.append(sys.exc_info())
finally:
self.tasks.discard(gt)
def wait(self, ignore_errors=False):
"""Waits until all tasks will be completed.
Do not use directly, will be called from context manager.
"""
self.executor.waitall()
if len(self.errors) > 0:
for exc_info in self.errors:
logger.exception("error details.", exc_info=exc_info)
self.errors[:] = []
if not ignore_errors:
raise RuntimeError(
"Operations completed with errors.\n"
"See log for more details."
)

View File

@ -1,133 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import zlib
class StreamWrapper(object):
"""Helper class to implement stream wrappers.
It is base-class for Streamers,
that provides functionality to transform stream on the fly.
The wrapped stream may return data more that required,
the extra read data will be kept in the internal buffer till
next read.
"""
CHUNK_SIZE = 1024
def __init__(self, stream):
"""Initializes.
:param stream: file-like object opened in binary mode.
"""
self.stream = stream
self.unread_tail = b""
def __getattr__(self, item):
return getattr(self.stream, item)
def _read_tail(self):
tmp = self.unread_tail
self.unread_tail = b""
return tmp
def _align_chunk(self, chunk, size):
self.unread_tail = chunk[size:]
return chunk[:size]
def read_chunk(self, chunksize):
"""Overrides this method to change default behaviour."""
return self.stream.read(chunksize)
def read(self, size=-1):
result = self._read_tail()
if size < 0:
while True:
chunk = self.read_chunk(self.CHUNK_SIZE)
if not chunk:
break
result += chunk
else:
if len(result) > size:
result = self._align_chunk(result, size)
size -= len(result)
while size > 0:
chunk = self.read_chunk(self.CHUNK_SIZE)
if not chunk:
break
if len(chunk) > size:
chunk = self._align_chunk(chunk, size)
size -= len(chunk)
result += chunk
return result
def readline(self):
pos = self.unread_tail.find(b"\n")
if pos >= 0:
line = self._align_chunk(self.unread_tail, pos + 1)
else:
line = self._read_tail()
while True:
chunk = self.read_chunk(self.CHUNK_SIZE)
if not chunk:
break
pos = chunk.find(b"\n")
if pos >= 0:
line += self._align_chunk(chunk, pos + 1)
break
line += chunk
return line
def readlines(self):
while True:
line = self.readline()
if not line:
break
yield line
def __iter__(self):
return self.readlines()
class GzipDecompress(StreamWrapper):
"""The decompress stream."""
def __init__(self, stream):
super(GzipDecompress, self).__init__(stream)
# Magic parameter makes zlib module understand gzip header
# http://stackoverflow.com/questions/1838699/how-can-i-decompress-a-gzip-stream-with-zlib
# This works on cpython and pypy, but not jython.
self.decompress = zlib.decompressobj(16 + zlib.MAX_WBITS)
def read_chunk(self, chunksize):
if self.decompress.unconsumed_tail:
uncompressed = self.decompress.decompress(
self.decompress.unconsumed_tail, chunksize
)
if uncompressed:
return uncompressed
while True:
chunk = self.stream.read(chunksize)
if not chunk:
break
uncompressed = self.decompress.decompress(chunk, chunksize)
if uncompressed:
return uncompressed
return self.decompress.flush()

View File

@ -1,115 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from __future__ import with_statement
import errno
import os
import six
urlparse = six.moves.urllib.parse.urlparse
def append_token_to_string(tokens, token):
"""Adds new token to space separated list of tokens.
:param tokens: the 'sep' separated list
:param token: new item
"""
values = tokens.split()
if token not in values:
values.append(token)
values.sort()
return ' '.join(values)
return tokens
def composite_writer(*args):
"""Makes helper, that writes into several files simultaneously.
:param args: the list of file objects
:return: the callable object - writer
"""
def write(text):
"""Writes simultaneously to all files with utf-8 encoding control.
:param text: the text, that needs to write
"""
if isinstance(text, six.text_type):
text = text.encode("utf-8")
for arg in args:
arg.write(text)
return write
def get_size_and_checksum_for_files(files, checksum_algo):
"""Gets the path, size and checksum for files.
:param files: the sequence of files
:param checksum_algo: the checksum calculator
:return the sequence of tuples(filename, size, checksum)
"""
for filename in files:
with open(filename, "rb") as fd:
size = os.fstat(fd.fileno()).st_size
checksum = checksum_algo(fd)
yield filename, size, checksum
def get_path_from_url(url, ensure_file=True):
"""Get the path from the URL.
:param url: the URL
:param ensure_file: If True, ensure that scheme is "file"
:return: the path component from URL
:raises ValueError
"""
comps = urlparse(url, scheme="file")
if ensure_file and comps.scheme != "file":
raise ValueError(
"The absolute path is expected, actual have: {0}.".format(url)
)
if os.sep != "/":
return comps.path.replace("/", os.sep)
return comps.path
def localize_repo_url(localurl, repo_url):
"""Gets local repository url.
:param localurl: the base local URL
:param repo_url: the origin URL of repository
:return: localurl + get_path_from_url(repo_url)
"""
return localurl.rstrip("/") + urlparse(repo_url).path
def ensure_dir_exist(path):
"""Creates directory if it does not exist.
:param path: the full path to directory
"""
try:
os.makedirs(path)
except OSError as e:
if e.errno != errno.EEXIST:
raise

View File

@ -1,38 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from packetary.objects.index import Index
from packetary.objects.package import FileChecksum
from packetary.objects.package import Package
from packetary.objects.package_relation import PackageRelation
from packetary.objects.package_relation import VersionRange
from packetary.objects.package_version import PackageVersion
from packetary.objects.packages_tree import PackagesTree
from packetary.objects.repository import Repository
__all__ = [
"FileChecksum",
"Index",
"Package",
"PackageRelation",
"PackagesTree",
"PackageVersion",
"Repository",
"VersionRange",
]

View File

@ -1,62 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import abc
import six
@six.add_metaclass(abc.ABCMeta)
class ComparableObject(object):
"""Superclass for objects, that should be comparable.
Note: because python3 does not support __cmp__ slot, use
cmp method to implement all of compare methods.
"""
@abc.abstractmethod
def cmp(self, other):
"""Compares with other object.
:return: value is negative if if self < other, zero if self == other
strictly positive if x > y
"""
def __lt__(self, other):
return self.cmp(other) < 0
def __le__(self, other):
return self.cmp(other) <= 0
def __gt__(self, other):
return self.cmp(other) > 0
def __ge__(self, other):
return self.cmp(other) >= 0
def __eq__(self, other):
if other is self:
return True
return isinstance(other, type(self)) and self.cmp(other) == 0
def __ne__(self, other):
if other is self:
return False
return not isinstance(other, type(self)) or self.cmp(other) != 0
def __cmp__(self, other):
return self.cmp(other)

View File

@ -1,208 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from bintrees import FastRBTree
from collections import defaultdict
import functools
import operator
import six
def _make_operator(direction, op):
"""Makes search operator from low-level operation and search direction."""
return functools.partial(direction, condition=op)
def _start_upperbound(versions, version, condition):
"""Gets all versions from [start, version] that meet condition.
:param versions: the tree of versions.
:param version: the required version
:param condition: condition for search
:return: the list of found versions
"""
result = list(versions.value_slice(None, version))
try:
bound = versions.ceiling_item(version)
if condition(bound[0], version):
result.append(bound[1])
except KeyError:
pass
return result
def _lowerbound_end(versions, version, condition):
"""Gets all versions from [version, end] that meet condition.
:param versions: the tree of versions.
:param version: the required version
:param condition: condition for search
:return: the list of found versions
"""
result = []
items = iter(versions.item_slice(version, None))
bound = next(items, None)
if bound is None:
return result
if condition(bound[0], version):
result.append(bound[1])
result.extend(x[1] for x in items)
return result
def _equal(tree, version):
"""Gets the package with specified version."""
if version in tree:
return [tree[version]]
return []
def _any(tree, _):
"""Gets the package with max version."""
return list(tree.values())
class Index(object):
"""The search index for packages.
Builds three search-indexes:
- index of packages with versions.
- index of virtual packages (provides).
- index of obsoleted packages (obsoletes).
Uses to find package by name and range of versions.
"""
operators = {
None: _any,
"lt": _make_operator(_start_upperbound, operator.lt),
"le": _make_operator(_start_upperbound, operator.le),
"gt": _make_operator(_lowerbound_end, operator.gt),
"ge": _make_operator(_lowerbound_end, operator.ge),
"eq": _equal,
}
def __init__(self):
self.packages = defaultdict(FastRBTree)
self.obsoletes = defaultdict(FastRBTree)
self.provides = defaultdict(FastRBTree)
def __iter__(self):
"""Iterates over all packages including versions."""
return self.get_all()
def __len__(self, _reduce=six.functools.reduce):
"""Returns the total number of packages with versions."""
return _reduce(
lambda x, y: x + len(y),
six.itervalues(self.packages),
0
)
def get_all(self):
"""Gets sequence from all of packages including versions."""
for versions in six.itervalues(self.packages):
for version in versions.values():
yield version
def find(self, name, version):
"""Finds the package by name and range of versions.
:param name: the package`s name.
:param version: the range of versions.
:return: the package if it is found, otherwise None
"""
candidates = self.find_all(name, version)
if len(candidates) > 0:
return candidates[-1]
return None
def find_all(self, name, version):
"""Finds the packages by name and range of versions.
:param name: the package`s name.
:param version: the range of versions.
:return: the list of suitable packages
"""
if name in self.packages:
candidates = self._find_versions(
self.packages[name], version
)
if len(candidates) > 0:
return candidates
if name in self.obsoletes:
return self._resolve_relation(
self.obsoletes[name], version
)
if name in self.provides:
return self._resolve_relation(
self.provides[name], version
)
return []
def add(self, package):
"""Adds new package to indexes.
:param package: the package object.
"""
self.packages[package.name][package.version] = package
key = package.name, package.version
for obsolete in package.obsoletes:
self.obsoletes[obsolete.name][key] = obsolete
for provide in package.provides:
self.provides[provide.name][key] = provide
def _resolve_relation(self, relations, version):
"""Resolve relation according to relations index.
:param relations: the index of relations
:param version: the range of versions
:return: package if found, otherwise None
"""
for key, candidate in relations.iter_items(reverse=True):
if candidate.version.has_intersection(version):
return [self.packages[key[0]][key[1]]]
return []
@staticmethod
def _find_versions(versions, version):
"""Searches accurate version.
Search for the highest version out of intersection
of existing and required range of versions.
:param versions: the existing versions
:param version: the required range of versions
:return: package if found, otherwise None
"""
try:
op = Index.operators[version.op]
except KeyError:
raise ValueError(
"Unsupported operation: {0}"
.format(version.op)
)
return op(versions, version.edge)

View File

@ -1,80 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from collections import namedtuple
from packetary.objects.base import ComparableObject
FileChecksum = namedtuple("FileChecksum", ("md5", "sha1", "sha256"))
class Package(ComparableObject):
"""Structure to describe package object."""
def __init__(self, repository, name, version, filename,
filesize, checksum, mandatory=False,
requires=None, provides=None, obsoletes=None):
"""Initialises.
:param name: the package`s name
:param version: the package`s version
:param filename: the package`s relative filename
:param filesize: the package`s file size
:param checksum: the package`s checksum
:param requires: the package`s requirements(optional)
:param provides: the package`s provides(optional)
:param obsoletes: the package`s obsoletes(optional)
:param mandatory: indicates that package is mandatory
"""
self.repository = repository
self.name = name
self.version = version
self.filename = filename
self.checksum = checksum
self.filesize = filesize
self.requires = requires or []
self.provides = provides or []
self.obsoletes = obsoletes or []
self.mandatory = mandatory
def __copy__(self):
"""Creates shallow copy of package."""
return Package(**self.__dict__)
def __str__(self):
return "{0} {1}".format(self.name, self.version)
def __unicode__(self):
return u"{0} {1}".format(self.name, self.version)
def __hash__(self):
return hash((self.name, self.version))
def cmp(self, other):
"""Compares with other Package object."""
if self.name < other.name:
return -1
if self.name > other.name:
return 1
if self.version < other.version:
return -1
if self.version > other.version:
return 1
return 0

View File

@ -1,157 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import operator
class VersionRange(object):
"""Describes the range of versions.
Range of version is compare operation and edge.
the compare operation can be one of:
equal, greater, less, greater or equal, less or equal.
"""
__slots__ = ["op", "edge"]
def __init__(self, op=None, edge=None):
"""Initialises.
:param op: the name of operator to compare.
:param edge: the edge of versions.
"""
self.op = op
self.edge = edge
def __contains__(self, point):
return getattr(operator, self.op)(point, self.edge)
def __hash__(self):
return hash((self.op, self.edge))
def __eq__(self, other):
if not isinstance(other, VersionRange):
return False
return self.op == other.op and \
self.edge == other.edge
def __str__(self):
if self.edge is not None:
return "{0} {1}".format(self.op, self.edge)
return "any"
def __unicode__(self):
if self.edge is not None:
return u"{0} {1}".format(self.op, self.edge)
return u"any"
def has_intersection(self, other):
"""Checks that 2 ranges has intersection.
:param other: the candidate to check
:return: True if intersection exists, otherwise False
:raise TypeError: when other does not instance of VersionRange
"""
if not isinstance(other, VersionRange):
raise TypeError(
"Unorderable type <type 'VersionRange'> and {0}"
.format(type(other))
)
if self.op is None or other.op is None:
return True
if self.op[0] == other.op[0]:
if self.op == 'eq':
return self.edge == other.edge
# the intersection is -inf or +inf
return True
if self.edge == other.edge:
# need to cover case < a and >= a
return self.edge in other and other.edge in self
# all other cases
return self.edge in other or other.edge in self
class PackageRelation(object):
"""Describes the package`s relation.
Relation includes the name of required package
and range of versions that satisfies requirement.
"""
__slots__ = ["name", "version", "alternative"]
def __init__(self, name, version=None, alternative=None):
"""Initialises.
:param name: the name of required package
:param version: the version range of required package
:param alternative: the alternative relation
"""
self.name = name
self.version = VersionRange() if version is None else version
self.alternative = alternative
@classmethod
def from_args(cls, *args):
"""Construct relation from list of arguments.
:param args: the list of tuples(name, [version_op, version_edge])
"""
if len(args) == 0:
return None
head = args[0]
name = head[0]
version = VersionRange(*head[1:])
alternative = cls.from_args(*args[1:])
return cls(name, version, alternative)
def __iter__(self):
"""Iterates over alternatives."""
r = self
while r is not None:
yield r
r = r.alternative
def __hash__(self):
return hash((self.name, self.version))
def __eq__(self, other):
if not isinstance(other, PackageRelation):
return False
return self.name == other.name and \
self.version == other.version
def __str__(self):
if self.alternative is None:
return "{0} ({1})".format(self.name, self.version)
return "{0} ({1}) | {2}".format(
self.name, self.version, self.alternative
)
def __unicode__(self):
if self.alternative is None:
return u"{0} ({1})".format(self.name, self.version)
return u"{0} ({1}) | {2}".format(
self.name, self.version, self.alternative
)

View File

@ -1,127 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from packetary.objects.base import ComparableObject
class PackageVersion(ComparableObject):
"""The Package version."""
__slots__ = ["epoch", "version", "release"]
def __init__(self, epoch, version, release):
self.epoch = int(epoch)
self.version = tuple(version)
self.release = tuple(release)
@classmethod
def from_string(cls, text):
"""Constructs from string.
:param text: the version in format '[{epoch-}]-{version}-{release}'
"""
components = text.split("-")
if len(components) > 2:
epoch = components[0]
components = components[1:]
else:
epoch = 0
return cls(epoch, components[0].split("."), components[1].split("."))
def cmp(self, other):
if not isinstance(other, PackageVersion):
other = PackageVersion.from_string(str(other))
if not isinstance(other, PackageVersion):
raise TypeError
if self.epoch < other.epoch:
return -1
if self.epoch > other.epoch:
return 1
res = self._cmp_version_part(self.version, other.version)
if res != 0:
return res
return self._cmp_version_part(self.release, other.release)
def __eq__(self, other):
if other is self:
return True
return self.cmp(other) == 0
def __str__(self):
return "{0}-{1}-{2}".format(
self.epoch,
".".join(str(x) for x in self.version),
".".join(str(x) for x in self.release)
)
@classmethod
def _order(cls, x):
"""Return an integer value for character x"""
if x.isdigit():
return int(x) + 1
if x.isalpha():
return ord(x)
return ord(x) + 256
@classmethod
def _cmp_version_string(cls, version1, version2):
"""Compares two versions as string."""
la = [cls._order(x) for x in version1]
lb = [cls._order(x) for x in version2]
while la or lb:
a = 0
b = 0
if la:
a = la.pop(0)
if lb:
b = lb.pop(0)
if a < b:
return -1
elif a > b:
return 1
return 0
@classmethod
def _cmp_version_part(cls, version1, version2):
"""Compares two versions."""
ver1_it = iter(version1)
ver2_it = iter(version2)
while True:
v1 = next(ver1_it, None)
v2 = next(ver2_it, None)
if v1 is None or v2 is None:
if v1 is not None:
return 1
if v2 is not None:
return -1
return 0
if v1.isdigit() and v2.isdigit():
a = int(v1)
b = int(v2)
if a < b:
return -1
if a > b:
return 1
else:
r = cls._cmp_version_string(v1, v2)
if r != 0:
return r

View File

@ -1,129 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import warnings
from packetary.objects.index import Index
class UnresolvedWarning(UserWarning):
"""Warning about unresolved depends."""
pass
class PackagesTree(Index):
"""Helper class to deal with dependency graph."""
def __init__(self):
super(PackagesTree, self).__init__()
self.mandatory_packages = []
def add(self, package):
super(PackagesTree, self).add(package)
# store all mandatory packages in separated list for quick access
if package.mandatory:
self.mandatory_packages.append(package)
def get_unresolved_dependencies(self, base=None):
"""Gets the set of unresolved dependencies.
:param base: the base index to resolve dependencies
:return: the set of unresolved depends.
"""
external = self.__get_unresolved_dependencies(self)
if base is None:
return external
unresolved = set()
for relation in external:
for rel in relation:
if base.find(rel.name, rel.version) is not None:
break
else:
unresolved.add(relation)
return unresolved
def get_minimal_subset(self, main, requirements):
"""Gets the minimal work subset.
:param main: the main index, to complete requirements.
:param requirements: additional requirements.
:return: The set of resolved depends.
"""
unresolved = set()
resolved = set()
if main is None:
def pkg_filter(*_):
pass
else:
pkg_filter = main.find
self.__get_unresolved_dependencies(main, requirements)
stack = list()
stack.append(requirements)
# add all mandatory packages
for pkg in self.mandatory_packages:
resolved.add(pkg)
stack.append(pkg.requires)
while len(stack) > 0:
required = stack.pop()
for require in required:
for rel in require:
if rel not in unresolved:
if pkg_filter(rel.name, rel.version) is not None:
break
# use all packages that meets depends
candidates = self.find_all(rel.name, rel.version)
for cand in candidates:
if cand not in resolved:
resolved.add(cand)
stack.append(cand.requires)
if len(candidates) > 0:
break
else:
unresolved.add(require)
msg = "Unresolved depends: {0}".format(require)
warnings.warn(UnresolvedWarning(msg))
return resolved
@staticmethod
def __get_unresolved_dependencies(index, unresolved=None):
"""Gets the set of unresolved dependencies.
:param index: the search index.
:param unresolved: the known list of unresolved packages.
:return: the set of unresolved depends.
"""
if unresolved is None:
unresolved = set()
for pkg in index:
for require in pkg.requires:
for rel in require:
if rel not in unresolved:
candidate = index.find(rel.name, rel.version)
if candidate is not None and candidate != pkg:
break
else:
unresolved.add(require)
return unresolved

View File

@ -1,48 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
class Repository(object):
"""Structure to describe repository object."""
def __init__(self, name, url, architecture, origin):
"""Initialises.
:param name: the repository`s name, may be tuple of strings
:param url: the repository`s URL
:param architecture: the repository`s architecture
:param origin: the repository`s origin
"""
self.name = name
self.url = url
self.architecture = architecture
self.origin = origin
def __str__(self):
if isinstance(self.name, tuple):
return ".".join(self.name)
return self.name or self.url
def __unicode__(self):
if isinstance(self.name, tuple):
return u".".join(self.name)
return self.name or self.url
def __copy__(self):
"""Creates shallow copy of package."""
return Repository(**self.__dict__)

View File

@ -1,47 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import copy
class CopyStatistics(object):
"""The statistics of packages copying"""
def __init__(self):
# the number of copied packages
self.copied = 0
# the number of total packages
self.total = 0
def on_package_copied(self, bytes_copied):
"""Proceed next copied package."""
if bytes_copied > 0:
self.copied += 1
self.total += 1
def __iadd__(self, other):
if not isinstance(other, CopyStatistics):
raise TypeError
self.copied += other.copied
self.total += other.total
return self
def __add__(self, other):
result = copy.copy(self)
result += other
return result

View File

@ -1,27 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Mirantis, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
try:
import unittest2 as unittest
except ImportError:
import unittest
class TestCase(unittest.TestCase):
"""Test case base class for all unit tests."""

View File

@ -1,19 +0,0 @@
Package: test
Source: test.src
Version: 1.1.1-1~u14.04+test
Architecture: all
Maintainer: Test
Installed-Size: 3509
Homepage: http://localhost/
Priority: required
Section: web
Filename: pool/main/t/test.deb
Size: 100
Depends: test2 (>= 0.8.16~exp9)|tes2-old, test3
Pre-Depends: test-main
Provides: file
Replaces: test-old
SHA256: 14d6e308d8699b7f9ba2fe1ef778c0e38cf295614d308039d687b6b097d50859
SHA1: 402bd18c145ae3b5344edf07f246be159397fd40
MD5sum: 1ae09f80109f40dfbfaf3ba423c8625a
Description: test package

View File

@ -1,6 +0,0 @@
Archive: trusty
Version: 14.04
Component: main
Origin: Ubuntu
Label: Ubuntu
Architecture: amd64

Some files were not shown because too many files have changed in this diff Show More