Fuel-mirror project is deprecated
Instead of fuel-mirror it is recommended to use Packetary to create partial repository mirrors and use Python-fuelclient to manage Fuel metadata. Fuel-mirror is going to be deprecated and master branch won't be supported any more. However we still need test jobs to support stable branches. Change-Id: I9fb76f3606c0888dd1c730a40e1bbb3b8c0f02ed
This commit is contained in:
parent
24b17cce5c
commit
afaa4ca6c7
|
@ -1,17 +0,0 @@
|
|||
If you would like to contribute to the development of OpenStack, you must
|
||||
follow the steps in this page:
|
||||
|
||||
http://docs.openstack.org/infra/manual/developers.html
|
||||
|
||||
If you already have a good understanding of how the system works and your
|
||||
OpenStack accounts are set up, you can skip to the development workflow
|
||||
section of this documentation to learn how changes to OpenStack should be
|
||||
submitted for review via the Gerrit tool:
|
||||
|
||||
http://docs.openstack.org/infra/manual/developers.html#development-workflow
|
||||
|
||||
Pull requests submitted through GitHub will be ignored.
|
||||
|
||||
Bugs should be filed on Launchpad, not GitHub:
|
||||
|
||||
https://bugs.launchpad.net/fuel
|
|
@ -1,4 +0,0 @@
|
|||
fuel-mirror Style Commandments
|
||||
==============================
|
||||
|
||||
Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/
|
340
LICENSE
340
LICENSE
|
@ -1,340 +0,0 @@
|
|||
GNU GENERAL PUBLIC LICENSE
|
||||
Version 2, June 1991
|
||||
|
||||
Copyright (C) 1989, 1991 Free Software Foundation, Inc., <http://fsf.org/>
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The licenses for most software are designed to take away your
|
||||
freedom to share and change it. By contrast, the GNU General Public
|
||||
License is intended to guarantee your freedom to share and change free
|
||||
software--to make sure the software is free for all its users. This
|
||||
General Public License applies to most of the Free Software
|
||||
Foundation's software and to any other program whose authors commit to
|
||||
using it. (Some other Free Software Foundation software is covered by
|
||||
the GNU Lesser General Public License instead.) You can apply it to
|
||||
your programs, too.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
this service if you wish), that you receive source code or can get it
|
||||
if you want it, that you can change the software or use pieces of it
|
||||
in new free programs; and that you know you can do these things.
|
||||
|
||||
To protect your rights, we need to make restrictions that forbid
|
||||
anyone to deny you these rights or to ask you to surrender the rights.
|
||||
These restrictions translate to certain responsibilities for you if you
|
||||
distribute copies of the software, or if you modify it.
|
||||
|
||||
For example, if you distribute copies of such a program, whether
|
||||
gratis or for a fee, you must give the recipients all the rights that
|
||||
you have. You must make sure that they, too, receive or can get the
|
||||
source code. And you must show them these terms so they know their
|
||||
rights.
|
||||
|
||||
We protect your rights with two steps: (1) copyright the software, and
|
||||
(2) offer you this license which gives you legal permission to copy,
|
||||
distribute and/or modify the software.
|
||||
|
||||
Also, for each author's protection and ours, we want to make certain
|
||||
that everyone understands that there is no warranty for this free
|
||||
software. If the software is modified by someone else and passed on, we
|
||||
want its recipients to know that what they have is not the original, so
|
||||
that any problems introduced by others will not reflect on the original
|
||||
authors' reputations.
|
||||
|
||||
Finally, any free program is threatened constantly by software
|
||||
patents. We wish to avoid the danger that redistributors of a free
|
||||
program will individually obtain patent licenses, in effect making the
|
||||
program proprietary. To prevent this, we have made it clear that any
|
||||
patent must be licensed for everyone's free use or not licensed at all.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
GNU GENERAL PUBLIC LICENSE
|
||||
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
|
||||
|
||||
0. This License applies to any program or other work which contains
|
||||
a notice placed by the copyright holder saying it may be distributed
|
||||
under the terms of this General Public License. The "Program", below,
|
||||
refers to any such program or work, and a "work based on the Program"
|
||||
means either the Program or any derivative work under copyright law:
|
||||
that is to say, a work containing the Program or a portion of it,
|
||||
either verbatim or with modifications and/or translated into another
|
||||
language. (Hereinafter, translation is included without limitation in
|
||||
the term "modification".) Each licensee is addressed as "you".
|
||||
|
||||
Activities other than copying, distribution and modification are not
|
||||
covered by this License; they are outside its scope. The act of
|
||||
running the Program is not restricted, and the output from the Program
|
||||
is covered only if its contents constitute a work based on the
|
||||
Program (independent of having been made by running the Program).
|
||||
Whether that is true depends on what the Program does.
|
||||
|
||||
1. You may copy and distribute verbatim copies of the Program's
|
||||
source code as you receive it, in any medium, provided that you
|
||||
conspicuously and appropriately publish on each copy an appropriate
|
||||
copyright notice and disclaimer of warranty; keep intact all the
|
||||
notices that refer to this License and to the absence of any warranty;
|
||||
and give any other recipients of the Program a copy of this License
|
||||
along with the Program.
|
||||
|
||||
You may charge a fee for the physical act of transferring a copy, and
|
||||
you may at your option offer warranty protection in exchange for a fee.
|
||||
|
||||
2. You may modify your copy or copies of the Program or any portion
|
||||
of it, thus forming a work based on the Program, and copy and
|
||||
distribute such modifications or work under the terms of Section 1
|
||||
above, provided that you also meet all of these conditions:
|
||||
|
||||
a) You must cause the modified files to carry prominent notices
|
||||
stating that you changed the files and the date of any change.
|
||||
|
||||
b) You must cause any work that you distribute or publish, that in
|
||||
whole or in part contains or is derived from the Program or any
|
||||
part thereof, to be licensed as a whole at no charge to all third
|
||||
parties under the terms of this License.
|
||||
|
||||
c) If the modified program normally reads commands interactively
|
||||
when run, you must cause it, when started running for such
|
||||
interactive use in the most ordinary way, to print or display an
|
||||
announcement including an appropriate copyright notice and a
|
||||
notice that there is no warranty (or else, saying that you provide
|
||||
a warranty) and that users may redistribute the program under
|
||||
these conditions, and telling the user how to view a copy of this
|
||||
License. (Exception: if the Program itself is interactive but
|
||||
does not normally print such an announcement, your work based on
|
||||
the Program is not required to print an announcement.)
|
||||
|
||||
These requirements apply to the modified work as a whole. If
|
||||
identifiable sections of that work are not derived from the Program,
|
||||
and can be reasonably considered independent and separate works in
|
||||
themselves, then this License, and its terms, do not apply to those
|
||||
sections when you distribute them as separate works. But when you
|
||||
distribute the same sections as part of a whole which is a work based
|
||||
on the Program, the distribution of the whole must be on the terms of
|
||||
this License, whose permissions for other licensees extend to the
|
||||
entire whole, and thus to each and every part regardless of who wrote it.
|
||||
|
||||
Thus, it is not the intent of this section to claim rights or contest
|
||||
your rights to work written entirely by you; rather, the intent is to
|
||||
exercise the right to control the distribution of derivative or
|
||||
collective works based on the Program.
|
||||
|
||||
In addition, mere aggregation of another work not based on the Program
|
||||
with the Program (or with a work based on the Program) on a volume of
|
||||
a storage or distribution medium does not bring the other work under
|
||||
the scope of this License.
|
||||
|
||||
3. You may copy and distribute the Program (or a work based on it,
|
||||
under Section 2) in object code or executable form under the terms of
|
||||
Sections 1 and 2 above provided that you also do one of the following:
|
||||
|
||||
a) Accompany it with the complete corresponding machine-readable
|
||||
source code, which must be distributed under the terms of Sections
|
||||
1 and 2 above on a medium customarily used for software interchange; or,
|
||||
|
||||
b) Accompany it with a written offer, valid for at least three
|
||||
years, to give any third party, for a charge no more than your
|
||||
cost of physically performing source distribution, a complete
|
||||
machine-readable copy of the corresponding source code, to be
|
||||
distributed under the terms of Sections 1 and 2 above on a medium
|
||||
customarily used for software interchange; or,
|
||||
|
||||
c) Accompany it with the information you received as to the offer
|
||||
to distribute corresponding source code. (This alternative is
|
||||
allowed only for noncommercial distribution and only if you
|
||||
received the program in object code or executable form with such
|
||||
an offer, in accord with Subsection b above.)
|
||||
|
||||
The source code for a work means the preferred form of the work for
|
||||
making modifications to it. For an executable work, complete source
|
||||
code means all the source code for all modules it contains, plus any
|
||||
associated interface definition files, plus the scripts used to
|
||||
control compilation and installation of the executable. However, as a
|
||||
special exception, the source code distributed need not include
|
||||
anything that is normally distributed (in either source or binary
|
||||
form) with the major components (compiler, kernel, and so on) of the
|
||||
operating system on which the executable runs, unless that component
|
||||
itself accompanies the executable.
|
||||
|
||||
If distribution of executable or object code is made by offering
|
||||
access to copy from a designated place, then offering equivalent
|
||||
access to copy the source code from the same place counts as
|
||||
distribution of the source code, even though third parties are not
|
||||
compelled to copy the source along with the object code.
|
||||
|
||||
4. You may not copy, modify, sublicense, or distribute the Program
|
||||
except as expressly provided under this License. Any attempt
|
||||
otherwise to copy, modify, sublicense or distribute the Program is
|
||||
void, and will automatically terminate your rights under this License.
|
||||
However, parties who have received copies, or rights, from you under
|
||||
this License will not have their licenses terminated so long as such
|
||||
parties remain in full compliance.
|
||||
|
||||
5. You are not required to accept this License, since you have not
|
||||
signed it. However, nothing else grants you permission to modify or
|
||||
distribute the Program or its derivative works. These actions are
|
||||
prohibited by law if you do not accept this License. Therefore, by
|
||||
modifying or distributing the Program (or any work based on the
|
||||
Program), you indicate your acceptance of this License to do so, and
|
||||
all its terms and conditions for copying, distributing or modifying
|
||||
the Program or works based on it.
|
||||
|
||||
6. Each time you redistribute the Program (or any work based on the
|
||||
Program), the recipient automatically receives a license from the
|
||||
original licensor to copy, distribute or modify the Program subject to
|
||||
these terms and conditions. You may not impose any further
|
||||
restrictions on the recipients' exercise of the rights granted herein.
|
||||
You are not responsible for enforcing compliance by third parties to
|
||||
this License.
|
||||
|
||||
7. If, as a consequence of a court judgment or allegation of patent
|
||||
infringement or for any other reason (not limited to patent issues),
|
||||
conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot
|
||||
distribute so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you
|
||||
may not distribute the Program at all. For example, if a patent
|
||||
license would not permit royalty-free redistribution of the Program by
|
||||
all those who receive copies directly or indirectly through you, then
|
||||
the only way you could satisfy both it and this License would be to
|
||||
refrain entirely from distribution of the Program.
|
||||
|
||||
If any portion of this section is held invalid or unenforceable under
|
||||
any particular circumstance, the balance of the section is intended to
|
||||
apply and the section as a whole is intended to apply in other
|
||||
circumstances.
|
||||
|
||||
It is not the purpose of this section to induce you to infringe any
|
||||
patents or other property right claims or to contest validity of any
|
||||
such claims; this section has the sole purpose of protecting the
|
||||
integrity of the free software distribution system, which is
|
||||
implemented by public license practices. Many people have made
|
||||
generous contributions to the wide range of software distributed
|
||||
through that system in reliance on consistent application of that
|
||||
system; it is up to the author/donor to decide if he or she is willing
|
||||
to distribute software through any other system and a licensee cannot
|
||||
impose that choice.
|
||||
|
||||
This section is intended to make thoroughly clear what is believed to
|
||||
be a consequence of the rest of this License.
|
||||
|
||||
8. If the distribution and/or use of the Program is restricted in
|
||||
certain countries either by patents or by copyrighted interfaces, the
|
||||
original copyright holder who places the Program under this License
|
||||
may add an explicit geographical distribution limitation excluding
|
||||
those countries, so that distribution is permitted only in or among
|
||||
countries not thus excluded. In such case, this License incorporates
|
||||
the limitation as if written in the body of this License.
|
||||
|
||||
9. The Free Software Foundation may publish revised and/or new versions
|
||||
of the General Public License from time to time. Such new versions will
|
||||
be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the Program
|
||||
specifies a version number of this License which applies to it and "any
|
||||
later version", you have the option of following the terms and conditions
|
||||
either of that version or of any later version published by the Free
|
||||
Software Foundation. If the Program does not specify a version number of
|
||||
this License, you may choose any version ever published by the Free Software
|
||||
Foundation.
|
||||
|
||||
10. If you wish to incorporate parts of the Program into other free
|
||||
programs whose distribution conditions are different, write to the author
|
||||
to ask for permission. For software which is copyrighted by the Free
|
||||
Software Foundation, write to the Free Software Foundation; we sometimes
|
||||
make exceptions for this. Our decision will be guided by the two goals
|
||||
of preserving the free status of all derivatives of our free software and
|
||||
of promoting the sharing and reuse of software generally.
|
||||
|
||||
NO WARRANTY
|
||||
|
||||
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
|
||||
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
|
||||
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
|
||||
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
|
||||
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
||||
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
|
||||
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
|
||||
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
|
||||
REPAIR OR CORRECTION.
|
||||
|
||||
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
|
||||
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
|
||||
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
|
||||
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
|
||||
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
|
||||
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
|
||||
POSSIBILITY OF SUCH DAMAGES.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
convey the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
{description}
|
||||
Copyright (C) {year} {fullname}
|
||||
|
||||
This program is free software; you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation; either version 2 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License along
|
||||
with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If the program is interactive, make it output a short notice like this
|
||||
when it starts in an interactive mode:
|
||||
|
||||
Gnomovision version 69, Copyright (C) year name of author
|
||||
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
||||
This is free software, and you are welcome to redistribute it
|
||||
under certain conditions; type `show c' for details.
|
||||
|
||||
The hypothetical commands `show w' and `show c' should show the appropriate
|
||||
parts of the General Public License. Of course, the commands you use may
|
||||
be called something other than `show w' and `show c'; they could even be
|
||||
mouse-clicks or menu items--whatever suits your program.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or your
|
||||
school, if any, to sign a "copyright disclaimer" for the program, if
|
||||
necessary. Here is a sample; alter the names:
|
||||
|
||||
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
|
||||
`Gnomovision' (which makes passes at compilers) written by James Hacker.
|
||||
|
||||
{signature of Ty Coon}, 1 April 1989
|
||||
Ty Coon, President of Vice
|
||||
|
||||
This General Public License does not permit incorporating your program into
|
||||
proprietary programs. If your program is a subroutine library, you may
|
||||
consider it more useful to permit linking proprietary applications with the
|
||||
library. If this is what you want to do, use the GNU Lesser General
|
||||
Public License instead of this License.
|
||||
|
45
MAINTAINERS
45
MAINTAINERS
|
@ -26,49 +26,6 @@ description:
|
|||
|
||||
maintainers:
|
||||
|
||||
- contrib/:
|
||||
- name: Bulat Gaifullin
|
||||
email: bgaifullin@mirantis.com
|
||||
IRC: bgaifullin
|
||||
|
||||
- name: Vladimir Kozhukalov
|
||||
email: vkozhukalov@mirantis.com
|
||||
IRC: kozhukalov
|
||||
|
||||
- debian/: &packaging_team
|
||||
- name: Mikhail Ivanov
|
||||
email: mivanov@mirantis.com
|
||||
IRC: mivanov
|
||||
|
||||
- name: Artem Silenkov
|
||||
email: asilenkov@mirantis.com
|
||||
IRC: asilenkov
|
||||
|
||||
- name: Alexander Tsamutali
|
||||
email: atsamutali@mirantis.com
|
||||
IRC: astsmtl
|
||||
|
||||
- name: Daniil Trishkin
|
||||
email: dtrishkin@mirantis.com
|
||||
IRC: dtrishkin
|
||||
|
||||
- name: Ivan Udovichenko
|
||||
email: iudovichenko@mirantis.com
|
||||
IRC: tlbr
|
||||
|
||||
- name: Igor Yozhikov
|
||||
email: iyozhikov@mirantis.com
|
||||
IRC: IgorYozhikov
|
||||
|
||||
- packetary/:
|
||||
- name: Bulat Gaifullin
|
||||
email: bgaifullin@mirantis.com
|
||||
IRC: bgaifullin
|
||||
|
||||
- name: Vladimir Kozhukalov
|
||||
email: vkozhukalov@mirantis.com
|
||||
IRC: kozhukalov
|
||||
|
||||
- perestroika/: &build_team
|
||||
- name: Dmitry Burmistrov
|
||||
email: dburmistrov@mirantis.com
|
||||
|
@ -77,5 +34,3 @@ maintainers:
|
|||
- name: Sergey Kulanov
|
||||
email: skulanov@mirantis.com
|
||||
IRC: SergK
|
||||
|
||||
- specs/: *packaging_team
|
||||
|
|
|
@ -1,5 +0,0 @@
|
|||
include AUTHORS
|
||||
include ChangeLog
|
||||
include *requirements.txt
|
||||
|
||||
global-exclude *.pyc
|
32
README.rst
32
README.rst
|
@ -2,38 +2,6 @@
|
|||
Repository structure
|
||||
====================
|
||||
|
||||
* contrib/fuel_mirror
|
||||
It is a command line utility that provides the same functionality
|
||||
and user interface as deprecated fuel-createmirror. It provides
|
||||
two major features:
|
||||
* clone/build mirror (full or partial)
|
||||
* update repository configuration in nailgun
|
||||
First one is a matter of packetary while second one should be left
|
||||
totally up to fuelclient. So this module is to be deprecated soon
|
||||
in favor of packetary and fuelclient.
|
||||
|
||||
WARNING: It is not designed to be used on 'live' repositories
|
||||
that are available to clients during synchronization. That means
|
||||
repositories will be inconsistent during the update. Please use these
|
||||
scripts in conjunction with snapshots, on inactive repos, etc.
|
||||
|
||||
* debian
|
||||
Specs for DEB packages.
|
||||
|
||||
* doc
|
||||
Documentation for packetary module.
|
||||
|
||||
* packetary
|
||||
It is a Python library and command line utilty that allows
|
||||
one to clone and build rpm/deb repositories.
|
||||
Features:
|
||||
* Common interface for different package-managers.
|
||||
* Utility to build dependency graph for package(s).
|
||||
* Utility to create mirror of repository according to dependency graph.
|
||||
|
||||
* perestroika
|
||||
It is a set shell/python script that are used to build DEB/RPM
|
||||
packages. These scripts are widely used by Fuel Packaging CI.
|
||||
|
||||
* specs
|
||||
Specs for RPM packages.
|
||||
|
|
|
@ -1,8 +0,0 @@
|
|||
include AUTHORS
|
||||
include ChangeLog
|
||||
recursive-include etc *
|
||||
|
||||
exclude .gitignore
|
||||
exclude .gitreview
|
||||
|
||||
global-exclude *.pyc
|
|
@ -1,16 +0,0 @@
|
|||
===========
|
||||
fuel_mirror
|
||||
===========
|
||||
|
||||
The fuel-mirror is utility, that allows to create local repositories
|
||||
with packages are required for the OpenStack deployment.
|
||||
|
||||
* Free software: Apache license
|
||||
* Documentation: http://docs.openstack.org/developer/fuel-mirror
|
||||
* Source: http://git.openstack.org/cgit/openstack/fuel-mirror/
|
||||
* Bugs: http://bugs.launchpad.net/fuel
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
* TODO
|
|
@ -1,2 +0,0 @@
|
|||
[python: **.py]
|
||||
|
|
@ -1,55 +0,0 @@
|
|||
fuel_release_match:
|
||||
version: $openstack_version
|
||||
operating_system: CentOS
|
||||
|
||||
repos:
|
||||
- ¢os
|
||||
name: "centos"
|
||||
uri: "http://vault.centos.org/7.1.1503/os/x86_64/"
|
||||
type: "rpm"
|
||||
priority: null
|
||||
|
||||
- ¢os_updates
|
||||
name: "centos-updates"
|
||||
uri: "http://vault.centos.org/7.1.1503/updates/x86_64/"
|
||||
type: "rpm"
|
||||
priority: null
|
||||
|
||||
- &mos
|
||||
name: "mos"
|
||||
uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/os/x86_64/"
|
||||
type: "rpm"
|
||||
priority: null
|
||||
|
||||
- &mos_updates
|
||||
name: "mos-updates"
|
||||
uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/updates/x86_64/"
|
||||
type: "rpm"
|
||||
priority: null
|
||||
|
||||
- &mos_security
|
||||
name: "mos-security"
|
||||
uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/security/x86_64/"
|
||||
type: "rpm"
|
||||
priority: null
|
||||
|
||||
- &mos_holdback
|
||||
name: "mos-holdback"
|
||||
uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/holdback/x86_64/"
|
||||
type: "rpm"
|
||||
priority: null
|
||||
|
||||
groups:
|
||||
mos:
|
||||
- *mos
|
||||
- *mos_updates
|
||||
- *mos_security
|
||||
- *mos_holdback
|
||||
|
||||
centos:
|
||||
- *centos
|
||||
- *centos_updates
|
||||
|
||||
|
||||
inheritance:
|
||||
centos: mos
|
|
@ -1,158 +0,0 @@
|
|||
# GLOBAL variables
|
||||
ubuntu_baseurl: &ubuntu_baseurl http://archive.ubuntu.com/ubuntu
|
||||
mos_baseurl: &mos_baseurl http://mirror.fuel-infra.org/mos-repos/ubuntu/$mos_version
|
||||
|
||||
fuel_release_match:
|
||||
version: $openstack_version
|
||||
operating_system: Ubuntu
|
||||
|
||||
# Main is a required parameter which defines what repository will be used
|
||||
# for images creation and that mirror should contain all packages for minimal
|
||||
# system creation.
|
||||
repos:
|
||||
- &ubuntu
|
||||
main: true
|
||||
name: "ubuntu"
|
||||
uri: *ubuntu_baseurl
|
||||
suite: "trusty"
|
||||
section: "main multiverse restricted universe"
|
||||
type: "deb"
|
||||
priority: null
|
||||
|
||||
- &ubuntu_updates
|
||||
name: "ubuntu-updates"
|
||||
uri: *ubuntu_baseurl
|
||||
suite: "trusty-updates"
|
||||
section: "main multiverse restricted universe"
|
||||
type: "deb"
|
||||
priority: null
|
||||
|
||||
- &ubuntu_security
|
||||
name: "ubuntu-security"
|
||||
uri: *ubuntu_baseurl
|
||||
suite: "trusty-security"
|
||||
section: "main multiverse restricted universe"
|
||||
type: "deb"
|
||||
priority: null
|
||||
|
||||
- &mos
|
||||
name: "mos"
|
||||
uri: *mos_baseurl
|
||||
suite: "mos$mos_version"
|
||||
section: "main restricted"
|
||||
type: "deb"
|
||||
priority: 1000
|
||||
|
||||
- &mos_updates
|
||||
name: "mos-updates"
|
||||
uri: *mos_baseurl
|
||||
suite: "mos$mos_version-updates"
|
||||
section: "main restricted"
|
||||
type: "deb"
|
||||
priority: 1000
|
||||
|
||||
- &mos_security
|
||||
name: "mos-security"
|
||||
uri: *mos_baseurl
|
||||
suite: "mos$mos_version-security"
|
||||
section: "main restricted"
|
||||
type: "deb"
|
||||
priority: 1000
|
||||
|
||||
- &mos_holdback
|
||||
name: "mos-holdback"
|
||||
uri: *mos_baseurl
|
||||
suite: "mos$mos_version-holdback"
|
||||
section: "main restricted"
|
||||
type: "deb"
|
||||
priority: 1000
|
||||
|
||||
# Packages are required to build bootstrap images for a system.
|
||||
# The mirror should contiain such packages in addition to local mirror.
|
||||
packages: &packages
|
||||
- "acpi-support"
|
||||
- "anacron"
|
||||
- "aptitude"
|
||||
- "atop"
|
||||
- "acct"
|
||||
- "bash-completion"
|
||||
- "bc"
|
||||
- "build-essential"
|
||||
- "cloud-init"
|
||||
- "conntrackd"
|
||||
- "cpu-checker"
|
||||
- "cpufrequtils"
|
||||
- "debconf-utils"
|
||||
- "devscripts"
|
||||
- "fping"
|
||||
- "git"
|
||||
- "grub-pc"
|
||||
- "htop"
|
||||
- "hwloc"
|
||||
- "ifenslave"
|
||||
- "iperf"
|
||||
- "iptables-persistent"
|
||||
- "irqbalance"
|
||||
- "language-pack-en"
|
||||
- "libapache2-mod-fastcgi"
|
||||
- "libnss3-tools"
|
||||
- "linux-firmware-nonfree"
|
||||
- "linux-headers-generic-lts-trusty"
|
||||
- "linux-image-generic-lts-trusty"
|
||||
- "live-boot"
|
||||
- "livecd-rootfs"
|
||||
- "mc"
|
||||
- "memcached"
|
||||
- "monit"
|
||||
- "msmtp-mta"
|
||||
- "multipath-tools"
|
||||
- "multipath-tools-boot"
|
||||
- "nginx"
|
||||
- "ntp"
|
||||
- "openssh-server"
|
||||
- "percona-toolkit"
|
||||
- "percona-xtrabackup"
|
||||
- "pm-utils"
|
||||
- "puppet"
|
||||
- "python-lesscpy"
|
||||
- "python-pip"
|
||||
- "rsyslog-gnutls"
|
||||
- "rsyslog-relp"
|
||||
- "screen"
|
||||
- "squashfs-tools"
|
||||
- "swift-plugin-s3"
|
||||
- "sysfsutils"
|
||||
- "sysstat"
|
||||
- "telnet"
|
||||
- "tmux"
|
||||
- "traceroute"
|
||||
- "ubuntu-standard"
|
||||
- "vim"
|
||||
- "virt-what"
|
||||
- "xinetd"
|
||||
- "xmlstarlet"
|
||||
- "tftpd-hpa"
|
||||
- "syslinux"
|
||||
|
||||
|
||||
groups:
|
||||
mos:
|
||||
- *mos
|
||||
- *mos_updates
|
||||
- *mos_security
|
||||
- *mos_holdback
|
||||
|
||||
ubuntu:
|
||||
- *ubuntu
|
||||
- *ubuntu_updates
|
||||
- *ubuntu_security
|
||||
|
||||
|
||||
inheritance:
|
||||
ubuntu: mos
|
||||
|
||||
osnames:
|
||||
mos: ubuntu
|
||||
|
||||
requirements:
|
||||
ubuntu: *packages
|
|
@ -1,79 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.abspath('../..'))
|
||||
# -- General configuration ----------------------------------------------------
|
||||
|
||||
# Add any Sphinx extension module names here, as strings. They can be
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
#'sphinx.ext.intersphinx',
|
||||
'oslosphinx'
|
||||
]
|
||||
|
||||
# autodoc generation is a bit aggressive and a nuisance when doing heavy
|
||||
# text edit cycles.
|
||||
# execute "export SPHINX_DEBUG=1" in your terminal to disable
|
||||
|
||||
# The suffix of source filenames.
|
||||
source_suffix = '.rst'
|
||||
|
||||
# The master toctree document.
|
||||
master_doc = 'index'
|
||||
|
||||
# General information about the project.
|
||||
project = u'fuel_mirror'
|
||||
copyright = u'2015, Mirantis, Inc'
|
||||
|
||||
# If true, '()' will be appended to :func: etc. cross-reference text.
|
||||
add_function_parentheses = True
|
||||
|
||||
# If true, the current module name will be prepended to all description
|
||||
# unit titles (such as .. function::).
|
||||
add_module_names = True
|
||||
|
||||
# The name of the Pygments (syntax highlighting) style to use.
|
||||
pygments_style = 'sphinx'
|
||||
|
||||
# -- Options for HTML output --------------------------------------------------
|
||||
|
||||
# The theme to use for HTML and HTML Help pages. Major themes that come with
|
||||
# Sphinx are currently 'default' and 'sphinxdoc'.
|
||||
# html_theme_path = ["."]
|
||||
# html_theme = '_theme'
|
||||
# html_static_path = ['static']
|
||||
|
||||
# Output file base name for HTML help builder.
|
||||
htmlhelp_basename = '%sdoc' % project
|
||||
|
||||
# Grouping the document tree into LaTeX files. List of tuples
|
||||
# (source start file, target name, title, author, documentclass
|
||||
# [howto/manual]).
|
||||
latex_documents = [
|
||||
('index',
|
||||
'%s.tex' % project,
|
||||
u'%s Documentation' % project,
|
||||
u'OpenStack Foundation', 'manual'),
|
||||
]
|
||||
|
||||
# Example configuration for intersphinx: refer to the Python standard library.
|
||||
#intersphinx_mapping = {'http://docs.python.org/': None}
|
|
@ -1,4 +0,0 @@
|
|||
============
|
||||
Contributing
|
||||
============
|
||||
.. include:: ../../../../CONTRIBUTING.rst
|
|
@ -1,25 +0,0 @@
|
|||
.. fuel_mirror documentation master file, created by
|
||||
sphinx-quickstart on Tue Jul 9 22:26:36 2013.
|
||||
You can adapt this file completely to your liking, but it should at least
|
||||
contain the root `toctree` directive.
|
||||
|
||||
Welcome to fuel_mirror's documentation!
|
||||
========================================================
|
||||
|
||||
Contents:
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
readme
|
||||
installation
|
||||
usage
|
||||
contributing
|
||||
|
||||
Indices and tables
|
||||
==================
|
||||
|
||||
* :ref:`genindex`
|
||||
* :ref:`modindex`
|
||||
* :ref:`search`
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
============
|
||||
Installation
|
||||
============
|
||||
|
||||
At the command line::
|
||||
|
||||
$ pip install fuel_mirror
|
||||
|
||||
Or, if you have virtualenvwrapper installed::
|
||||
|
||||
$ mkvirtualenv fuel_mirror
|
||||
$ pip install fuel_mirror
|
|
@ -1 +0,0 @@
|
|||
.. include:: ../../README.rst
|
|
@ -1,7 +0,0 @@
|
|||
========
|
||||
Usage
|
||||
========
|
||||
|
||||
To use fuel_mirror in a project::
|
||||
|
||||
import fuel_createmirror
|
|
@ -1,11 +0,0 @@
|
|||
threads_num: 10
|
||||
ignore_errors_num: 2
|
||||
retries_num: 3
|
||||
target_dir: "/var/www/nailgun/mirrors"
|
||||
pattern_dir: "/usr/share/fuel-mirror"
|
||||
base_url: "http://{FUEL_SERVER_IP}:8080/mirrors/"
|
||||
|
||||
# uncomment if need
|
||||
# http_proxy: null
|
||||
# https_proxy: null
|
||||
# fuel_server: 10.20.0.2
|
|
@ -1,26 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import pbr.version
|
||||
|
||||
try:
|
||||
__version__ = pbr.version.VersionInfo(
|
||||
'fuel_mirror').version_string()
|
||||
except Exception as e:
|
||||
print("ERROR", e)
|
||||
__version__ = "0.0.0-test"
|
|
@ -1,157 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from cliff import app
|
||||
from cliff.commandmanager import CommandManager
|
||||
import yaml
|
||||
|
||||
|
||||
import fuel_mirror
|
||||
from fuel_mirror.common import accessors
|
||||
from fuel_mirror.common import utils
|
||||
|
||||
|
||||
class Application(app.App):
|
||||
"""Main cliff application class.
|
||||
|
||||
Performs initialization of the command manager and
|
||||
configuration of basic engines.
|
||||
"""
|
||||
|
||||
config = None
|
||||
fuel = None
|
||||
repo_manager_accessor = None
|
||||
sources = None
|
||||
versions = None
|
||||
|
||||
def build_option_parser(self, description, version, argparse_kwargs=None):
|
||||
"""Specifies common cmdline arguments."""
|
||||
p_inst = super(Application, self)
|
||||
parser = p_inst.build_option_parser(description=description,
|
||||
version=version,
|
||||
argparse_kwargs=argparse_kwargs)
|
||||
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
default="/etc/fuel-mirror/config.yaml",
|
||||
metavar="PATH",
|
||||
help="Path to config file."
|
||||
)
|
||||
parser.add_argument(
|
||||
"-S", "--fuel-server",
|
||||
metavar="FUEL-SERVER",
|
||||
help="The public address of Fuel Master."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--fuel-user",
|
||||
help="Fuel Master admin login."
|
||||
" Alternatively, use env var KEYSTONE_USER)."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--fuel-password",
|
||||
help="Fuel Master admin password."
|
||||
" Alternatively, use env var KEYSTONE_PASSWORD)."
|
||||
)
|
||||
return parser
|
||||
|
||||
def initialize_app(self, argv):
|
||||
"""Initialises common options."""
|
||||
with open(self.options.config, "r") as stream:
|
||||
self.config = yaml.load(stream)
|
||||
|
||||
self._initialize_fuel_accessor()
|
||||
self._initialize_repo_manager()
|
||||
|
||||
def _initialize_repo_manager(self):
|
||||
self.repo_manager_accessor = accessors.get_packetary_accessor(
|
||||
threads_num=int(self.config.get('threads_num', 0)),
|
||||
retries_num=int(self.config.get('retries_num', 0)),
|
||||
ignore_errors_num=int(self.config.get('ignore_errors_num', 0)),
|
||||
http_proxy=self.config.get('http_proxy'),
|
||||
https_proxy=self.config.get('https_proxy'),
|
||||
)
|
||||
|
||||
def _initialize_fuel_accessor(self):
|
||||
fuel_default = utils.get_fuel_settings()
|
||||
|
||||
fuel_server = utils.first(
|
||||
self.options.fuel_server,
|
||||
self.config.get("fuel_server"),
|
||||
fuel_default.get("server")
|
||||
)
|
||||
fuel_user = utils.first(
|
||||
self.options.fuel_user,
|
||||
fuel_default.get("user")
|
||||
)
|
||||
fuel_password = utils.first(
|
||||
self.options.fuel_password,
|
||||
fuel_default.get("password")
|
||||
)
|
||||
|
||||
if not fuel_server:
|
||||
for option in ("mos_version", "openstack_version"):
|
||||
if not self.config.setdefault(option, ''):
|
||||
self.LOG.warning(
|
||||
"The option '{0}' is not defined."
|
||||
"Please specify the option 'fuel-server' or '{0}'."
|
||||
.format(option)
|
||||
)
|
||||
return
|
||||
|
||||
self.config["base_url"] = self.config["base_url"].format(
|
||||
FUEL_SERVER_IP=fuel_server.split(":", 1)[0]
|
||||
)
|
||||
|
||||
self.fuel = accessors.get_fuel_api_accessor(
|
||||
fuel_server,
|
||||
fuel_user,
|
||||
fuel_password
|
||||
)
|
||||
fuel_ver = self.fuel.FuelVersion.get_all_data()
|
||||
self.config.setdefault(
|
||||
'mos_version', fuel_ver['release']
|
||||
)
|
||||
self.config.setdefault(
|
||||
'openstack_version', fuel_ver['openstack_version']
|
||||
)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
"""Entry point."""
|
||||
return Application(
|
||||
description="The utility to create local mirrors.",
|
||||
version=fuel_mirror.__version__,
|
||||
command_manager=CommandManager("fuel_mirror", convert_underscores=True)
|
||||
).run(argv)
|
||||
|
||||
|
||||
def debug(name, cmd_class, argv=None):
|
||||
"""Helps to debug command."""
|
||||
import sys
|
||||
|
||||
if argv is None:
|
||||
argv = sys.argv[1:]
|
||||
|
||||
argv = [name] + argv + ["-v", "-v", "--debug"]
|
||||
cmd_mgr = CommandManager("test_fuel_mirror", convert_underscores=True)
|
||||
cmd_mgr.add_command(name, cmd_class)
|
||||
return Application(
|
||||
description="The fuel mirror utility test.",
|
||||
version="0.0.1",
|
||||
command_manager=cmd_mgr
|
||||
).run(argv)
|
|
@ -1,203 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import six
|
||||
|
||||
from packetary.library.utils import localize_repo_url
|
||||
|
||||
from fuel_mirror.commands.base import BaseCommand
|
||||
from fuel_mirror.common.utils import is_subdict
|
||||
from fuel_mirror.common.utils import lists_merge
|
||||
|
||||
|
||||
class ApplyCommand(BaseCommand):
|
||||
"""Applies local mirrors for Fuel-environments."""
|
||||
|
||||
def get_parser(self, prog_name):
|
||||
parser = super(ApplyCommand, self).get_parser(prog_name)
|
||||
parser.add_argument(
|
||||
"--default",
|
||||
dest="set_default",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Set as default repository."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--replace",
|
||||
dest="replace",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Replace default repos with generated mirrors."
|
||||
)
|
||||
parser.add_argument(
|
||||
"-e", "--env",
|
||||
dest="env", nargs="+",
|
||||
help="Fuel environment ID to update, "
|
||||
"by default applies for all environments."
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
def take_action(self, parsed_args):
|
||||
if self.app.fuel is None:
|
||||
raise ValueError("Please specify the fuel-server option.")
|
||||
|
||||
data = self.load_data(parsed_args)
|
||||
base_url = self.app.config["base_url"]
|
||||
release_match = data["fuel_release_match"]
|
||||
replace_repos = parsed_args.replace
|
||||
|
||||
localized_repos = []
|
||||
for _, repos in self.get_groups(parsed_args, data):
|
||||
for repo_data in repos:
|
||||
new_data = repo_data.copy()
|
||||
new_data['uri'] = localize_repo_url(
|
||||
base_url, repo_data['uri']
|
||||
)
|
||||
localized_repos.append(new_data)
|
||||
|
||||
localized_repos.sort(key=lambda x: not x.pop('main', False))
|
||||
|
||||
self.update_clusters(
|
||||
parsed_args.env,
|
||||
localized_repos,
|
||||
release_match,
|
||||
replace_repos=replace_repos)
|
||||
|
||||
if parsed_args.set_default:
|
||||
self.update_release_repos(
|
||||
localized_repos,
|
||||
release_match,
|
||||
replace_repos=replace_repos)
|
||||
|
||||
self.app.stdout.write(
|
||||
"Operations have been completed successfully.\n"
|
||||
)
|
||||
|
||||
def update_clusters(self,
|
||||
ids,
|
||||
repositories,
|
||||
release_match,
|
||||
replace_repos=False):
|
||||
"""Applies repositories for existing clusters.
|
||||
|
||||
:param ids: the cluster ids.
|
||||
:param repositories: the meta information of repositories
|
||||
:param release_match: The pattern to check Fuel Release
|
||||
"""
|
||||
self.app.stdout.write("Updating the Cluster repositories...\n")
|
||||
|
||||
if ids:
|
||||
clusters = self.app.fuel.Environment.get_by_ids(ids)
|
||||
else:
|
||||
clusters = self.app.fuel.Environment.get_all()
|
||||
|
||||
for cluster in clusters:
|
||||
releases = six.moves.filter(
|
||||
lambda x: is_subdict(release_match, x.data),
|
||||
self.app.fuel.Release.get_by_ids([cluster.data["release_id"]])
|
||||
)
|
||||
if next(releases, None) is None:
|
||||
continue
|
||||
|
||||
modified = self._update_repository_settings(
|
||||
cluster.get_settings_data(),
|
||||
repositories,
|
||||
replace_repos=replace_repos)
|
||||
|
||||
if modified:
|
||||
self.app.LOG.info(
|
||||
"Try to update the Cluster '%s'",
|
||||
cluster.data['name']
|
||||
)
|
||||
self.app.LOG.debug(
|
||||
"The modified cluster attributes: %s",
|
||||
modified
|
||||
)
|
||||
cluster.set_settings_data(modified)
|
||||
|
||||
def update_release_repos(self,
|
||||
repositories,
|
||||
release_match,
|
||||
replace_repos=False):
|
||||
"""Applies repositories for existing default settings.
|
||||
|
||||
:param repositories: the meta information of repositories
|
||||
:param release_match: The pattern to check Fuel Release
|
||||
"""
|
||||
self.app.stdout.write("Updating the release repositories...\n")
|
||||
releases = six.moves.filter(
|
||||
lambda x: is_subdict(release_match, x.data),
|
||||
self.app.fuel.Release.get_all()
|
||||
)
|
||||
for release in releases:
|
||||
modified = self._update_repository_settings(
|
||||
release.data["attributes_metadata"],
|
||||
repositories,
|
||||
replace_repos=replace_repos)
|
||||
if modified:
|
||||
release.data["attributes_metadata"] = modified
|
||||
self.app.LOG.info(
|
||||
"Try to update the Release '%s'",
|
||||
release.data['name']
|
||||
)
|
||||
self.app.LOG.debug(
|
||||
"The modified release attributes: %s",
|
||||
release.data
|
||||
)
|
||||
# TODO(need to add method for release object)
|
||||
release.connection.put_request(
|
||||
release.instance_api_path.format(release.id),
|
||||
release.data
|
||||
)
|
||||
|
||||
def _update_repository_settings(self,
|
||||
settings,
|
||||
repositories,
|
||||
replace_repos=False):
|
||||
"""Updates repository settings.
|
||||
|
||||
:param settings: the target settings
|
||||
:param repositories: the meta of repositories
|
||||
"""
|
||||
editable = settings["editable"]
|
||||
if 'repo_setup' not in editable:
|
||||
self.app.LOG.info('Attributes are read-only.')
|
||||
return
|
||||
|
||||
repos_attr = editable["repo_setup"]["repos"]
|
||||
if replace_repos:
|
||||
repos_attr['value'] = repositories
|
||||
else:
|
||||
lists_merge(repos_attr['value'], repositories, "name")
|
||||
|
||||
# NOTE(akostrikov) That assignment is only for informational purpose.
|
||||
settings["editable"]["repo_setup"]["repos"] = repos_attr
|
||||
|
||||
return settings
|
||||
|
||||
|
||||
def debug(argv=None):
|
||||
"""Helper for debugging Apply command."""
|
||||
from fuel_mirror.app import debug
|
||||
|
||||
return debug("apply", ApplyCommand, argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
debug()
|
|
@ -1,125 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import os.path
|
||||
|
||||
from cliff import command
|
||||
from jsonschema import validate
|
||||
from jsonschema import ValidationError
|
||||
import six
|
||||
|
||||
from fuel_mirror.common.utils import load_input_data
|
||||
from fuel_mirror.schemas.input_data_schema import SCHEMA
|
||||
|
||||
|
||||
class BaseCommand(command.Command):
|
||||
|
||||
"""The Base command for fuel-mirror."""
|
||||
REPO_ARCH = "x86_64"
|
||||
|
||||
@property
|
||||
def stdout(self):
|
||||
"""Shortcut for self.app.stdout."""
|
||||
return self.app.stdout
|
||||
|
||||
def get_parser(self, prog_name):
|
||||
"""Specifies common options."""
|
||||
parser = super(BaseCommand, self).get_parser(prog_name)
|
||||
|
||||
input_group = parser.add_mutually_exclusive_group(required=True)
|
||||
input_group.add_argument(
|
||||
'-I', '--input-file',
|
||||
metavar='PATH',
|
||||
help='The path to file with input data.')
|
||||
|
||||
input_group.add_argument(
|
||||
'-P', '--pattern',
|
||||
metavar='NAME',
|
||||
help='The builtin input file name.'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-G", "--group",
|
||||
dest="groups",
|
||||
required=True,
|
||||
nargs='+',
|
||||
help="The name of repository groups."
|
||||
)
|
||||
return parser
|
||||
|
||||
def resolve_input_pattern(self, pattern):
|
||||
"""Gets the full path to input file by pattern.
|
||||
|
||||
:param pattern: the config file name without ext
|
||||
:return: the full path
|
||||
"""
|
||||
return os.path.join(
|
||||
self.app.config['pattern_dir'], pattern + ".yaml"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def validate_data(data, schema):
|
||||
"""Validate the input data using jsonschema validation.
|
||||
|
||||
:param data: a data to validate represented as a dict
|
||||
:param schema: a schema to validate represented as a dict;
|
||||
must be in JSON Schema Draft 4 format.
|
||||
"""
|
||||
try:
|
||||
validate(data, schema)
|
||||
except ValidationError as ex:
|
||||
if len(ex.path) > 0:
|
||||
join_ex_path = '.'.join(six.text_type(x) for x in ex.path)
|
||||
detail = ("Invalid input for field/attribute {0}."
|
||||
" Value: {1}. {2}").format(join_ex_path,
|
||||
ex.instance, ex.message)
|
||||
else:
|
||||
detail = ex.message
|
||||
raise ValidationError(detail)
|
||||
|
||||
def load_data(self, parsed_args):
|
||||
"""Load the input data.
|
||||
|
||||
:param parsed_args: the command-line arguments
|
||||
:return: the input data
|
||||
"""
|
||||
if parsed_args.pattern:
|
||||
input_file = self.resolve_input_pattern(parsed_args.pattern)
|
||||
else:
|
||||
input_file = parsed_args.input_file
|
||||
|
||||
data = load_input_data(
|
||||
input_file,
|
||||
mos_version=self.app.config["mos_version"],
|
||||
openstack_version=self.app.config["openstack_version"]
|
||||
)
|
||||
self.validate_data(data, SCHEMA)
|
||||
return data
|
||||
|
||||
@classmethod
|
||||
def get_groups(cls, parsed_args, data):
|
||||
"""Gets repository groups from input data.
|
||||
|
||||
:param parsed_args: the command-line arguments
|
||||
:param data: the input data
|
||||
:return: the sequence of pairs (group_name, repositories)
|
||||
"""
|
||||
all_groups = data['groups']
|
||||
return (
|
||||
(x, all_groups[x]) for x in parsed_args.groups if x in all_groups
|
||||
)
|
|
@ -1,79 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from fuel_mirror.commands.base import BaseCommand
|
||||
from fuel_mirror.common.url_builder import get_url_builder
|
||||
|
||||
|
||||
class CreateCommand(BaseCommand):
|
||||
"""Creates a new local mirrors."""
|
||||
|
||||
def take_action(self, parsed_args):
|
||||
"""See the Command.take_action."""
|
||||
data = self.load_data(parsed_args)
|
||||
repos_reqs = data.get('requirements', {})
|
||||
inheritance = data.get('inheritance', {})
|
||||
target_dir = self.app.config["target_dir"]
|
||||
|
||||
total_stats = None
|
||||
for group_name, repos in self.get_groups(parsed_args, data):
|
||||
url_builder = get_url_builder(repos[0]["type"])
|
||||
repo_manager = self.app.repo_manager_accessor(
|
||||
repos[0]["type"], self.REPO_ARCH
|
||||
)
|
||||
if group_name in inheritance:
|
||||
child_group = inheritance[group_name]
|
||||
dependencies = [
|
||||
url_builder.get_repo_url(x)
|
||||
for x in data['groups'][child_group]
|
||||
]
|
||||
else:
|
||||
dependencies = None
|
||||
|
||||
stat = repo_manager.clone_repositories(
|
||||
[url_builder.get_repo_url(x) for x in repos],
|
||||
target_dir,
|
||||
dependencies,
|
||||
repos_reqs.get(group_name)
|
||||
)
|
||||
|
||||
if total_stats is None:
|
||||
total_stats = stat
|
||||
else:
|
||||
total_stats += stat
|
||||
|
||||
if total_stats is not None:
|
||||
self.stdout.write(
|
||||
"Packages processed: {0.copied}/{0.total}\n"
|
||||
.format(total_stats)
|
||||
)
|
||||
else:
|
||||
self.stdout.write(
|
||||
"No packages.\n"
|
||||
)
|
||||
|
||||
|
||||
def debug(argv=None):
|
||||
"""Helper for debugging Create command."""
|
||||
from fuel_mirror.app import debug
|
||||
|
||||
return debug("create", CreateCommand, argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
debug()
|
|
@ -1,64 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import functools
|
||||
import os
|
||||
|
||||
|
||||
def get_packetary_accessor(**kwargs):
|
||||
"""Gets the configured repository manager.
|
||||
|
||||
:param kwargs: The packetary configuration parameters.
|
||||
"""
|
||||
|
||||
import packetary
|
||||
|
||||
return functools.partial(
|
||||
packetary.RepositoryApi.create,
|
||||
packetary.Context(packetary.Configuration(**kwargs))
|
||||
)
|
||||
|
||||
|
||||
def get_fuel_api_accessor(address=None, user=None, password=None):
|
||||
"""Gets the fuel client api accessor.
|
||||
|
||||
:param address: The address of Fuel Master node.
|
||||
:param user: The username to access to the Fuel Master node.
|
||||
:param user: The password to access to the Fuel Master node.
|
||||
"""
|
||||
if address:
|
||||
host_and_port = address.split(":")
|
||||
os.environ["SERVER_ADDRESS"] = host_and_port[0]
|
||||
if len(host_and_port) > 1:
|
||||
os.environ["LISTEN_PORT"] = host_and_port[1]
|
||||
|
||||
if user is not None:
|
||||
os.environ["KEYSTONE_USER"] = user
|
||||
if password is not None:
|
||||
os.environ["KEYSTONE_PASS"] = password
|
||||
|
||||
# import fuelclient.ClientAPI after configuring
|
||||
# environment variables
|
||||
try:
|
||||
from fuelclient import objects
|
||||
except ImportError:
|
||||
raise RuntimeError(
|
||||
"fuelclient module seems not installed. "
|
||||
"This action requires it to be available."
|
||||
)
|
||||
return objects
|
|
@ -1,59 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
|
||||
def get_url_builder(repotype):
|
||||
"""Gets the instance of RepoUrlBuilder.
|
||||
|
||||
:param repotype: the type of repository: rpm|deb
|
||||
:return: the RepoBuilder implementation
|
||||
"""
|
||||
return {
|
||||
"deb": AptRepoUrlBuilder,
|
||||
"rpm": YumRepoUrlBuilder
|
||||
}[repotype]
|
||||
|
||||
|
||||
class RepoUrlBuilder(object):
|
||||
REPO_FOLDER = "mirror"
|
||||
|
||||
@classmethod
|
||||
def get_repo_url(cls, repo_data):
|
||||
"""Gets the url with replaced variable holders.
|
||||
|
||||
:param repo_data: the repositories`s meta data
|
||||
:return: the full repository`s url
|
||||
"""
|
||||
|
||||
|
||||
class AptRepoUrlBuilder(RepoUrlBuilder):
|
||||
"""URL builder for apt-repository(es)."""
|
||||
|
||||
@classmethod
|
||||
def get_repo_url(cls, repo_data):
|
||||
return " ".join(
|
||||
repo_data[x] for x in ("uri", "suite", "section")
|
||||
)
|
||||
|
||||
|
||||
class YumRepoUrlBuilder(RepoUrlBuilder):
|
||||
"""URL builder for Yum repository(es)."""
|
||||
|
||||
@classmethod
|
||||
def get_repo_url(cls, repo_data):
|
||||
return repo_data["uri"]
|
|
@ -1,106 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
from string import Template
|
||||
|
||||
import six
|
||||
import yaml
|
||||
|
||||
|
||||
def lists_merge(main, patch, key):
|
||||
"""Merges the list of dicts with same keys.
|
||||
|
||||
>>> lists_merge([{"a": 1, "c": 2}], [{"a": 1, "c": 3}], key="a")
|
||||
[{'a': 1, 'c': 3}]
|
||||
|
||||
:param main: the main list
|
||||
:type main: list
|
||||
:param patch: the list of additional elements
|
||||
:type patch: list
|
||||
:param key: the key for compare
|
||||
"""
|
||||
main_idx = dict(
|
||||
(x[key], i) for i, x in enumerate(main)
|
||||
)
|
||||
|
||||
patch_idx = dict(
|
||||
(x[key], i) for i, x in enumerate(patch)
|
||||
)
|
||||
|
||||
for k in sorted(patch_idx):
|
||||
if k in main_idx:
|
||||
main[main_idx[k]].update(patch[patch_idx[k]])
|
||||
else:
|
||||
main.append(patch[patch_idx[k]])
|
||||
return main
|
||||
|
||||
|
||||
def is_subdict(dict1, dict2):
|
||||
"""Checks that dict1 is subdict of dict2.
|
||||
|
||||
>>> is_subdict({"a": 1}, {'a': 1, 'b': 1})
|
||||
True
|
||||
|
||||
:param dict1: the candidate
|
||||
:param dict2: the super dict
|
||||
:return: True if all keys from dict1 are present
|
||||
and has same value in dict2 otherwise False
|
||||
"""
|
||||
for k, v in six.iteritems(dict1):
|
||||
if k not in dict2 or dict2[k] != v:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def first(*args):
|
||||
"""Get first not empty value.
|
||||
|
||||
>>> first(0, 1) == next(iter(filter(None, [0, 1])))
|
||||
True
|
||||
|
||||
:param args: the list of arguments
|
||||
:return first value that bool(v) is True, None if not found.
|
||||
"""
|
||||
for arg in args:
|
||||
if arg:
|
||||
return arg
|
||||
|
||||
|
||||
def get_fuel_settings():
|
||||
"""Gets the fuel settings from astute container, if it is available."""
|
||||
|
||||
try:
|
||||
with open("/etc/fuel/astute.yaml", "r") as fd:
|
||||
settings = yaml.load(fd)
|
||||
return {
|
||||
"server": settings.get("ADMIN_NETWORK", {}).get("ipaddress"),
|
||||
"user": settings.get("FUEL_ACCESS", {}).get("user"),
|
||||
"password": settings.get("FUEL_ACCESS", {}).get("password")
|
||||
}
|
||||
except (OSError, IOError):
|
||||
return {}
|
||||
|
||||
|
||||
def load_input_data(input_file, **kwargs):
|
||||
"""Load yaml file and parse it to dict with replacement by kwargs.
|
||||
|
||||
:param input_file: name of file to parse fuel mirror template
|
||||
:param kwargs: arguments to substitute template
|
||||
:return: processed from yaml file dict.
|
||||
"""
|
||||
with open(input_file, "r") as fd:
|
||||
return yaml.load(Template(fd.read()).safe_substitute(**kwargs))
|
|
@ -1,120 +0,0 @@
|
|||
SCHEMA = {
|
||||
"$schema": "http://json-schema.org/draft-04/schema#",
|
||||
"definitions": {
|
||||
"DEB_REPO_SCHEMA": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"name",
|
||||
"uri",
|
||||
"suite",
|
||||
"section"
|
||||
],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["deb"]
|
||||
},
|
||||
"uri": {
|
||||
"type": "string"
|
||||
},
|
||||
"priority": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "integer"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
},
|
||||
"suite": {
|
||||
"type": "string"
|
||||
},
|
||||
"section": {
|
||||
"type": "string"
|
||||
},
|
||||
}
|
||||
},
|
||||
"RPM_REPO_SCHEMA": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"name",
|
||||
"uri",
|
||||
],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["rpm"]
|
||||
},
|
||||
"uri": {
|
||||
"type": "string"
|
||||
},
|
||||
"priority": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "integer"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
},
|
||||
"REPO_SCHEMA": {
|
||||
"anyOf":
|
||||
[
|
||||
{"$ref": "#/definitions/DEB_REPO_SCHEMA"},
|
||||
{"$ref": "#/definitions/RPM_REPO_SCHEMA"}
|
||||
]
|
||||
},
|
||||
|
||||
"REPOS_SCHEMA": {
|
||||
"type": "array", "items": {"$ref": "#/definitions/REPO_SCHEMA"}
|
||||
}
|
||||
},
|
||||
"type": "object",
|
||||
"required": [
|
||||
"groups",
|
||||
],
|
||||
"properties": {
|
||||
"fuel_release_match": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"operating_system": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"operating_system"
|
||||
]
|
||||
},
|
||||
"requirements": {
|
||||
"type": "object",
|
||||
"patternProperties": {
|
||||
"^[0-9a-z_-]+$": {"type": "array"}
|
||||
},
|
||||
"additionalProperties": False,
|
||||
},
|
||||
"groups": {
|
||||
"type": "object",
|
||||
"patternProperties": {
|
||||
"^[0-9a-z_-]+$": {"$ref": "#/definitions/REPOS_SCHEMA"}
|
||||
},
|
||||
"additionalProperties": False,
|
||||
},
|
||||
"inheritance": {
|
||||
"type": "object",
|
||||
"patternProperties": {
|
||||
"^[0-9a-z_-]+$": {"type": "string"}
|
||||
},
|
||||
"additionalProperties": False,
|
||||
}
|
||||
}
|
||||
}
|
|
@ -1,33 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
try:
|
||||
import unittest2 as unittest
|
||||
except ImportError:
|
||||
import unittest
|
||||
|
||||
|
||||
class TestCase(unittest.TestCase):
|
||||
|
||||
"""Test case base class for all unit tests."""
|
||||
|
||||
def assertNotRaises(self, exception, method, *args, **kwargs):
|
||||
try:
|
||||
method(*args, **kwargs)
|
||||
except exception as e:
|
||||
self.fail("Unexpected error: {0}".format(e))
|
|
@ -1,22 +0,0 @@
|
|||
fuel_release_match:
|
||||
operating_system: CentOS
|
||||
|
||||
inheritance:
|
||||
centos: mos
|
||||
|
||||
groups:
|
||||
mos:
|
||||
- name: "mos"
|
||||
type: "rpm"
|
||||
uri: "http://localhost/mos$mos_version/x86_64"
|
||||
priority: 10
|
||||
|
||||
centos:
|
||||
- name: "centos"
|
||||
type: "rpm"
|
||||
uri: "http://localhost/centos/os/x86_64"
|
||||
priority: 5
|
||||
|
||||
requirements:
|
||||
centos:
|
||||
- "package_rpm"
|
|
@ -1,7 +0,0 @@
|
|||
threads_num: 1
|
||||
ignore_errors_num: 2
|
||||
retries_num: 3
|
||||
http_proxy: "http://localhost"
|
||||
https_proxy: "https://localhost"
|
||||
target_dir: "/var/www/"
|
||||
base_url: "http://{FUEL_SERVER_IP}:8080/"
|
|
@ -1,9 +0,0 @@
|
|||
fuel_release_match:
|
||||
operating_system: Ubuntu
|
||||
|
||||
inheritance:
|
||||
ubuntu: mos
|
||||
|
||||
requirements:
|
||||
ubuntu:
|
||||
- "package_deb"
|
|
@ -1,27 +0,0 @@
|
|||
fuel_release_match:
|
||||
operating_system: Ubuntu
|
||||
|
||||
inheritance:
|
||||
ubuntu: mos
|
||||
|
||||
groups:
|
||||
mos:
|
||||
- name: "mos"
|
||||
type: "deb"
|
||||
uri: "http://localhost/mos"
|
||||
suite: "mos$mos_version"
|
||||
section: "main restricted"
|
||||
priority: 1000
|
||||
|
||||
ubuntu:
|
||||
- name: "ubuntu"
|
||||
type: "deb"
|
||||
main: true
|
||||
uri: "http://localhost/ubuntu"
|
||||
suite: "trusty"
|
||||
section: "main multiverse restricted universe"
|
||||
priority: 500
|
||||
|
||||
requirements:
|
||||
ubuntu:
|
||||
- "package_deb"
|
|
@ -1,92 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import mock
|
||||
|
||||
from fuel_mirror.common import accessors
|
||||
from fuel_mirror.tests import base
|
||||
|
||||
|
||||
class TestAccessors(base.TestCase):
|
||||
def test_get_packetary_accessor(self):
|
||||
packetary = mock.MagicMock()
|
||||
with mock.patch.dict("sys.modules", packetary=packetary):
|
||||
accessor = accessors.get_packetary_accessor(
|
||||
http_proxy="http://localhost",
|
||||
https_proxy="https://localhost",
|
||||
retries_num=1,
|
||||
threads_num=2,
|
||||
ignore_errors_num=3
|
||||
)
|
||||
accessor("deb")
|
||||
accessor("yum")
|
||||
packetary.Configuration.assert_called_once_with(
|
||||
http_proxy="http://localhost",
|
||||
https_proxy="https://localhost",
|
||||
retries_num=1,
|
||||
threads_num=2,
|
||||
ignore_errors_num=3
|
||||
)
|
||||
packetary.Context.assert_called_once_with(
|
||||
packetary.Configuration()
|
||||
)
|
||||
self.assertEqual(2, packetary.RepositoryApi.create.call_count)
|
||||
packetary.RepositoryApi.create.assert_any_call(
|
||||
packetary.Context(), "deb"
|
||||
)
|
||||
packetary.RepositoryApi.create.assert_any_call(
|
||||
packetary.Context(), "yum"
|
||||
)
|
||||
|
||||
@mock.patch("fuel_mirror.common.accessors.os")
|
||||
def test_get_fuel_api_accessor(self, os):
|
||||
fuelclient = mock.MagicMock()
|
||||
patch = {
|
||||
"fuelclient": fuelclient,
|
||||
"fuelclient.objects": fuelclient.objects
|
||||
}
|
||||
with mock.patch.dict("sys.modules", patch):
|
||||
accessor = accessors.get_fuel_api_accessor(
|
||||
"localhost:8080", "guest", "123"
|
||||
)
|
||||
accessor.Environment.get_all()
|
||||
|
||||
os.environ.__setitem__.asseert_any_call(
|
||||
"SERVER_ADDRESS", "localhost"
|
||||
)
|
||||
os.environ.__setitem__.asseert_any_call(
|
||||
"LISTEN_PORT", "8080"
|
||||
)
|
||||
os.environ.__setitem__.asseert_any_call(
|
||||
"KEYSTONE_USER", "guest"
|
||||
)
|
||||
os.environ.__setitem__.asseert_any_call(
|
||||
"KEYSTONE_PASS", "123"
|
||||
)
|
||||
fuelclient.objects.Environment.get_all.assert_called_once_with()
|
||||
|
||||
@mock.patch("fuel_mirror.common.accessors.os")
|
||||
def test_get_fuel_api_accessor_with_default_parameters(self, os):
|
||||
fuelclient = mock.MagicMock()
|
||||
patch = {
|
||||
"fuelclient": fuelclient,
|
||||
"fuelclient.objects": fuelclient.objects
|
||||
}
|
||||
with mock.patch.dict("sys.modules", patch):
|
||||
accessors.get_fuel_api_accessor()
|
||||
os.environ.__setitem__.assert_not_called()
|
|
@ -1,499 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import mock
|
||||
import os.path
|
||||
import subprocess
|
||||
|
||||
from jsonschema import ValidationError
|
||||
|
||||
# The cmd2 does not work with python3.5
|
||||
# because it tries to get access to the property mswindows,
|
||||
# that was removed in 3.5
|
||||
subprocess.mswindows = False
|
||||
|
||||
from fuel_mirror.commands import apply
|
||||
from fuel_mirror.commands import create
|
||||
from fuel_mirror.common.utils import load_input_data
|
||||
from fuel_mirror.tests import base
|
||||
|
||||
|
||||
CONFIG_PATH = os.path.join(
|
||||
os.path.dirname(__file__), "data", "test_config.yaml"
|
||||
)
|
||||
|
||||
UBUNTU_PATH = os.path.join(
|
||||
os.path.dirname(__file__), "data", "test_ubuntu.yaml"
|
||||
)
|
||||
|
||||
CENTOS_PATH = os.path.join(
|
||||
os.path.dirname(__file__), "data", "test_centos.yaml"
|
||||
)
|
||||
|
||||
INVALID_DATA_PATH = os.path.join(
|
||||
os.path.dirname(__file__), "data", "test_invalid_ubuntu.yaml"
|
||||
)
|
||||
|
||||
|
||||
# TODO(akostrikov) lists_merge we are using is not stable so we have to use
|
||||
# different local repos in cases with merge and without it.
|
||||
# We pass sorted by priority list, but in lists_merge we sort it by key.
|
||||
# As we are aiming to use existing repos as primary source - it is not issue.
|
||||
def local_repos(mirror_host='10.25.0.10:8080', name_postfix='', reverse=True):
|
||||
mirror_lists = load_input_data(UBUNTU_PATH, mos_version=1)
|
||||
sorted_repos = reduce(lambda x, y: x + y, mirror_lists['groups'].values())
|
||||
sorted_repos.sort(key=lambda x: x['priority'], reverse=reverse)
|
||||
for repo in sorted_repos:
|
||||
repo.pop('main', None)
|
||||
repo['name'] = repo['name'] + name_postfix
|
||||
repo['uri'] = repo['uri'].replace('localhost', mirror_host)
|
||||
return sorted_repos
|
||||
|
||||
|
||||
def mirror_repos():
|
||||
return local_repos(mirror_host='mirror.com:8080', name_postfix='-mirror')
|
||||
|
||||
|
||||
@mock.patch.multiple(
|
||||
"fuel_mirror.app",
|
||||
accessors=mock.DEFAULT
|
||||
)
|
||||
class TestCliCommands(base.TestCase):
|
||||
common_argv = [
|
||||
"--config", CONFIG_PATH,
|
||||
"--fuel-server=10.25.0.10",
|
||||
"--fuel-user=test",
|
||||
"--fuel-password=test1"
|
||||
]
|
||||
|
||||
def start_cmd(self, cmd, argv, data_file):
|
||||
cmd.debug(
|
||||
argv + self.common_argv + ["--input-file", data_file]
|
||||
)
|
||||
|
||||
def _setup_fuel_versions(self, fuel_mock):
|
||||
fuel_mock.FuelVersion.get_all_data.return_value = {
|
||||
"release": "1",
|
||||
"openstack_version": "2"
|
||||
}
|
||||
|
||||
def _create_fuel_release(self, fuel_mock, osname, repos=None):
|
||||
if repos is None:
|
||||
repos = []
|
||||
release = mock.MagicMock(data={
|
||||
"name": "test release",
|
||||
"operating_system": osname,
|
||||
"attributes_metadata": {
|
||||
"editable": {"repo_setup": {"repos": {"value": repos}}}
|
||||
}
|
||||
})
|
||||
|
||||
fuel_mock.Release.get_by_ids.return_value = [release]
|
||||
fuel_mock.Release.get_all.return_value = [release]
|
||||
return release
|
||||
|
||||
def _create_fuel_env(self, fuel_mock, repos=None):
|
||||
if repos is None:
|
||||
repos = []
|
||||
env = mock.MagicMock(data={
|
||||
"name": "test",
|
||||
"release_id": 1
|
||||
})
|
||||
env.get_settings_data.return_value = {
|
||||
"editable": {"repo_setup": {"repos": {"value": repos}}}
|
||||
}
|
||||
fuel_mock.Environment.get_by_ids.return_value = [env]
|
||||
fuel_mock.Environment.get_all.return_value = [env]
|
||||
return env
|
||||
|
||||
def test_create_mos_ubuntu(self, accessors):
|
||||
self._setup_fuel_versions(accessors.get_fuel_api_accessor())
|
||||
packetary = accessors.get_packetary_accessor()
|
||||
|
||||
self.start_cmd(create, ["--group", "mos"], UBUNTU_PATH)
|
||||
accessors.get_packetary_accessor.assert_called_with(
|
||||
threads_num=1,
|
||||
ignore_errors_num=2,
|
||||
retries_num=3,
|
||||
http_proxy="http://localhost",
|
||||
https_proxy="https://localhost",
|
||||
)
|
||||
packetary.assert_called_with("deb", "x86_64")
|
||||
api = packetary()
|
||||
api.clone_repositories.assert_called_once_with(
|
||||
['http://localhost/mos mos1 main restricted'],
|
||||
'/var/www/',
|
||||
None, None
|
||||
)
|
||||
|
||||
def test_create_partial_ubuntu(self, accessors):
|
||||
self._setup_fuel_versions(accessors.get_fuel_api_accessor())
|
||||
packetary = accessors.get_packetary_accessor()
|
||||
|
||||
self.start_cmd(create, ["--group", "ubuntu"], UBUNTU_PATH)
|
||||
accessors.get_packetary_accessor.assert_called_with(
|
||||
threads_num=1,
|
||||
ignore_errors_num=2,
|
||||
retries_num=3,
|
||||
http_proxy="http://localhost",
|
||||
https_proxy="https://localhost",
|
||||
)
|
||||
packetary.assert_called_with("deb", "x86_64")
|
||||
api = packetary()
|
||||
api.clone_repositories.assert_called_once_with(
|
||||
['http://localhost/ubuntu trusty '
|
||||
'main multiverse restricted universe'],
|
||||
'/var/www/',
|
||||
['http://localhost/mos mos1 main restricted'],
|
||||
['package_deb']
|
||||
)
|
||||
|
||||
def test_create_mos_centos(self, accessors):
|
||||
self._setup_fuel_versions(accessors.get_fuel_api_accessor())
|
||||
packetary = accessors.get_packetary_accessor()
|
||||
|
||||
self.start_cmd(create, ["--group", "mos"], CENTOS_PATH)
|
||||
accessors.get_packetary_accessor.assert_called_with(
|
||||
threads_num=1,
|
||||
ignore_errors_num=2,
|
||||
retries_num=3,
|
||||
http_proxy="http://localhost",
|
||||
https_proxy="https://localhost",
|
||||
)
|
||||
packetary.assert_called_with("rpm", "x86_64")
|
||||
api = packetary()
|
||||
api.clone_repositories.assert_called_once_with(
|
||||
['http://localhost/mos1/x86_64'],
|
||||
'/var/www/',
|
||||
None, None
|
||||
)
|
||||
|
||||
def test_create_partial_centos(self, accessors):
|
||||
self._setup_fuel_versions(accessors.get_fuel_api_accessor())
|
||||
packetary = accessors.get_packetary_accessor()
|
||||
|
||||
self.start_cmd(create, ["--group", "centos"], CENTOS_PATH)
|
||||
accessors.get_packetary_accessor.assert_called_with(
|
||||
threads_num=1,
|
||||
ignore_errors_num=2,
|
||||
retries_num=3,
|
||||
http_proxy="http://localhost",
|
||||
https_proxy="https://localhost",
|
||||
)
|
||||
packetary.assert_called_with("rpm", "x86_64")
|
||||
api = packetary()
|
||||
api.clone_repositories.assert_called_once_with(
|
||||
['http://localhost/centos/os/x86_64'],
|
||||
'/var/www/',
|
||||
['http://localhost/mos1/x86_64'],
|
||||
["package_rpm"]
|
||||
)
|
||||
|
||||
def test_apply_for_ubuntu_based_env(self, accessors):
|
||||
fuel = accessors.get_fuel_api_accessor()
|
||||
self._setup_fuel_versions(fuel)
|
||||
env = self._create_fuel_env(fuel)
|
||||
self._create_fuel_release(fuel, "Ubuntu")
|
||||
self.start_cmd(
|
||||
apply, ['--group', 'mos', 'ubuntu', '--env', '1'],
|
||||
UBUNTU_PATH
|
||||
)
|
||||
accessors.get_fuel_api_accessor.assert_called_with(
|
||||
"10.25.0.10", "test", "test1"
|
||||
)
|
||||
fuel.FuelVersion.get_all_data.assert_called_once_with()
|
||||
env.set_settings_data.assert_called_with(
|
||||
{
|
||||
'editable': {
|
||||
'repo_setup': {
|
||||
'repos': {'value': local_repos()}
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_with_existing_mirrors(self, accessors):
|
||||
fuel = accessors.get_fuel_api_accessor()
|
||||
self._setup_fuel_versions(fuel)
|
||||
env = self._create_fuel_env(fuel, repos=mirror_repos())
|
||||
self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos())
|
||||
self.start_cmd(
|
||||
apply, ['--group', 'mos', 'ubuntu', '--env', '1'],
|
||||
UBUNTU_PATH
|
||||
)
|
||||
accessors.get_fuel_api_accessor.assert_called_with(
|
||||
"10.25.0.10", "test", "test1"
|
||||
)
|
||||
fuel.FuelVersion.get_all_data.assert_called_once_with()
|
||||
env.set_settings_data.assert_called_with(
|
||||
{
|
||||
'editable': {
|
||||
'repo_setup': {
|
||||
'repos': {'value': mirror_repos() + local_repos()}
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_replace_existing_mirrors_with_local(self, accessors):
|
||||
fuel = accessors.get_fuel_api_accessor()
|
||||
self._setup_fuel_versions(fuel)
|
||||
env = self._create_fuel_env(fuel, repos=mirror_repos())
|
||||
self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos())
|
||||
self.start_cmd(
|
||||
apply, ['--group', 'mos', 'ubuntu', '--env', '1', '--replace'],
|
||||
UBUNTU_PATH
|
||||
)
|
||||
accessors.get_fuel_api_accessor.assert_called_with(
|
||||
"10.25.0.10", "test", "test1"
|
||||
)
|
||||
fuel.FuelVersion.get_all_data.assert_called_once_with()
|
||||
env.set_settings_data.assert_called_with(
|
||||
{
|
||||
'editable': {
|
||||
'repo_setup': {
|
||||
'repos': {'value': local_repos(reverse=False)}
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_apply_for_centos_based_env(self, accessors):
|
||||
fuel = accessors.get_fuel_api_accessor()
|
||||
self._setup_fuel_versions(fuel)
|
||||
env = self._create_fuel_env(fuel)
|
||||
self._create_fuel_release(fuel, "CentOS")
|
||||
self.start_cmd(
|
||||
apply, ['--group', 'mos', 'centos', '--env', '1'],
|
||||
CENTOS_PATH
|
||||
)
|
||||
accessors.get_fuel_api_accessor.assert_called_with(
|
||||
"10.25.0.10", "test", "test1"
|
||||
)
|
||||
fuel.FuelVersion.get_all_data.assert_called_once_with()
|
||||
env.set_settings_data.assert_called_with(
|
||||
{'editable': {'repo_setup': {'repos': {'value': [
|
||||
{
|
||||
'priority': 5,
|
||||
'name': 'centos',
|
||||
'type': 'rpm',
|
||||
'uri': 'http://10.25.0.10:8080/centos/os/x86_64'
|
||||
},
|
||||
{
|
||||
'priority': 10,
|
||||
'name': 'mos',
|
||||
'type': 'rpm',
|
||||
'uri': 'http://10.25.0.10:8080/mos1/x86_64'
|
||||
}]
|
||||
}}}}
|
||||
)
|
||||
|
||||
def test_apply_for_ubuntu_release(self, accessors):
|
||||
fuel = accessors.get_fuel_api_accessor()
|
||||
self._setup_fuel_versions(fuel)
|
||||
env = self._create_fuel_env(fuel)
|
||||
release = self._create_fuel_release(fuel, "Ubuntu")
|
||||
self.start_cmd(
|
||||
apply, ['--group', 'mos', 'ubuntu', '--default'],
|
||||
UBUNTU_PATH
|
||||
)
|
||||
accessors.get_fuel_api_accessor.assert_called_with(
|
||||
"10.25.0.10", "test", "test1"
|
||||
)
|
||||
fuel.FuelVersion.get_all_data.assert_called_once_with()
|
||||
self.assertEqual(1, env.set_settings_data.call_count)
|
||||
release.connection.put_request.assert_called_once_with(
|
||||
release.instance_api_path.format(),
|
||||
{
|
||||
'name': "test release",
|
||||
'operating_system': 'Ubuntu',
|
||||
'attributes_metadata': {
|
||||
'editable': {'repo_setup': {'repos': {'value': [
|
||||
{
|
||||
'name': 'mos',
|
||||
'priority': 1000,
|
||||
'suite': 'mos1',
|
||||
'section': 'main restricted',
|
||||
'type': 'deb',
|
||||
'uri': 'http://10.25.0.10:8080/mos'
|
||||
},
|
||||
{
|
||||
'name': 'ubuntu',
|
||||
'priority': 500,
|
||||
'suite': 'trusty',
|
||||
'section': 'main multiverse restricted universe',
|
||||
'type': 'deb',
|
||||
'uri': 'http://10.25.0.10:8080/ubuntu'
|
||||
}
|
||||
]}}}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_update_release_and_cluster_repositories(self, accessors):
|
||||
fuel = accessors.get_fuel_api_accessor()
|
||||
self._setup_fuel_versions(fuel)
|
||||
env = self._create_fuel_env(fuel, repos=mirror_repos())
|
||||
release = \
|
||||
self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos())
|
||||
self.start_cmd(
|
||||
apply, ['--group', 'mos', 'ubuntu', '--default'],
|
||||
UBUNTU_PATH
|
||||
)
|
||||
accessors.get_fuel_api_accessor.assert_called_with(
|
||||
"10.25.0.10", "test", "test1"
|
||||
)
|
||||
fuel.FuelVersion.get_all_data.assert_called_once_with()
|
||||
expected_repos = mirror_repos() + local_repos()
|
||||
env.set_settings_data.assert_called_with(
|
||||
{
|
||||
'editable': {
|
||||
'repo_setup': {
|
||||
'repos': {'value': expected_repos}
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
release.connection.put_request.assert_called_once_with(
|
||||
release.instance_api_path.format(),
|
||||
{
|
||||
'name': "test release",
|
||||
'operating_system': 'Ubuntu',
|
||||
'attributes_metadata': {
|
||||
'editable': {'repo_setup': {'repos': {
|
||||
'value': expected_repos
|
||||
}}}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_replace_release_and_cluster_repositories(self, accessors):
|
||||
fuel = accessors.get_fuel_api_accessor()
|
||||
self._setup_fuel_versions(fuel)
|
||||
env = self._create_fuel_env(fuel, repos=mirror_repos())
|
||||
release = \
|
||||
self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos())
|
||||
self.start_cmd(
|
||||
apply, ['--group', 'mos', 'ubuntu', '--default', '--replace'],
|
||||
UBUNTU_PATH
|
||||
)
|
||||
accessors.get_fuel_api_accessor.assert_called_with(
|
||||
"10.25.0.10", "test", "test1"
|
||||
)
|
||||
fuel.FuelVersion.get_all_data.assert_called_once_with()
|
||||
expected_repos = local_repos(reverse=False)
|
||||
env.set_settings_data.assert_called_with(
|
||||
{
|
||||
'editable': {
|
||||
'repo_setup': {
|
||||
'repos': {'value': expected_repos}
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
release.connection.put_request.assert_called_once_with(
|
||||
release.instance_api_path.format(),
|
||||
{
|
||||
'name': "test release",
|
||||
'operating_system': 'Ubuntu',
|
||||
'attributes_metadata': {
|
||||
'editable': {'repo_setup': {'repos': {
|
||||
'value': expected_repos
|
||||
}}}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_apply_for_centos_release(self, accessors):
|
||||
fuel = accessors.get_fuel_api_accessor()
|
||||
self._setup_fuel_versions(fuel)
|
||||
env = self._create_fuel_env(fuel)
|
||||
release = self._create_fuel_release(fuel, "CentOS")
|
||||
self.start_cmd(
|
||||
apply, ['--group', 'mos', 'centos', '--default'],
|
||||
CENTOS_PATH
|
||||
)
|
||||
accessors.get_fuel_api_accessor.assert_called_with(
|
||||
"10.25.0.10", "test", "test1"
|
||||
)
|
||||
fuel.FuelVersion.get_all_data.assert_called_once_with()
|
||||
self.assertEqual(1, env.set_settings_data.call_count)
|
||||
release.connection.put_request.assert_called_once_with(
|
||||
release.instance_api_path.format(),
|
||||
{
|
||||
'name': "test release",
|
||||
'operating_system': 'CentOS',
|
||||
'attributes_metadata': {
|
||||
'editable': {'repo_setup': {'repos': {'value': [
|
||||
{
|
||||
'name': 'centos',
|
||||
'priority': 5,
|
||||
'type': 'rpm',
|
||||
'uri': 'http://10.25.0.10:8080/centos/os/x86_64'
|
||||
},
|
||||
{
|
||||
'name': 'mos',
|
||||
'priority': 10,
|
||||
'type': 'rpm',
|
||||
'uri': 'http://10.25.0.10:8080/mos1/x86_64'
|
||||
},
|
||||
]}}}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
@mock.patch("fuel_mirror.app.utils.get_fuel_settings")
|
||||
def test_apply_fail_if_no_fuel_address(self, m_get_settings, accessors):
|
||||
m_get_settings.return_value = {}
|
||||
with self.assertRaisesRegexp(
|
||||
ValueError, "Please specify the fuel-server option"):
|
||||
apply.debug(
|
||||
["--config", CONFIG_PATH, "-G", "mos", "-I", UBUNTU_PATH]
|
||||
)
|
||||
self.assertFalse(accessors.get_fuel_api_accessor.called)
|
||||
|
||||
@mock.patch("fuel_mirror.app.utils.get_fuel_settings")
|
||||
def test_create_without_fuel_address(self, m_get_settings, accessors):
|
||||
m_get_settings.return_value = {}
|
||||
packetary = accessors.get_packetary_accessor()
|
||||
create.debug(
|
||||
["--config", CONFIG_PATH, "-G", "mos", "-I", UBUNTU_PATH]
|
||||
)
|
||||
self.assertFalse(accessors.get_fuel_api_accessor.called)
|
||||
accessors.get_packetary_accessor.assert_called_with(
|
||||
threads_num=1,
|
||||
ignore_errors_num=2,
|
||||
retries_num=3,
|
||||
http_proxy="http://localhost",
|
||||
https_proxy="https://localhost",
|
||||
)
|
||||
packetary.assert_called_with("deb", "x86_64")
|
||||
api = packetary()
|
||||
api.clone_repositories.assert_called_once_with(
|
||||
['http://localhost/mos mos main restricted'],
|
||||
'/var/www/',
|
||||
None,
|
||||
None
|
||||
)
|
||||
|
||||
@mock.patch("fuel_mirror.app.utils.get_fuel_settings")
|
||||
def test_create_with_invalid_data(self, m_get_settings, accessors):
|
||||
self.assertRaises(
|
||||
ValidationError, create.debug, ["--config", CONFIG_PATH, "-G",
|
||||
"mos", "-I", INVALID_DATA_PATH]
|
||||
)
|
|
@ -1,70 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from fuel_mirror.common import url_builder
|
||||
from fuel_mirror.tests import base
|
||||
|
||||
|
||||
class TestUrlBuilder(base.TestCase):
|
||||
def test_get_url_builder(self):
|
||||
self.assertTrue(issubclass(
|
||||
url_builder.get_url_builder("deb"),
|
||||
url_builder.AptRepoUrlBuilder
|
||||
))
|
||||
self.assertTrue(issubclass(
|
||||
url_builder.get_url_builder("rpm"),
|
||||
url_builder.YumRepoUrlBuilder
|
||||
))
|
||||
with self.assertRaises(KeyError):
|
||||
url_builder.get_url_builder("unknown")
|
||||
|
||||
|
||||
class TestAptUrlBuilder(base.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.builder = url_builder.get_url_builder("deb")
|
||||
cls.repo_data = {
|
||||
"name": "ubuntu",
|
||||
"suite": "trusty",
|
||||
"section": "main restricted",
|
||||
"type": "deb",
|
||||
"uri": "http://localhost/ubuntu"
|
||||
}
|
||||
|
||||
def test_get_repo_url(self):
|
||||
self.assertEqual(
|
||||
"http://localhost/ubuntu trusty main restricted",
|
||||
self.builder.get_repo_url(self.repo_data)
|
||||
)
|
||||
|
||||
|
||||
class TestYumUrlBuilder(base.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.builder = url_builder.get_url_builder("rpm")
|
||||
cls.repo_data = {
|
||||
"name": "centos",
|
||||
"type": "rpm",
|
||||
"uri": "http://localhost/os/x86_64"
|
||||
}
|
||||
|
||||
def test_get_repo_url(self):
|
||||
self.assertEqual(
|
||||
"http://localhost/os/x86_64",
|
||||
self.builder.get_repo_url(self.repo_data)
|
||||
)
|
|
@ -1,114 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import mock
|
||||
import six
|
||||
|
||||
from fuel_mirror.common import utils
|
||||
from fuel_mirror.tests import base
|
||||
|
||||
|
||||
class DictAsObj(object):
|
||||
def __init__(self, d):
|
||||
self.__dict__.update(d)
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.__dict__ == other.__dict__
|
||||
|
||||
|
||||
class TestUtils(base.TestCase):
|
||||
def test_lists_merge(self):
|
||||
main = [{"a": 1, "b": 2, "c": 0}, {"a": 2, "b": 3, "c": 1}]
|
||||
patch = [{"a": 2, "b": 4}, {"a": 3, "b": 5}]
|
||||
utils.lists_merge(
|
||||
main,
|
||||
patch,
|
||||
key="a"
|
||||
)
|
||||
self.assertItemsEqual(
|
||||
[{"a": 1, "b": 2, "c": 0},
|
||||
{"a": 2, "b": 4, "c": 1},
|
||||
{"a": 3, "b": 5}],
|
||||
main
|
||||
)
|
||||
|
||||
def test_first(self):
|
||||
self.assertEqual(
|
||||
1,
|
||||
utils.first(0, 1, 0),
|
||||
)
|
||||
self.assertEqual(
|
||||
1,
|
||||
utils.first(None, [], '', 1),
|
||||
)
|
||||
self.assertIsNone(
|
||||
utils.first(None, [], 0, ''),
|
||||
)
|
||||
self.assertIsNone(
|
||||
utils.first(),
|
||||
)
|
||||
|
||||
def test_is_subdict(self):
|
||||
self.assertFalse(utils.is_subdict({"c": 1}, {"a": 1, "b": 1}))
|
||||
self.assertFalse(utils.is_subdict({"a": 1, "b": 2}, {"a": 1, "b": 1}))
|
||||
self.assertFalse(
|
||||
utils.is_subdict({"a": 1, "b": 1, "c": 2}, {"a": 1, "b": 1})
|
||||
)
|
||||
self.assertFalse(
|
||||
utils.is_subdict({"a": 1, "b": None}, {"a": 1})
|
||||
)
|
||||
self.assertTrue(utils.is_subdict({}, {"a": 1}))
|
||||
self.assertTrue(utils.is_subdict({"a": 1}, {"a": 1, "b": 1}))
|
||||
self.assertTrue(utils.is_subdict({"a": 1, "b": 1}, {"a": 1, "b": 1}))
|
||||
|
||||
@mock.patch("fuel_mirror.common.utils.open")
|
||||
def test_get_fuel_settings(self, m_open):
|
||||
m_open().__enter__.side_effect = [
|
||||
six.StringIO(
|
||||
'ADMIN_NETWORK:\n'
|
||||
' ipaddress: "10.20.0.4"\n'
|
||||
'FUEL_ACCESS:\n'
|
||||
' user: "test"\n'
|
||||
' password: "test_pwd"\n',
|
||||
),
|
||||
OSError
|
||||
]
|
||||
|
||||
self.assertEqual(
|
||||
{
|
||||
"server": "10.20.0.4",
|
||||
"user": "test",
|
||||
"password": "test_pwd",
|
||||
},
|
||||
utils.get_fuel_settings()
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
{},
|
||||
utils.get_fuel_settings()
|
||||
)
|
||||
|
||||
@mock.patch("fuel_mirror.common.utils.yaml")
|
||||
@mock.patch("fuel_mirror.common.utils.open")
|
||||
def test_load_input_data(self, open_mock, yaml_mock):
|
||||
data = "$param1: $param2"
|
||||
open_mock().__enter__().read.return_value = data
|
||||
v = utils.load_input_data("data.yaml", param1="key", param2="value")
|
||||
open_mock.assert_called_with("data.yaml", "r")
|
||||
yaml_mock.load.assert_called_once_with("key: value")
|
||||
self.assertIs(yaml_mock.load(), v)
|
|
@ -1,158 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import os.path
|
||||
|
||||
from jsonschema import validate
|
||||
from jsonschema import ValidationError
|
||||
import yaml
|
||||
|
||||
from fuel_mirror.schemas.input_data_schema import SCHEMA
|
||||
from fuel_mirror.tests import base
|
||||
|
||||
|
||||
DATA_DIR = os.path.join(os.path.dirname(__file__), "..", "..", "data")
|
||||
|
||||
|
||||
class TestValidateConfigs(base.TestCase):
|
||||
|
||||
def test_validate_data_files(self):
|
||||
for f in os.listdir(DATA_DIR):
|
||||
with open(os.path.join(DATA_DIR, f), "r") as fd:
|
||||
data = yaml.load(fd)
|
||||
self.assertNotRaises(ValidationError, validate, data, SCHEMA)
|
||||
self.assertIn("groups", data)
|
||||
self.assertIn("fuel_release_match", data)
|
||||
|
||||
def test_validate_fail_with_empty_data(self):
|
||||
self.assertRaises(ValidationError, validate, {}, SCHEMA)
|
||||
|
||||
def test_validate_fail_without_groups(self):
|
||||
invalid_data = {
|
||||
"requirements": {
|
||||
"ubuntu": ["package_deb"]
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "'groups' is a required property", validate,
|
||||
invalid_data, SCHEMA)
|
||||
|
||||
def test_invalid_requirements_in_pattern_properies(self):
|
||||
invalid_data = {
|
||||
"requirements": {
|
||||
"ubun.tu": ["package_deb"]
|
||||
},
|
||||
"groups": {
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "'ubun.tu' was unexpected", validate,
|
||||
invalid_data, SCHEMA)
|
||||
|
||||
def test_invalid_requirements_type_array(self):
|
||||
invalid_data = {
|
||||
"requirements": {
|
||||
"ubuntu": "package_deb"
|
||||
},
|
||||
"groups": {
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "'package_deb' is not of type 'array'", validate,
|
||||
invalid_data, SCHEMA)
|
||||
|
||||
def test_invalid_inheritens_in_pattern_properies(self):
|
||||
invalid_data = {
|
||||
"inheritance": {
|
||||
"ubun.tu": "mos"
|
||||
},
|
||||
"groups": {
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "'ubun.tu' was unexpected", validate,
|
||||
invalid_data, SCHEMA)
|
||||
|
||||
def test_invalid_inheritens_type_string(self):
|
||||
invalid_data = {
|
||||
"inheritance": {
|
||||
"ubuntu": 123
|
||||
},
|
||||
"groups": {
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "123 is not of type 'string'", validate,
|
||||
invalid_data, SCHEMA)
|
||||
|
||||
def test_invalid_groups_in_pattern_properies(self):
|
||||
invalid_data = {
|
||||
"groups": {
|
||||
"mo.s": []
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "'mo.s' was unexpected", validate,
|
||||
invalid_data, SCHEMA)
|
||||
|
||||
def test_invalid_groups_type_array(self):
|
||||
invalid_data = {
|
||||
"groups": {
|
||||
"mos": "string"
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "'string' is not of type 'array'", validate,
|
||||
invalid_data, SCHEMA)
|
||||
|
||||
def test_without_name_in_groups_array(self):
|
||||
invalid_data = {
|
||||
"groups": {
|
||||
"mos": [
|
||||
{
|
||||
'type': 'deb',
|
||||
'uri': 'http://localhost/mos',
|
||||
'priority': None,
|
||||
'suite': 'mos$mos_version',
|
||||
'section': 'main restricted'
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "is not valid under any of the given schemas",
|
||||
validate, invalid_data, SCHEMA)
|
||||
|
||||
def test_with_invalid_type_in_groups_array(self):
|
||||
invalid_data = {
|
||||
"groups": {
|
||||
"mos": [
|
||||
{
|
||||
'name': 'mos',
|
||||
'type': 'adf',
|
||||
'uri': 'http://localhost/mos',
|
||||
'priority': None,
|
||||
'suite': 'mos$mos_version',
|
||||
'section': 'main restricted'
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
self.assertRaisesRegexp(
|
||||
ValidationError, "is not valid under any of the given schemas",
|
||||
validate, invalid_data, SCHEMA)
|
|
@ -1,12 +0,0 @@
|
|||
# The order of packages is significant, because pip processes them in the order
|
||||
# of appearance. Changing the order has an impact on the overall integration
|
||||
# process, which may cause wedges in the gate later.
|
||||
|
||||
pbr>=0.8
|
||||
Babel>=1.3
|
||||
cliff>=1.7.0
|
||||
six>=1.5.2
|
||||
PyYAML>=3.10
|
||||
packetary>=0.1.0
|
||||
python-fuelclient>=7.0.0
|
||||
jsonschema>=2.3.0
|
|
@ -1,80 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
echo "This script is DEPRECATED. Please use fuel-mirror utility!"
|
||||
|
||||
# This shell script was wraps the fuel-mirror utility to provide backward compatibility
|
||||
# with previous version of tool.
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: `basename $0` [options]
|
||||
|
||||
Create and update local mirrors of MOS and/or Ubuntu.
|
||||
|
||||
IMPORTANT!
|
||||
If NO parameters specified, this script will:
|
||||
- Create/Update both MOS and Ubuntu local mirrors
|
||||
- Set them as repositories for existing NEW environments in Fuel UI
|
||||
- Set them as DEFAULT repositories for new environments
|
||||
|
||||
Options:
|
||||
|
||||
-h| --help This help screen.
|
||||
-d| --no-default Don't change default repositories for new environments
|
||||
-a| --no-apply Don't apply changes to Fuel environments
|
||||
-M| --mos Create/Update MOS local mirror only
|
||||
-U| --ubuntu Create/Update Ubuntu local mirror only
|
||||
-p| --password Fuel Master admin password (defaults to admin)
|
||||
-v| --verbose Add detailed output
|
||||
|
||||
EOF
|
||||
}
|
||||
|
||||
# Parse options
|
||||
OPTS=`getopt -o hdaMUNp: -l help,no-default,no-apply,mos,ubuntu,password:,dry-run,verbose -- "$@"`
|
||||
if [ $? != 0 ]; then
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
eval set -- "$OPTS"
|
||||
|
||||
CMD_OPTS="--pattern=ubuntu"
|
||||
REPO_GROUPS=""
|
||||
|
||||
while true ; do
|
||||
case "$1" in
|
||||
-h| --help ) usage ; exit 0;;
|
||||
-d | --no-default ) OPT_NO_DEFAULT=1; shift;;
|
||||
-a | --no-apply ) OPT_NO_APPLY=1; shift;;
|
||||
-N | --dry-run ) EXEC_PREFIX="echo EXEC "; shift;;
|
||||
-M | --mos ) REPO_GROUPS="$REPO_GROUPS mos"; shift;;
|
||||
-U | --ubuntu ) REPO_GROUPS="$REPO_GROUPS ubuntu"; shift;;
|
||||
-p | --password ) CMD_OPTS="$CMD_OPTS --fuel-password=$2"; shift; shift;;
|
||||
-v | --verbose ) CMD_OPTS="$CMD_OPTS --verbose"; shift;;
|
||||
-- ) shift; break;;
|
||||
* ) break;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ "$@" != "" ]]; then
|
||||
echo "Invalid option -- $@"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$REPO_GROUPS" == "" ]]; then
|
||||
REPO_GROUPS="mos ubuntu"
|
||||
fi
|
||||
|
||||
CMD_OPTS="$CMD_OPTS --group $REPO_GROUPS"
|
||||
|
||||
$EXEC_PREFIX fuel-mirror create ${CMD_OPTS}
|
||||
|
||||
if [[ "$OPT_NO_DEFAULT" == "" ]]; then
|
||||
CMD_OPTS="$CMD_OPTS --default"
|
||||
fi
|
||||
|
||||
if [[ "$OPT_NO_APPLY" == "" ]]; then
|
||||
$EXEC_PREFIX fuel-mirror apply ${CMD_OPTS}
|
||||
fi
|
|
@ -1,67 +0,0 @@
|
|||
[metadata]
|
||||
name = fuel_mirror
|
||||
version = 10.0.0
|
||||
summary = The Utility to create local repositories with packages is
|
||||
required for openstack deployment.
|
||||
description-file =
|
||||
README.rst
|
||||
author = Mirantis Inc.
|
||||
author_email = product@mirantis.com
|
||||
url = http://mirantis.com
|
||||
home-page = http://mirantis.com
|
||||
classifier =
|
||||
Development Status :: 4 - Beta
|
||||
Environment :: OpenStack
|
||||
Intended Audience :: Information Technology
|
||||
Intended Audience :: System Administrators
|
||||
License :: OSI Approved :: GNU General Public License v2 (GPLv2)
|
||||
Operating System :: POSIX :: Linux
|
||||
Programming Language :: Python
|
||||
Programming Language :: Python :: 2
|
||||
Programming Language :: Python :: 2.7
|
||||
Programming Language :: Python :: 3
|
||||
Programming Language :: Python :: 3.3
|
||||
Programming Language :: Python :: 3.4
|
||||
Topic :: Utilities
|
||||
|
||||
[files]
|
||||
packages =
|
||||
fuel_mirror
|
||||
data_files =
|
||||
etc/fuel-mirror = etc/*
|
||||
share/fuel-mirror = data/*
|
||||
|
||||
[build_sphinx]
|
||||
source-dir = doc/source
|
||||
build-dir = doc/build
|
||||
all_files = 1
|
||||
|
||||
[entry_points]
|
||||
console_scripts =
|
||||
fuel-mirror=fuel_mirror.app:main
|
||||
|
||||
fuel_mirror =
|
||||
apply=fuel_mirror.commands.apply:ApplyCommand
|
||||
create=fuel_mirror.commands.create:CreateCommand
|
||||
|
||||
[upload_sphinx]
|
||||
upload-dir = doc/build/html
|
||||
|
||||
[compile_catalog]
|
||||
directory = locale
|
||||
domain = fuel_mirror
|
||||
|
||||
[update_catalog]
|
||||
domain = fuel_mirror
|
||||
output_dir = locale
|
||||
input_file = locale/fuel_mirror.pot
|
||||
|
||||
[extract_messages]
|
||||
keywords = _ gettext ngettext l_ lazy_gettext
|
||||
mapping_file = babel.cfg
|
||||
output_file = locale/fuel_mirror.pot
|
||||
|
||||
[global]
|
||||
setup-hooks =
|
||||
pbr.hooks.setup_hook
|
||||
setup_hooks.setup_hook
|
|
@ -1,32 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
|
||||
import setuptools
|
||||
|
||||
# In python < 2.7.4, a lazy loading of package `pbr` will break
|
||||
# setuptools if some other modules registered functions in `atexit`.
|
||||
# solution from: http://bugs.python.org/issue15881#msg170215
|
||||
try:
|
||||
import multiprocessing # noqa
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
setuptools.setup(
|
||||
setup_requires=['pbr'],
|
||||
pbr=True)
|
|
@ -1,25 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
|
||||
def setup_hook(config):
|
||||
import pbr
|
||||
import pbr.packaging
|
||||
|
||||
# this monkey patch is to avoid appending git version to version
|
||||
pbr.packaging._get_version_from_git = lambda pre_version: pre_version
|
|
@ -1,17 +0,0 @@
|
|||
# The order of packages is significant, because pip processes them in the order
|
||||
# of appearance. Changing the order has an impact on the overall integration
|
||||
# process, which may cause wedges in the gate later.
|
||||
|
||||
hacking<0.11,>=0.10.0
|
||||
|
||||
coverage>=3.6
|
||||
discover
|
||||
python-subunit>=0.0.18
|
||||
sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2
|
||||
oslosphinx>=2.5.0 # Apache-2.0
|
||||
oslotest>=1.10.0 # Apache-2.0
|
||||
testrepository>=0.0.18
|
||||
testscenarios>=0.4
|
||||
testtools>=1.4.0
|
||||
cliff>=1.7.0
|
||||
six>=1.5.2
|
|
@ -1,35 +0,0 @@
|
|||
[tox]
|
||||
minversion = 1.6
|
||||
envlist = py34,py27,py26,pep8
|
||||
skipsdist = True
|
||||
|
||||
[testenv]
|
||||
usedevelop = True
|
||||
install_command = pip install -U {opts} {packages}
|
||||
setenv =
|
||||
VIRTUAL_ENV={envdir}
|
||||
deps = -r{toxinidir}/test-requirements.txt
|
||||
commands = python setup.py test --slowest --testr-args='{posargs:fuel_mirror}'
|
||||
|
||||
[testenv:pep8]
|
||||
commands = flake8
|
||||
|
||||
[testenv:venv]
|
||||
commands = {posargs}
|
||||
|
||||
[testenv:cover]
|
||||
commands = python setup.py test --coverage --testr-args='{posargs:fuel_mirror}'
|
||||
|
||||
[testenv:docs]
|
||||
commands = python setup.py build_sphinx
|
||||
|
||||
[testenv:debug]
|
||||
commands = oslo_debug_helper {posargs}
|
||||
|
||||
[flake8]
|
||||
# E123, E125 skipped as they are invalid PEP-8.
|
||||
|
||||
show-source = True
|
||||
ignore = E123,E125
|
||||
builtins = _
|
||||
exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build
|
|
@ -1,17 +0,0 @@
|
|||
fuel-mirror (10.0.0-1) experimental; urgency=medium
|
||||
|
||||
* Bump version to 10.0
|
||||
|
||||
-- Sergey Kulanov <skulanov@mirantis.com> Mon, 21 Mar 2016 13:29:18 +0200
|
||||
|
||||
fuel-mirror (9.0.0-1) experimental; urgency=low
|
||||
|
||||
* Bump version to 9.0
|
||||
|
||||
-- Sergey Kulanov <skulanov@mirantis.com> Thu, 17 Dec 2015 16:39:55 +0200
|
||||
|
||||
fuel-mirror (8.0.0-1) experimental; urgency=low
|
||||
|
||||
* Initial release.
|
||||
|
||||
-- bgaifullin <bgaifullin@mirantis.com> Fri, 27 Nov 2015 00:28:26 +0300
|
|
@ -1 +0,0 @@
|
|||
9
|
|
@ -1,54 +0,0 @@
|
|||
Source: fuel-mirror
|
||||
Section: Utilities
|
||||
Priority: extra
|
||||
Maintainer: Mirantis Product <product@mirantis.com>
|
||||
Build-Depends: debhelper (>= 9),
|
||||
dh-python,
|
||||
openstack-pkg-tools (>= 23~),
|
||||
python-all,
|
||||
python-pbr (>= 0.8),
|
||||
python-setuptools
|
||||
Standards-Version: 3.9.6
|
||||
Homepage: mirantis.com
|
||||
|
||||
Package: fuel-mirror
|
||||
Architecture: all
|
||||
Section: python
|
||||
Depends: python-babel,
|
||||
python-cliff (>= 1.7.0),
|
||||
python-packetary (= ${binary:Version}),
|
||||
python-pbr (>= 0.8),
|
||||
python-six,
|
||||
python-yaml,
|
||||
python-tz,
|
||||
${python:Depends}
|
||||
Recommends: python-fuelclient (>= 7.0.0)
|
||||
Description: Utility to create RPM and DEB mirror
|
||||
Provides two commands fuel-mirror and fuel-createmirror.
|
||||
Second one is for backward compatibility with the previous
|
||||
generation of the utility. These commands could be used
|
||||
to create local copies of MOS and upstream deb and rpm
|
||||
repositories.
|
||||
|
||||
Package: python-packetary
|
||||
Architecture: all
|
||||
Depends: createrepo,
|
||||
python-babel,
|
||||
python-bintrees (>= 2.0.2),
|
||||
python-chardet,
|
||||
python-cliff (>= 1.7.0),
|
||||
python-debian (>= 0.1.21),
|
||||
python-eventlet (>= 0.15),
|
||||
python-lxml,
|
||||
python-pbr (>= 0.8),
|
||||
python-six,
|
||||
python-stevedore (>= 1.1.0),
|
||||
python-tz,
|
||||
${python:Depends}
|
||||
Description: Library allows to build and clone deb and rpm repos
|
||||
Provides object model and API for dealing with deb
|
||||
and rpm repositories. One can use this framework to
|
||||
implement operations like building repository
|
||||
from a set of packages, clone repository, find package
|
||||
dependencies, mix repositories, pull out a subset of
|
||||
packages into a separate repository, etc.
|
|
@ -1,43 +0,0 @@
|
|||
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
|
||||
Upstream-Name: fuel-mirror
|
||||
Source: git://github.com/openstack/fuel-mirror.git
|
||||
|
||||
Files: debian/*
|
||||
Copyright: (c) 2014, Mirantis
|
||||
License: GPL-2
|
||||
|
||||
Files: *
|
||||
Copyright: (c) 2014, Mirantis
|
||||
License: Apache-2
|
||||
|
||||
License: Apache-2
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not
|
||||
use this file except in compliance with the License. You may obtain a copy of
|
||||
the License at:
|
||||
.
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
.
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
License for the specific language governing permissions and limitations under
|
||||
the License.
|
||||
.
|
||||
On Debian-based systems the full text of the Apache version 2.0 license can be
|
||||
found in /usr/share/common-licenses/Apache-2.0.
|
||||
|
||||
License: GPL-2
|
||||
Licensed under the GPL License, Version 2.0 (the "License"); you may not
|
||||
use this file except in compliance with the License. You may obtain a copy of
|
||||
the License at:
|
||||
.
|
||||
http://www.opensource.org/licenses/GPL-2.0
|
||||
.
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
License for the specific language governing permissions and limitations under
|
||||
the License.
|
||||
.
|
||||
On Debian-based systems the full text of the GPL version 2.0 license can be
|
||||
found in /usr/share/common-licenses/GPL-2.
|
|
@ -1,2 +0,0 @@
|
|||
contrib/fuel_mirror/etc/config.yaml /etc/fuel-mirror
|
||||
contrib/fuel_mirror/scripts/fuel-createmirror /usr/bin/
|
|
@ -1,39 +0,0 @@
|
|||
#!/usr/bin/make -f
|
||||
|
||||
PYTHONS:=$(shell pyversions -vr)
|
||||
|
||||
include /usr/share/openstack-pkg-tools/pkgos.make
|
||||
export OSLO_PACKAGE_VERSION=$(shell dpkg-parsechangelog | grep Version: | cut -d' ' -f2 | sed -e 's/^[[:digit:]]*://' -e 's/[-].*//' -e 's/~/.0/' | head -n 1)
|
||||
|
||||
%:
|
||||
dh $@ --buildsystem=python_distutils --with python2
|
||||
|
||||
override_dh_clean:
|
||||
rm -rf build
|
||||
dh_clean -O--buildsystem=python_distutils
|
||||
|
||||
|
||||
override_dh_auto_install:
|
||||
set -e ; for pyvers in $(PYTHONS); do \
|
||||
python$$pyvers setup.py install --install-layout=deb \
|
||||
--root $(CURDIR)/debian/python-packetary; \
|
||||
done
|
||||
set -e ; cd contrib/fuel_mirror/; \
|
||||
for pyvers in $(PYTHONS); do \
|
||||
python$$pyvers ./setup.py install --install-layout=deb \
|
||||
--root $(CURDIR)/debian/fuel-mirror; \
|
||||
done
|
||||
|
||||
override_dh_fixperms:
|
||||
set -e; chmod 755 $(CURDIR)/debian/fuel-mirror/usr/bin/fuel-createmirror
|
||||
|
||||
override_dh_python2:
|
||||
dh_python2 --no-guessing-deps
|
||||
|
||||
override_dh_installcatalogs:
|
||||
override_dh_installemacsen override_dh_installifupdown:
|
||||
override_dh_installinfo override_dh_installmenu override_dh_installmime:
|
||||
override_dh_installmodules override_dh_installlogcheck:
|
||||
override_dh_installpam override_dh_installppp override_dh_installudev override_dh_installwm:
|
||||
override_dh_installxfonts override_dh_gconf override_dh_icons override_dh_perl override_dh_usrlocal:
|
||||
override_dh_installgsettings:
|
|
@ -1 +0,0 @@
|
|||
3.0 (quilt)
|
|
@ -1,79 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.abspath('../..'))
|
||||
# -- General configuration ----------------------------------------------------
|
||||
|
||||
# Add any Sphinx extension module names here, as strings. They can be
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
#'sphinx.ext.intersphinx',
|
||||
'oslosphinx'
|
||||
]
|
||||
|
||||
# autodoc generation is a bit aggressive and a nuisance when doing heavy
|
||||
# text edit cycles.
|
||||
# execute "export SPHINX_DEBUG=1" in your terminal to disable
|
||||
|
||||
# The suffix of source filenames.
|
||||
source_suffix = '.rst'
|
||||
|
||||
# The master toctree document.
|
||||
master_doc = 'index'
|
||||
|
||||
# General information about the project.
|
||||
project = u'packetary'
|
||||
copyright = u'2015, Mirantis Inc.'
|
||||
|
||||
# If true, '()' will be appended to :func: etc. cross-reference text.
|
||||
add_function_parentheses = True
|
||||
|
||||
# If true, the current module name will be prepended to all description
|
||||
# unit titles (such as .. function::).
|
||||
add_module_names = True
|
||||
|
||||
# The name of the Pygments (syntax highlighting) style to use.
|
||||
pygments_style = 'sphinx'
|
||||
|
||||
# -- Options for HTML output --------------------------------------------------
|
||||
|
||||
# The theme to use for HTML and HTML Help pages. Major themes that come with
|
||||
# Sphinx are currently 'default' and 'sphinxdoc'.
|
||||
# html_theme_path = ["."]
|
||||
# html_theme = '_theme'
|
||||
# html_static_path = ['static']
|
||||
|
||||
# Output file base name for HTML help builder.
|
||||
htmlhelp_basename = '%sdoc' % project
|
||||
|
||||
# Grouping the document tree into LaTeX files. List of tuples
|
||||
# (source start file, target name, title, author, documentclass
|
||||
# [howto/manual]).
|
||||
latex_documents = [
|
||||
('index',
|
||||
'%s.tex' % project,
|
||||
u'%s Documentation' % project,
|
||||
u'OpenStack Foundation', 'manual'),
|
||||
]
|
||||
|
||||
# Example configuration for intersphinx: refer to the Python standard library.
|
||||
#intersphinx_mapping = {'http://docs.python.org/': None}
|
|
@ -1,4 +0,0 @@
|
|||
============
|
||||
Contributing
|
||||
============
|
||||
.. include:: ../../CONTRIBUTING.rst
|
|
@ -1,25 +0,0 @@
|
|||
.. packetary documentation master file, created by
|
||||
sphinx-quickstart on Tue Jul 9 22:26:36 2013.
|
||||
You can adapt this file completely to your liking, but it should at least
|
||||
contain the root `toctree` directive.
|
||||
|
||||
Welcome to packetary's documentation!
|
||||
========================================================
|
||||
|
||||
Contents:
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
readme
|
||||
installation
|
||||
usage
|
||||
contributing
|
||||
|
||||
Indices and tables
|
||||
==================
|
||||
|
||||
* :ref:`genindex`
|
||||
* :ref:`modindex`
|
||||
* :ref:`search`
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
============
|
||||
Installation
|
||||
============
|
||||
|
||||
At the command line::
|
||||
|
||||
$ pip install packetary
|
||||
|
||||
Or, if you have virtualenvwrapper installed::
|
||||
|
||||
$ mkvirtualenv packetary
|
||||
$ pip install packetary
|
|
@ -1 +0,0 @@
|
|||
.. include:: ../../README.rst
|
|
@ -1,7 +0,0 @@
|
|||
========
|
||||
Usage
|
||||
========
|
||||
|
||||
To use packetary in a project::
|
||||
|
||||
import packetary
|
|
@ -1,6 +0,0 @@
|
|||
[DEFAULT]
|
||||
|
||||
# The list of modules to copy from oslo-incubator.git
|
||||
|
||||
# The base module to hold the copy of openstack.common
|
||||
base=packetary
|
|
@ -1,33 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import pbr.version
|
||||
|
||||
from packetary.api import Configuration
|
||||
from packetary.api import Context
|
||||
from packetary.api import RepositoryApi
|
||||
|
||||
|
||||
__all__ = [
|
||||
"Configuration",
|
||||
"Context",
|
||||
"RepositoryApi",
|
||||
]
|
||||
|
||||
__version__ = pbr.version.VersionInfo(
|
||||
'packetary').version_string()
|
233
packetary/api.py
233
packetary/api.py
|
@ -1,233 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import logging
|
||||
|
||||
import six
|
||||
|
||||
from packetary.controllers import RepositoryController
|
||||
from packetary.library.connections import ConnectionsManager
|
||||
from packetary.library.executor import AsynchronousSection
|
||||
from packetary.objects import Index
|
||||
from packetary.objects import PackageRelation
|
||||
from packetary.objects import PackagesTree
|
||||
from packetary.objects.statistics import CopyStatistics
|
||||
|
||||
|
||||
logger = logging.getLogger(__package__)
|
||||
|
||||
|
||||
class Configuration(object):
|
||||
"""The configuration holder."""
|
||||
|
||||
def __init__(self, http_proxy=None, https_proxy=None,
|
||||
retries_num=0, threads_num=0,
|
||||
ignore_errors_num=0, retry_interval=0):
|
||||
"""Initialises.
|
||||
|
||||
:param http_proxy: the url of proxy for connections over http,
|
||||
no-proxy will be used if it is not specified
|
||||
:param https_proxy: the url of proxy for connections over https,
|
||||
no-proxy will be used if it is not specified
|
||||
:param retries_num: the number of retries on errors
|
||||
:param retry_interval: the time between retries (in seconds)
|
||||
:param threads_num: the max number of active threads
|
||||
:param ignore_errors_num: the number of errors that may occurs
|
||||
before stop processing
|
||||
"""
|
||||
|
||||
self.http_proxy = http_proxy
|
||||
self.https_proxy = https_proxy
|
||||
self.ignore_errors_num = ignore_errors_num
|
||||
self.retries_num = retries_num
|
||||
self.retry_interval = retry_interval
|
||||
self.threads_num = threads_num
|
||||
|
||||
|
||||
class Context(object):
|
||||
"""The infra-objects holder."""
|
||||
|
||||
def __init__(self, config):
|
||||
"""Initialises.
|
||||
|
||||
:param config: the configuration
|
||||
"""
|
||||
self._connection = ConnectionsManager(
|
||||
proxy=config.http_proxy,
|
||||
secure_proxy=config.https_proxy,
|
||||
retries_num=config.retries_num,
|
||||
retry_interval=config.retry_interval
|
||||
)
|
||||
self._threads_num = config.threads_num
|
||||
self._ignore_errors_num = config.ignore_errors_num
|
||||
|
||||
@property
|
||||
def connection(self):
|
||||
"""Gets the connection."""
|
||||
return self._connection
|
||||
|
||||
def async_section(self, ignore_errors_num=None):
|
||||
"""Gets the execution scope.
|
||||
|
||||
:param ignore_errors_num: custom value for ignore_errors_num,
|
||||
the class value is used if omitted.
|
||||
"""
|
||||
if ignore_errors_num is None:
|
||||
ignore_errors_num = self._ignore_errors_num
|
||||
|
||||
return AsynchronousSection(self._threads_num, ignore_errors_num)
|
||||
|
||||
|
||||
class RepositoryApi(object):
|
||||
"""Provides high-level API to operate with repositories."""
|
||||
|
||||
def __init__(self, controller):
|
||||
"""Initialises.
|
||||
|
||||
:param controller: the repository controller.
|
||||
"""
|
||||
self.controller = controller
|
||||
|
||||
@classmethod
|
||||
def create(cls, config, repotype, repoarch):
|
||||
"""Creates the repository API instance.
|
||||
|
||||
:param config: the configuration
|
||||
:param repotype: the kind of repository(deb, yum, etc)
|
||||
:param repoarch: the architecture of repository (x86_64 or i386)
|
||||
"""
|
||||
context = config if isinstance(config, Context) else Context(config)
|
||||
return cls(RepositoryController.load(context, repotype, repoarch))
|
||||
|
||||
def get_packages(self, origin, debs=None, requirements=None):
|
||||
"""Gets the list of packages from repository(es).
|
||||
|
||||
:param origin: The list of repository`s URLs
|
||||
:param debs: the list of repository`s URL to calculate list of
|
||||
dependencies, that will be used to filter packages.
|
||||
:param requirements: the list of package relations,
|
||||
to resolve the list of mandatory packages.
|
||||
:return: the set of packages
|
||||
"""
|
||||
repositories = self._get_repositories(origin)
|
||||
return self._get_packages(repositories, debs, requirements)
|
||||
|
||||
def clone_repositories(self, origin, destination, debs=None,
|
||||
requirements=None, keep_existing=True,
|
||||
include_source=False, include_locale=False):
|
||||
"""Creates the clones of specified repositories in local folder.
|
||||
|
||||
:param origin: The list of repository`s URLs
|
||||
:param destination: the destination folder path
|
||||
:param debs: the list of repository`s URL to calculate list of
|
||||
dependencies, that will be used to filter packages.
|
||||
:param requirements: the list of package relations,
|
||||
to resolve the list of mandatory packages.
|
||||
:param keep_existing: If False - local packages that does not exist
|
||||
in original repo will be removed.
|
||||
:param include_source: if True, the source packages
|
||||
will be copied as well.
|
||||
:param include_locale: if True, the locales
|
||||
will be copied as well.
|
||||
:return: count of copied and total packages.
|
||||
"""
|
||||
repositories = self._get_repositories(origin)
|
||||
packages = self._get_packages(repositories, debs, requirements)
|
||||
mirrors = self.controller.clone_repositories(
|
||||
repositories, destination, include_source, include_locale
|
||||
)
|
||||
|
||||
package_groups = dict((x, set()) for x in repositories)
|
||||
for pkg in packages:
|
||||
package_groups[pkg.repository].add(pkg)
|
||||
|
||||
stat = CopyStatistics()
|
||||
for repo, packages in six.iteritems(package_groups):
|
||||
mirror = mirrors[repo]
|
||||
logger.info("copy packages from - %s", repo)
|
||||
self.controller.copy_packages(
|
||||
mirror, packages, keep_existing, stat.on_package_copied
|
||||
)
|
||||
return stat
|
||||
|
||||
def get_unresolved_dependencies(self, origin, main=None):
|
||||
"""Gets list of unresolved dependencies for repository(es).
|
||||
|
||||
:param origin: The list of repository`s URLs
|
||||
:param main: The main repository(es) URL
|
||||
:return: list of unresolved dependencies
|
||||
"""
|
||||
packages = PackagesTree()
|
||||
self.controller.load_packages(
|
||||
self._get_repositories(origin),
|
||||
packages.add
|
||||
)
|
||||
|
||||
if main is not None:
|
||||
base = Index()
|
||||
self.controller.load_packages(
|
||||
self._get_repositories(main),
|
||||
base.add
|
||||
)
|
||||
else:
|
||||
base = None
|
||||
|
||||
return packages.get_unresolved_dependencies(base)
|
||||
|
||||
def _get_repositories(self, urls):
|
||||
"""Gets the set of repositories by url."""
|
||||
repositories = set()
|
||||
self.controller.load_repositories(urls, repositories.add)
|
||||
return repositories
|
||||
|
||||
def _get_packages(self, repositories, master, requirements):
|
||||
"""Gets the list of packages according to master and requirements."""
|
||||
if master is None and requirements is None:
|
||||
packages = set()
|
||||
self.controller.load_packages(repositories, packages.add)
|
||||
return packages
|
||||
|
||||
packages = PackagesTree()
|
||||
self.controller.load_packages(repositories, packages.add)
|
||||
if master is not None:
|
||||
main_index = Index()
|
||||
self.controller.load_packages(
|
||||
self._get_repositories(master),
|
||||
main_index.add
|
||||
)
|
||||
else:
|
||||
main_index = None
|
||||
|
||||
return packages.get_minimal_subset(
|
||||
main_index,
|
||||
self._parse_requirements(requirements)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _parse_requirements(requirements):
|
||||
"""Gets the list of relations from requirements.
|
||||
|
||||
:param requirements: the list of requirement in next format:
|
||||
'name [cmp version]|[alt [cmp version]]'
|
||||
"""
|
||||
if requirements is not None:
|
||||
return set(
|
||||
PackageRelation.from_args(
|
||||
*(x.split() for x in r.split("|"))) for r in requirements
|
||||
)
|
||||
return set()
|
|
@ -1,104 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from cliff import app
|
||||
from cliff.commandmanager import CommandManager
|
||||
|
||||
import packetary
|
||||
|
||||
|
||||
class Application(app.App):
|
||||
"""Main cliff application class.
|
||||
|
||||
Performs initialization of the command manager and
|
||||
configuration of basic engines.
|
||||
"""
|
||||
|
||||
def build_option_parser(self, description, version, argparse_kwargs=None):
|
||||
"""Specifies global options."""
|
||||
p_inst = super(Application, self)
|
||||
parser = p_inst.build_option_parser(description=description,
|
||||
version=version,
|
||||
argparse_kwargs=argparse_kwargs)
|
||||
|
||||
parser.add_argument(
|
||||
"--ignore-errors-num",
|
||||
type=int,
|
||||
default=2,
|
||||
metavar="NUMBER",
|
||||
help="The number of errors that can be ignored."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--retries-num",
|
||||
type=int,
|
||||
default=5,
|
||||
metavar="NUMBER",
|
||||
help="The number of retries."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--retry-interval",
|
||||
type=int,
|
||||
default=2,
|
||||
metavar="SECONDS",
|
||||
help="The minimal time between retries in seconds."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--threads-num",
|
||||
default=3,
|
||||
type=int,
|
||||
metavar="NUMBER",
|
||||
help="The number of threads."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--http-proxy",
|
||||
default=None,
|
||||
metavar="http://username:password@proxy_host:proxy_port",
|
||||
help="The URL of http proxy."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--https-proxy",
|
||||
default=None,
|
||||
metavar="https://username:password@proxy_host:proxy_port",
|
||||
help="The URL of https proxy."
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
return Application(
|
||||
description="The utility manages packages and repositories.",
|
||||
version=packetary.__version__,
|
||||
command_manager=CommandManager("packetary", convert_underscores=True)
|
||||
).run(argv)
|
||||
|
||||
|
||||
def debug(name, cmd_class, argv=None):
|
||||
"""Helper for debugging single command without package installation."""
|
||||
import sys
|
||||
|
||||
if argv is None:
|
||||
argv = sys.argv[1:]
|
||||
|
||||
argv = [name] + argv + ["-v", "-v", "--debug"]
|
||||
cmd_mgr = CommandManager("test_packetary", convert_underscores=True)
|
||||
cmd_mgr.add_command(name, cmd_class)
|
||||
return Application(
|
||||
description="The utility manages packages and repositories.",
|
||||
version="0.0.1",
|
||||
command_manager=cmd_mgr
|
||||
).run(argv)
|
|
@ -1,185 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import abc
|
||||
|
||||
from cliff import command
|
||||
import six
|
||||
|
||||
from packetary.cli.commands.utils import make_display_attr_getter
|
||||
from packetary.cli.commands.utils import read_lines_from_file
|
||||
from packetary import RepositoryApi
|
||||
|
||||
|
||||
@six.add_metaclass(abc.ABCMeta)
|
||||
class BaseRepoCommand(command.Command):
|
||||
"""Super class for packetary commands."""
|
||||
|
||||
@property
|
||||
def stdout(self):
|
||||
"""Shortcut for self.app.stdout."""
|
||||
return self.app.stdout
|
||||
|
||||
def get_parser(self, prog_name):
|
||||
"""Specifies common options."""
|
||||
parser = super(BaseRepoCommand, self).get_parser(prog_name)
|
||||
parser.add_argument(
|
||||
'-t',
|
||||
'--type',
|
||||
type=str,
|
||||
choices=['deb', 'rpm'],
|
||||
metavar='TYPE',
|
||||
default='deb',
|
||||
help='The type of repository.')
|
||||
|
||||
parser.add_argument(
|
||||
'-a',
|
||||
'--arch',
|
||||
type=str,
|
||||
choices=["x86_64", "i386"],
|
||||
metavar='ARCHITECTURE',
|
||||
default="x86_64",
|
||||
help='The target architecture.')
|
||||
|
||||
origin_gr = parser.add_mutually_exclusive_group(required=True)
|
||||
origin_gr.add_argument(
|
||||
'-o', '--origin-url',
|
||||
nargs="+",
|
||||
dest='origins',
|
||||
type=six.text_type,
|
||||
metavar='URL',
|
||||
help='Space separated list of URLs of origin repositories.')
|
||||
|
||||
origin_gr.add_argument(
|
||||
'-O', '--origin-file',
|
||||
type=read_lines_from_file,
|
||||
dest='origins',
|
||||
metavar='FILENAME',
|
||||
help='The path to file with URLs of origin repositories.')
|
||||
|
||||
return parser
|
||||
|
||||
def take_action(self, parsed_args):
|
||||
"""See the Command.take_action.
|
||||
|
||||
:param parsed_args: the command-line arguments
|
||||
:return: the result of take_repo_action
|
||||
:rtype: object
|
||||
"""
|
||||
return self.take_repo_action(
|
||||
RepositoryApi.create(
|
||||
self.app_args, parsed_args.type, parsed_args.arch
|
||||
),
|
||||
parsed_args
|
||||
)
|
||||
|
||||
@abc.abstractmethod
|
||||
def take_repo_action(self, api, parsed_args):
|
||||
"""Takes action on repository.
|
||||
|
||||
:param api: the RepositoryApi instance
|
||||
:param parsed_args: the command-line arguments
|
||||
:return: the action result
|
||||
"""
|
||||
|
||||
|
||||
class BaseProduceOutputCommand(BaseRepoCommand):
|
||||
columns = None
|
||||
|
||||
def get_parser(self, prog_name):
|
||||
parser = super(BaseProduceOutputCommand, self).get_parser(prog_name)
|
||||
|
||||
group = parser.add_argument_group(
|
||||
title='output formatter',
|
||||
description='output formatter options',
|
||||
)
|
||||
group.add_argument(
|
||||
'-c', '--column',
|
||||
nargs='+',
|
||||
choices=self.columns,
|
||||
dest='columns',
|
||||
metavar='COLUMN',
|
||||
default=[],
|
||||
help='Space separated list of columns to include.',
|
||||
)
|
||||
group.add_argument(
|
||||
'-s',
|
||||
'--sort-columns',
|
||||
type=str,
|
||||
nargs='+',
|
||||
choices=self.columns,
|
||||
metavar='SORT_COLUMN',
|
||||
default=[self.columns[0]],
|
||||
help='Space separated list of keys for sorting '
|
||||
'the data.'
|
||||
)
|
||||
group.add_argument(
|
||||
'--sep',
|
||||
type=six.text_type,
|
||||
metavar='ROW SEPARATOR',
|
||||
default=six.text_type('; '),
|
||||
help='The row separator.'
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
def produce_output(self, parsed_args, data):
|
||||
indexes = dict(
|
||||
(c, i) for i, c in enumerate(self.columns)
|
||||
)
|
||||
sort_index = [indexes[c] for c in parsed_args.sort_columns]
|
||||
if isinstance(data, list):
|
||||
data.sort(key=lambda x: [x[i] for i in sort_index])
|
||||
else:
|
||||
data = sorted(data, key=lambda x: [x[i] for i in sort_index])
|
||||
|
||||
if parsed_args.columns:
|
||||
include_index = [
|
||||
indexes[c] for c in parsed_args.columns
|
||||
]
|
||||
data = ((row[i] for i in include_index) for row in data)
|
||||
columns = parsed_args.columns
|
||||
else:
|
||||
columns = self.columns
|
||||
|
||||
stdout = self.stdout
|
||||
sep = parsed_args.sep
|
||||
|
||||
# header
|
||||
stdout.write("# ")
|
||||
stdout.write(sep.join(columns))
|
||||
stdout.write("\n")
|
||||
|
||||
for row in data:
|
||||
stdout.write(sep.join(row))
|
||||
stdout.write("\n")
|
||||
|
||||
def run(self, parsed_args):
|
||||
# Use custom output producer.
|
||||
# cliff.lister with default formatters does not work
|
||||
# with large arrays of data, because it does not support streaming
|
||||
# TODO(implement custom formatter)
|
||||
|
||||
formatter = make_display_attr_getter(self.columns)
|
||||
data = six.moves.map(formatter, self.take_action(parsed_args))
|
||||
self.produce_output(parsed_args, data)
|
||||
return 0
|
||||
|
||||
@abc.abstractmethod
|
||||
def take_repo_action(self, driver, parsed_args):
|
||||
"""See Command.take_repo_action."""
|
|
@ -1,115 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from packetary.cli.commands.base import BaseRepoCommand
|
||||
from packetary.cli.commands.utils import read_lines_from_file
|
||||
|
||||
|
||||
class CloneCommand(BaseRepoCommand):
|
||||
"""Clones the specified repository to local folder."""
|
||||
|
||||
def get_parser(self, prog_name):
|
||||
parser = super(CloneCommand, self).get_parser(prog_name)
|
||||
|
||||
parser.add_argument(
|
||||
"-d", "--destination",
|
||||
required=True,
|
||||
help="The path to the destination folder."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--clean",
|
||||
dest="keep_existing",
|
||||
action='store_false',
|
||||
default=True,
|
||||
help="Remove packages that does not exist in origin repo."
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--sources",
|
||||
action='store_true',
|
||||
default=False,
|
||||
help="Also copy source packages."
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--locales",
|
||||
action='store_true',
|
||||
default=False,
|
||||
help="Also copy localisation files."
|
||||
)
|
||||
|
||||
bootstrap_group = parser.add_mutually_exclusive_group(required=False)
|
||||
bootstrap_group.add_argument(
|
||||
"-b", "--bootstrap",
|
||||
nargs='+',
|
||||
dest='bootstrap',
|
||||
metavar='PACKAGE [OP VERSION]',
|
||||
help="Space separated list of package relations, "
|
||||
"to resolve the list of mandatory packages."
|
||||
)
|
||||
bootstrap_group.add_argument(
|
||||
"-B", "--bootstrap-file",
|
||||
type=read_lines_from_file,
|
||||
dest='bootstrap',
|
||||
metavar='FILENAME',
|
||||
help="Path to the file with list of package relations, "
|
||||
"to resolve the list of mandatory packages."
|
||||
)
|
||||
|
||||
requires_group = parser.add_mutually_exclusive_group(required=False)
|
||||
requires_group.add_argument(
|
||||
'-r', '--requires-url',
|
||||
nargs="+",
|
||||
dest='requires',
|
||||
metavar='URL',
|
||||
help="Space separated list of repository`s URL to calculate list "
|
||||
"of dependencies, that will be used to filter packages")
|
||||
|
||||
requires_group.add_argument(
|
||||
'-R', '--requires-file',
|
||||
type=read_lines_from_file,
|
||||
dest='requires',
|
||||
metavar='FILENAME',
|
||||
help="The path to the file with list of repository`s URL "
|
||||
"to calculate list of dependencies, "
|
||||
"that will be used to filter packages")
|
||||
return parser
|
||||
|
||||
def take_repo_action(self, api, parsed_args):
|
||||
stat = api.clone_repositories(
|
||||
parsed_args.origins,
|
||||
parsed_args.destination,
|
||||
parsed_args.requires,
|
||||
parsed_args.bootstrap,
|
||||
parsed_args.keep_existing,
|
||||
parsed_args.sources,
|
||||
parsed_args.locales
|
||||
)
|
||||
self.stdout.write(
|
||||
"Packages copied: {0.copied}/{0.total}.\n".format(stat)
|
||||
)
|
||||
|
||||
|
||||
def debug(argv=None):
|
||||
"""Helper to debug the Clone command."""
|
||||
from packetary.cli.app import debug
|
||||
debug("clone", CloneCommand, argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
debug()
|
|
@ -1,93 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from packetary.cli.commands.base import BaseProduceOutputCommand
|
||||
from packetary.cli.commands.utils import read_lines_from_file
|
||||
|
||||
|
||||
class ListOfPackages(BaseProduceOutputCommand):
|
||||
"""Gets the list of packages from repository(es)."""
|
||||
|
||||
columns = (
|
||||
"name",
|
||||
"repository",
|
||||
"version",
|
||||
"filename",
|
||||
"filesize",
|
||||
"checksum",
|
||||
"obsoletes",
|
||||
"provides",
|
||||
"requires",
|
||||
)
|
||||
|
||||
def get_parser(self, prog_name):
|
||||
parser = super(ListOfPackages, self).get_parser(prog_name)
|
||||
|
||||
bootstrap_group = parser.add_mutually_exclusive_group(required=False)
|
||||
bootstrap_group.add_argument(
|
||||
"-b", "--bootstrap",
|
||||
nargs='+',
|
||||
dest='bootstrap',
|
||||
metavar='PACKAGE [OP VERSION]',
|
||||
help="Space separated list of package relations, "
|
||||
"to resolve the list of mandatory packages."
|
||||
)
|
||||
bootstrap_group.add_argument(
|
||||
"-B", "--bootstrap-file",
|
||||
type=read_lines_from_file,
|
||||
dest='bootstrap',
|
||||
metavar='FILENAME',
|
||||
help="Path to the file with list of package relations, "
|
||||
"to resolve the list of mandatory packages."
|
||||
)
|
||||
|
||||
requires_group = parser.add_mutually_exclusive_group(required=False)
|
||||
requires_group.add_argument(
|
||||
'-r', '--requires-url',
|
||||
nargs="+",
|
||||
dest='requires',
|
||||
metavar='URL',
|
||||
help="Space separated list of repository`s URL to calculate list "
|
||||
"of dependencies, that will be used to filter packages")
|
||||
|
||||
requires_group.add_argument(
|
||||
'-R', '--requires-file',
|
||||
type=read_lines_from_file,
|
||||
dest='requires',
|
||||
metavar='FILENAME',
|
||||
help="The path to the file with list of repository`s URL "
|
||||
"to calculate list of dependencies, "
|
||||
"that will be used to filter packages")
|
||||
return parser
|
||||
|
||||
def take_repo_action(self, api, parsed_args):
|
||||
return api.get_packages(
|
||||
parsed_args.origins,
|
||||
parsed_args.requires,
|
||||
parsed_args.bootstrap,
|
||||
)
|
||||
|
||||
|
||||
def debug(argv=None):
|
||||
"""Helper to debug the ListOfPackages command."""
|
||||
from packetary.cli.app import debug
|
||||
debug("packages", ListOfPackages, argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
debug()
|
|
@ -1,68 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from packetary.cli.commands.base import BaseProduceOutputCommand
|
||||
from packetary.cli.commands.utils import read_lines_from_file
|
||||
|
||||
|
||||
class ListOfUnresolved(BaseProduceOutputCommand):
|
||||
"""Gets the list of external dependencies for repository(es)."""
|
||||
|
||||
columns = (
|
||||
"name",
|
||||
"version",
|
||||
"alternative",
|
||||
)
|
||||
|
||||
def get_parser(self, prog_name):
|
||||
parser = super(ListOfUnresolved, self).get_parser(prog_name)
|
||||
main_group = parser.add_mutually_exclusive_group(required=False)
|
||||
main_group.add_argument(
|
||||
'-m', '--main-url',
|
||||
nargs="+",
|
||||
dest='main',
|
||||
metavar='URL',
|
||||
help='Space separated list of URLs of repository(es) '
|
||||
' that are used to resolve dependencies.')
|
||||
|
||||
main_group.add_argument(
|
||||
'-M', '--main-file',
|
||||
type=read_lines_from_file,
|
||||
dest='main',
|
||||
metavar='FILENAME',
|
||||
help='The path to the file, that contains '
|
||||
'list of URLs of repository(es) '
|
||||
' that are used to resolve dependencies.')
|
||||
return parser
|
||||
|
||||
def take_repo_action(self, api, parsed_args):
|
||||
return api.get_unresolved_dependencies(
|
||||
parsed_args.origins,
|
||||
parsed_args.main,
|
||||
)
|
||||
|
||||
|
||||
def debug(argv=None):
|
||||
"""Helper to debug the ListOfUnresolved command."""
|
||||
|
||||
from packetary.cli.app import debug
|
||||
debug("unresolved", ListOfUnresolved, argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
debug()
|
|
@ -1,72 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import operator
|
||||
|
||||
import six
|
||||
|
||||
|
||||
def read_lines_from_file(filename):
|
||||
"""Reads lines from file.
|
||||
|
||||
Note: the line starts with '#' will be skipped.
|
||||
|
||||
:param filename: the path of target file
|
||||
:return: the list of lines from file
|
||||
"""
|
||||
with open(filename, 'r') as f:
|
||||
return [
|
||||
x
|
||||
for x in six.moves.map(operator.methodcaller("strip"), f)
|
||||
if x and not x.startswith("#")
|
||||
]
|
||||
|
||||
|
||||
def get_object_attrs(obj, attrs):
|
||||
"""Gets object attributes as list.
|
||||
|
||||
:param obj: the target object
|
||||
:param attrs: the list of attributes
|
||||
:return: list of values from specified attributes.
|
||||
"""
|
||||
return [getattr(obj, f) for f in attrs]
|
||||
|
||||
|
||||
def get_display_value(value):
|
||||
"""Get the displayable string for value.
|
||||
|
||||
:param value: the target value
|
||||
:return: the displayable string for value
|
||||
"""
|
||||
if value is None:
|
||||
return u"-"
|
||||
|
||||
if isinstance(value, list):
|
||||
return u", ".join(six.text_type(x) for x in value)
|
||||
return six.text_type(value)
|
||||
|
||||
|
||||
def make_display_attr_getter(attrs):
|
||||
"""Gets formatter to convert attributes of object in displayable format.
|
||||
|
||||
:param attrs: the list of attributes
|
||||
:return: the formatter (callable object)
|
||||
"""
|
||||
return lambda x: [
|
||||
get_display_value(v) for v in get_object_attrs(x, attrs)
|
||||
]
|
|
@ -1,23 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from packetary.controllers.repository import RepositoryController
|
||||
|
||||
__all__ = [
|
||||
"RepositoryController"
|
||||
]
|
|
@ -1,171 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
import six
|
||||
import stevedore
|
||||
|
||||
|
||||
logger = logging.getLogger(__package__)
|
||||
|
||||
urljoin = six.moves.urllib.parse.urljoin
|
||||
|
||||
|
||||
class RepositoryController(object):
|
||||
"""Implements low-level functionality to communicate with drivers."""
|
||||
|
||||
_drivers = None
|
||||
|
||||
def __init__(self, context, driver, arch):
|
||||
self.context = context
|
||||
self.driver = driver
|
||||
self.arch = arch
|
||||
|
||||
@classmethod
|
||||
def load(cls, context, driver_name, repoarch):
|
||||
"""Creates the repository manager.
|
||||
|
||||
:param context: the context
|
||||
:param driver_name: the name of required driver
|
||||
:param repoarch: the architecture of repository (x86_64 or i386)
|
||||
"""
|
||||
if cls._drivers is None:
|
||||
cls._drivers = stevedore.ExtensionManager(
|
||||
"packetary.drivers", invoke_on_load=True
|
||||
)
|
||||
try:
|
||||
driver = cls._drivers[driver_name].obj
|
||||
except KeyError:
|
||||
raise NotImplementedError(
|
||||
"The driver {0} is not supported yet.".format(driver_name)
|
||||
)
|
||||
return cls(context, driver, repoarch)
|
||||
|
||||
def load_repositories(self, urls, consumer):
|
||||
"""Loads the repository objects from url.
|
||||
|
||||
:param urls: the list of repository urls.
|
||||
:param consumer: the callback to consume objects
|
||||
"""
|
||||
if isinstance(urls, six.string_types):
|
||||
urls = [urls]
|
||||
|
||||
connection = self.context.connection
|
||||
for parsed_url in self.driver.parse_urls(urls):
|
||||
self.driver.get_repository(
|
||||
connection, parsed_url, self.arch, consumer
|
||||
)
|
||||
|
||||
def load_packages(self, repositories, consumer):
|
||||
"""Loads packages from repository.
|
||||
|
||||
:param repositories: the repository object
|
||||
:param consumer: the callback to consume objects
|
||||
"""
|
||||
connection = self.context.connection
|
||||
for r in repositories:
|
||||
self.driver.get_packages(connection, r, consumer)
|
||||
|
||||
def assign_packages(self, repository, packages, keep_existing=True):
|
||||
"""Assigns new packages to the repository.
|
||||
|
||||
It replaces the current repository`s packages.
|
||||
|
||||
:param repository: the target repository
|
||||
:param packages: the set of new packages
|
||||
:param keep_existing:
|
||||
if True, all existing packages will be kept as is.
|
||||
if False, all existing packages, that are not included
|
||||
to new packages will be removed.
|
||||
"""
|
||||
|
||||
if not isinstance(packages, set):
|
||||
packages = set(packages)
|
||||
else:
|
||||
packages = packages.copy()
|
||||
|
||||
if keep_existing:
|
||||
consume_exist = packages.add
|
||||
else:
|
||||
def consume_exist(package):
|
||||
if package not in packages:
|
||||
filepath = os.path.join(
|
||||
package.repository.url, package.filename
|
||||
)
|
||||
logger.info("remove package - %s.", filepath)
|
||||
os.remove(filepath)
|
||||
|
||||
self.driver.get_packages(
|
||||
self.context.connection, repository, consume_exist
|
||||
)
|
||||
self.driver.rebuild_repository(repository, packages)
|
||||
|
||||
def copy_packages(self, repository, packages, keep_existing, observer):
|
||||
"""Copies packages to repository.
|
||||
|
||||
:param repository: the target repository
|
||||
:param packages: the set of packages
|
||||
:param keep_existing: see assign_packages for more details
|
||||
:param observer: the package copying process observer
|
||||
"""
|
||||
with self.context.async_section() as section:
|
||||
for package in packages:
|
||||
section.execute(
|
||||
self._copy_package, repository, package, observer
|
||||
)
|
||||
self.assign_packages(repository, packages, keep_existing)
|
||||
|
||||
def clone_repositories(self, repositories, destination,
|
||||
source=False, locale=False):
|
||||
"""Creates copy of repositories.
|
||||
|
||||
:param repositories: the origin repositories
|
||||
:param destination: the target folder
|
||||
:param source: If True, the source packages will be copied too.
|
||||
:param locale: If True, the localisation will be copied too.
|
||||
:return: the mapping origin to cloned repository.
|
||||
"""
|
||||
mirros = dict()
|
||||
destination = os.path.abspath(destination)
|
||||
with self.context.async_section(0) as section:
|
||||
for r in repositories:
|
||||
section.execute(
|
||||
self._fork_repository,
|
||||
r, destination, source, locale, mirros
|
||||
)
|
||||
return mirros
|
||||
|
||||
def _fork_repository(self, r, destination, source, locale, mirrors):
|
||||
"""Creates clone of repository and stores it in mirrors."""
|
||||
new_repository = self.driver.fork_repository(
|
||||
self.context.connection, r, destination, source, locale
|
||||
)
|
||||
mirrors[r] = new_repository
|
||||
|
||||
def _copy_package(self, target, package, observer):
|
||||
"""Synchronises remote file to local fs."""
|
||||
dst_path = os.path.join(target.url, package.filename)
|
||||
src_path = urljoin(package.repository.url, package.filename)
|
||||
bytes_copied = self.context.connection.retrieve(
|
||||
src_path, dst_path, size=package.filesize
|
||||
)
|
||||
if package.filesize < 0:
|
||||
package.filesize = bytes_copied
|
||||
observer(bytes_copied)
|
|
@ -1,82 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import abc
|
||||
import logging
|
||||
|
||||
import six
|
||||
|
||||
|
||||
@six.add_metaclass(abc.ABCMeta)
|
||||
class RepositoryDriverBase(object):
|
||||
"""The super class for Repository Drivers.
|
||||
|
||||
For implementing support of new type of repository:
|
||||
- inherit this class
|
||||
- implement all abstract methods
|
||||
- register implementation in 'packetary.drivers' namespace
|
||||
"""
|
||||
def __init__(self):
|
||||
self.logger = logging.getLogger(__package__)
|
||||
|
||||
@abc.abstractmethod
|
||||
def parse_urls(self, urls):
|
||||
"""Parses the repository url.
|
||||
|
||||
:return: the sequence of parsed urls
|
||||
"""
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_repository(self, connection, url, arch, consumer):
|
||||
"""Loads the repository meta information from URL.
|
||||
|
||||
:param connection: the connection manager instance
|
||||
:param url: the repository`s url
|
||||
:param arch: the repository`s architecture
|
||||
:param consumer: the callback to consume result
|
||||
"""
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_packages(self, connection, repository, consumer):
|
||||
"""Loads packages from repository.
|
||||
|
||||
:param connection: the connection manager instance
|
||||
:param repository: the repository object
|
||||
:param consumer: the callback to consume result
|
||||
"""
|
||||
|
||||
@abc.abstractmethod
|
||||
def fork_repository(self, connection, repository, destination,
|
||||
source=False, locale=False):
|
||||
"""Creates the new repository with same metadata.
|
||||
|
||||
:param connection: the connection manager instance
|
||||
:param repository: the source repository
|
||||
:param destination: the destination folder
|
||||
:param source: copy source files
|
||||
:param locale: copy localisation
|
||||
:return: The copy of repository
|
||||
"""
|
||||
|
||||
@abc.abstractmethod
|
||||
def rebuild_repository(self, repository, packages):
|
||||
"""Re-builds the repository.
|
||||
|
||||
:param repository: the target repository
|
||||
:param packages: the set of packages
|
||||
"""
|
|
@ -1,374 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from contextlib import closing
|
||||
import copy
|
||||
import datetime
|
||||
import fcntl
|
||||
import gzip
|
||||
import os
|
||||
|
||||
from debian import deb822
|
||||
from debian import debfile
|
||||
from debian.debian_support import Version
|
||||
import six
|
||||
|
||||
from packetary.drivers.base import RepositoryDriverBase
|
||||
from packetary.library.checksum import composite as checksum_composite
|
||||
from packetary.library.streams import GzipDecompress
|
||||
from packetary.library import utils
|
||||
from packetary.objects import FileChecksum
|
||||
from packetary.objects import Package
|
||||
from packetary.objects import PackageRelation
|
||||
from packetary.objects import Repository
|
||||
|
||||
|
||||
_OPERATORS_MAPPING = {
|
||||
'>>': 'gt',
|
||||
'<<': 'lt',
|
||||
'=': 'eq',
|
||||
'>=': 'ge',
|
||||
'<=': 'le',
|
||||
}
|
||||
|
||||
_ARCHITECTURES = {
|
||||
"x86_64": "amd64",
|
||||
"i386": "i386",
|
||||
"source": "Source",
|
||||
"amd64": "x86_64",
|
||||
}
|
||||
|
||||
_PRIORITIES = {
|
||||
"required": 1,
|
||||
"important": 2,
|
||||
"standard": 3,
|
||||
"optional": 4,
|
||||
"extra": 5
|
||||
}
|
||||
|
||||
# Order is important
|
||||
_REPOSITORY_FILES = [
|
||||
"Packages",
|
||||
"Release",
|
||||
"Packages.gz"
|
||||
]
|
||||
|
||||
# TODO(should be configurable)
|
||||
_MANDATORY_PRIORITY = 3
|
||||
|
||||
_CHECKSUM_METHODS = (
|
||||
"MD5Sum",
|
||||
"SHA1",
|
||||
"SHA256"
|
||||
)
|
||||
|
||||
_checksum_collector = checksum_composite('md5', 'sha1', 'sha256')
|
||||
|
||||
|
||||
class DebRepositoryDriver(RepositoryDriverBase):
|
||||
def parse_urls(self, urls):
|
||||
"""Overrides method of superclass."""
|
||||
for url in urls:
|
||||
try:
|
||||
tokens = iter(x for x in url.split(" ") if x)
|
||||
base, suite = next(tokens), next(tokens)
|
||||
components = list(tokens)
|
||||
except StopIteration:
|
||||
raise ValueError("Invalid url: {0}".format(url))
|
||||
|
||||
base = base.rstrip("/")
|
||||
if base.endswith("/dists"):
|
||||
base = base[:-6]
|
||||
|
||||
# TODO(Flat Repository Format[1])
|
||||
# [1] https://wiki.debian.org/RepositoryFormat
|
||||
for component in components:
|
||||
yield (base, suite, component)
|
||||
|
||||
def get_repository(self, connection, url, arch, consumer):
|
||||
"""Overrides method of superclass."""
|
||||
|
||||
base, suite, component = url
|
||||
release = self._get_url_of_metafile(
|
||||
(base, suite, component, arch), "Release"
|
||||
)
|
||||
deb_release = deb822.Release(connection.open_stream(release))
|
||||
consumer(Repository(
|
||||
name=(deb_release["Archive"], deb_release["Component"]),
|
||||
architecture=arch,
|
||||
origin=deb_release["origin"],
|
||||
url=base + "/"
|
||||
))
|
||||
|
||||
def get_packages(self, connection, repository, consumer):
|
||||
"""Overrides method of superclass."""
|
||||
index = self._get_url_of_metafile(repository, "Packages.gz")
|
||||
stream = GzipDecompress(connection.open_stream(index))
|
||||
self.logger.info("loading packages from %s ...", repository)
|
||||
pkg_iter = deb822.Packages.iter_paragraphs(stream)
|
||||
counter = 0
|
||||
for dpkg in pkg_iter:
|
||||
try:
|
||||
consumer(Package(
|
||||
repository=repository,
|
||||
name=dpkg["package"],
|
||||
version=Version(dpkg['version']),
|
||||
filesize=int(dpkg.get('size', -1)),
|
||||
filename=dpkg["filename"],
|
||||
checksum=FileChecksum(
|
||||
md5=dpkg.get("md5sum"),
|
||||
sha1=dpkg.get("sha1"),
|
||||
sha256=dpkg.get("sha256"),
|
||||
),
|
||||
mandatory=self._is_mandatory(dpkg),
|
||||
# Recommends are installed by default (since Lucid)
|
||||
requires=self._get_relations(
|
||||
dpkg, "depends", "pre-depends", "recommends"
|
||||
),
|
||||
# The deb does not have obsoletes section
|
||||
obsoletes=[],
|
||||
provides=self._get_relations(dpkg, "provides"),
|
||||
))
|
||||
except KeyError as e:
|
||||
self.logger.error(
|
||||
"Malformed index %s - %s: %s",
|
||||
repository, six.text_type(dpkg), six.text_type(e)
|
||||
)
|
||||
raise
|
||||
counter += 1
|
||||
|
||||
self.logger.info("loaded: %d packages from %s.", counter, repository)
|
||||
|
||||
def rebuild_repository(self, repository, packages):
|
||||
"""Overrides method of superclass."""
|
||||
basedir = utils.get_path_from_url(repository.url)
|
||||
index_file = utils.get_path_from_url(
|
||||
self._get_url_of_metafile(repository, "Packages")
|
||||
)
|
||||
utils.ensure_dir_exist(os.path.dirname(index_file))
|
||||
index_gz = index_file + ".gz"
|
||||
count = 0
|
||||
with open(index_file, "wb") as fd1:
|
||||
with closing(gzip.open(index_gz, "wb")) as fd2:
|
||||
writer = utils.composite_writer(fd1, fd2)
|
||||
for pkg in packages:
|
||||
filename = os.path.join(basedir, pkg.filename)
|
||||
with closing(debfile.DebFile(filename)) as deb:
|
||||
debcontrol = deb.debcontrol()
|
||||
debcontrol.setdefault("Origin", repository.origin)
|
||||
debcontrol["Size"] = str(pkg.filesize)
|
||||
debcontrol["Filename"] = pkg.filename
|
||||
for k, v in six.moves.zip(_CHECKSUM_METHODS, pkg.checksum):
|
||||
debcontrol[k] = v
|
||||
writer(debcontrol.dump())
|
||||
writer("\n")
|
||||
count += 1
|
||||
self.logger.info("saved %d packages in %s", count, repository)
|
||||
self._update_suite_index(repository)
|
||||
|
||||
def fork_repository(self, connection, repository, destination,
|
||||
source=False, locale=False):
|
||||
# TODO(download gpk)
|
||||
# TODO(sources and locales)
|
||||
new_repo = copy.copy(repository)
|
||||
new_repo.url = utils.localize_repo_url(destination, repository.url)
|
||||
packages_file = utils.get_path_from_url(
|
||||
self._get_url_of_metafile(new_repo, "Packages")
|
||||
)
|
||||
release_file = utils.get_path_from_url(
|
||||
self._get_url_of_metafile(new_repo, "Release")
|
||||
)
|
||||
self.logger.info(
|
||||
"clone repository %s to %s", repository, new_repo.url
|
||||
)
|
||||
utils.ensure_dir_exist(os.path.dirname(release_file))
|
||||
|
||||
release = deb822.Release()
|
||||
release["Origin"] = repository.origin
|
||||
release["Label"] = repository.origin
|
||||
release["Archive"] = repository.name[0]
|
||||
release["Component"] = repository.name[1]
|
||||
release["Architecture"] = _ARCHITECTURES[repository.architecture]
|
||||
with open(release_file, "wb") as fd:
|
||||
release.dump(fd)
|
||||
|
||||
open(packages_file, "ab").close()
|
||||
gzip.open(packages_file + ".gz", "ab").close()
|
||||
return new_repo
|
||||
|
||||
def _update_suite_index(self, repository):
|
||||
"""Updates the Release file in the suite."""
|
||||
path = os.path.join(
|
||||
utils.get_path_from_url(repository.url),
|
||||
"dists", repository.name[0]
|
||||
)
|
||||
release_path = os.path.join(path, "Release")
|
||||
self.logger.info(
|
||||
"added repository suite release file: %s", release_path
|
||||
)
|
||||
with open(release_path, "a+b") as fd:
|
||||
fcntl.flock(fd.fileno(), fcntl.LOCK_EX)
|
||||
try:
|
||||
fd.seek(0)
|
||||
release = deb822.Release(fd)
|
||||
self._add_to_release(release, repository)
|
||||
for m in _CHECKSUM_METHODS:
|
||||
release.setdefault(m, [])
|
||||
|
||||
self._add_files_to_release(
|
||||
release, path, self._get_metafiles(repository)
|
||||
)
|
||||
|
||||
fd.truncate(0)
|
||||
release.dump(fd)
|
||||
finally:
|
||||
fcntl.flock(fd.fileno(), fcntl.LOCK_UN)
|
||||
|
||||
def _get_relations(self, dpkg, *names):
|
||||
"""Gets the package relations.
|
||||
|
||||
:param dpkg: the debian-package object
|
||||
:type dpkg: deb822.Packages
|
||||
:param names: the relation names
|
||||
:return: the list of PackageRelation objects
|
||||
"""
|
||||
relations = list()
|
||||
for name in names:
|
||||
for variants in dpkg.relations[name]:
|
||||
relation = PackageRelation.from_args(
|
||||
*(self._unparse_relation(v) for v in variants)
|
||||
)
|
||||
if relation is not None:
|
||||
relations.append(relation)
|
||||
return relations
|
||||
|
||||
def _get_metafiles(self, repository):
|
||||
"""Gets the sequence of metafiles for repository."""
|
||||
return (
|
||||
utils.get_path_from_url(
|
||||
self._get_url_of_metafile(repository, filename)
|
||||
)
|
||||
for filename in _REPOSITORY_FILES
|
||||
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _unparse_relation(relation):
|
||||
"""Gets the relation parameters.
|
||||
|
||||
:param relation: the deb822.Releation object
|
||||
:return: tuple(name, version_compare, version_edge)
|
||||
"""
|
||||
name = relation['name']
|
||||
version = relation.get("version")
|
||||
if version is None:
|
||||
return name, None
|
||||
else:
|
||||
return name, _OPERATORS_MAPPING[version[0]], version[1]
|
||||
|
||||
@staticmethod
|
||||
def _is_mandatory(dpkg):
|
||||
"""Checks that package is mandatory.
|
||||
|
||||
:param dpkg: the debian-package object
|
||||
:type dpkg: deb822.Packages
|
||||
"""
|
||||
if dpkg.get("essential") == "yes":
|
||||
return True
|
||||
|
||||
return _PRIORITIES.get(
|
||||
dpkg.get("priority"), _MANDATORY_PRIORITY + 1
|
||||
) < _MANDATORY_PRIORITY
|
||||
|
||||
@staticmethod
|
||||
def _get_url_of_metafile(repo_or_comps, filename):
|
||||
"""Gets the URL of meta-file.
|
||||
|
||||
:param repo_or_comps: the repository object or
|
||||
tuple(baseurl, suite, component, architecture)
|
||||
:param filename: the name of meta-file
|
||||
"""
|
||||
if isinstance(repo_or_comps, Repository):
|
||||
baseurl = repo_or_comps.url
|
||||
suite, component = repo_or_comps.name
|
||||
arch = repo_or_comps.architecture
|
||||
else:
|
||||
baseurl, suite, component, arch = repo_or_comps
|
||||
|
||||
return "/".join((
|
||||
baseurl.rstrip("/"), "dists", suite, component,
|
||||
"binary-" + _ARCHITECTURES[arch],
|
||||
filename
|
||||
))
|
||||
|
||||
@staticmethod
|
||||
def _add_to_release(release, repository):
|
||||
"""Adds repository information to debian release.
|
||||
|
||||
:param release: the deb822.Release instance
|
||||
:param repository: the repository object
|
||||
"""
|
||||
|
||||
# reset the date
|
||||
release["Date"] = datetime.datetime.now().strftime(
|
||||
"%a, %d %b %Y %H:%M:%S %Z"
|
||||
)
|
||||
release.setdefault("Origin", repository.origin)
|
||||
release.setdefault("Label", repository.origin)
|
||||
release.setdefault("Suite", repository.name[0])
|
||||
release.setdefault("Codename", repository.name[0].split("-", 1)[0])
|
||||
release.setdefault("Description", "The packages repository.")
|
||||
|
||||
keys = ("Architectures", "Components")
|
||||
values = (repository.architecture, repository.name[1])
|
||||
for key, value in six.moves.zip(keys, values):
|
||||
if key in release:
|
||||
release[key] = utils.append_token_to_string(
|
||||
release[key],
|
||||
value
|
||||
)
|
||||
else:
|
||||
release[key] = value
|
||||
|
||||
@staticmethod
|
||||
def _add_files_to_release(release, basepath, files):
|
||||
"""Adds information about meta files to debian release.
|
||||
|
||||
:param release: the deb822.Release instance
|
||||
:param basepath: the suite folder path
|
||||
:param files: the sequence of files
|
||||
"""
|
||||
|
||||
files_info = utils.get_size_and_checksum_for_files(
|
||||
files, _checksum_collector
|
||||
)
|
||||
for filepath, size, cs in files_info:
|
||||
fname = filepath[len(basepath) + 1:]
|
||||
size = six.text_type(size)
|
||||
for m, checksum in six.moves.zip(_CHECKSUM_METHODS, cs):
|
||||
for v in release[m]:
|
||||
if v["name"] == fname:
|
||||
v[m] = checksum
|
||||
v["size"] = size
|
||||
break
|
||||
else:
|
||||
release[m].append(deb822.Deb822Dict({
|
||||
m: checksum,
|
||||
"size": size,
|
||||
"name": fname
|
||||
}))
|
|
@ -1,283 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import copy
|
||||
import multiprocessing
|
||||
import os
|
||||
import shutil
|
||||
|
||||
import createrepo
|
||||
import lxml.etree as etree
|
||||
import six
|
||||
|
||||
from packetary.drivers.base import RepositoryDriverBase
|
||||
from packetary.library.streams import GzipDecompress
|
||||
from packetary.library import utils
|
||||
from packetary.objects import FileChecksum
|
||||
from packetary.objects import Package
|
||||
from packetary.objects import PackageRelation
|
||||
from packetary.objects import PackageVersion
|
||||
from packetary.objects import Repository
|
||||
|
||||
|
||||
urljoin = six.moves.urllib.parse.urljoin
|
||||
|
||||
# TODO(configurable option for drivers)
|
||||
_CORE_GROUPS = ("core", "base")
|
||||
|
||||
_MANDATORY_TYPES = ("mandatory", "default")
|
||||
|
||||
# The namespaces are used in metadata xml of repository
|
||||
_NAMESPACES = {
|
||||
"main": "http://linux.duke.edu/metadata/common",
|
||||
"md": "http://linux.duke.edu/metadata/repo",
|
||||
"rpm": "http://linux.duke.edu/metadata/rpm"
|
||||
}
|
||||
|
||||
|
||||
class CreaterepoCallBack(object):
|
||||
"""Callback object for createrepo"""
|
||||
def __init__(self, logger):
|
||||
self.logger = logger
|
||||
|
||||
def errorlog(self, msg):
|
||||
"""Error log output."""
|
||||
self.logger.error(msg)
|
||||
|
||||
def log(self, msg):
|
||||
"""Logs message."""
|
||||
self.logger.info(msg)
|
||||
|
||||
def progress(self, item, current, total):
|
||||
""""Progress bar."""
|
||||
pass
|
||||
|
||||
|
||||
class RpmRepositoryDriver(RepositoryDriverBase):
|
||||
def parse_urls(self, urls):
|
||||
"""Overrides method of superclass."""
|
||||
return (url.rstrip("/") for url in urls)
|
||||
|
||||
def get_repository(self, connection, url, arch, consumer):
|
||||
name = utils.get_path_from_url(url, False)
|
||||
consumer(Repository(
|
||||
name=name,
|
||||
url=url + "/",
|
||||
architecture=arch,
|
||||
origin=""
|
||||
))
|
||||
|
||||
def get_packages(self, connection, repository, consumer):
|
||||
"""Overrides method of superclass."""
|
||||
baseurl = repository.url
|
||||
repomd = urljoin(baseurl, "repodata/repomd.xml")
|
||||
self.logger.debug("repomd: %s", repomd)
|
||||
|
||||
repomd_tree = etree.parse(connection.open_stream(repomd))
|
||||
mandatory = self._get_mandatory_packages(
|
||||
self._load_db(
|
||||
connection, baseurl, repomd_tree, "group_gz", "group"
|
||||
)
|
||||
)
|
||||
primary_db = self._load_db(connection, baseurl, repomd_tree, "primary")
|
||||
if primary_db is None:
|
||||
raise ValueError("Malformed repository: {0}".format(repository))
|
||||
|
||||
counter = 0
|
||||
for tag in primary_db.iterfind("./main:package", _NAMESPACES):
|
||||
try:
|
||||
name = tag.find("./main:name", _NAMESPACES).text
|
||||
consumer(Package(
|
||||
repository=repository,
|
||||
name=tag.find("./main:name", _NAMESPACES).text,
|
||||
version=self._unparse_version_attrs(
|
||||
tag.find("./main:version", _NAMESPACES).attrib
|
||||
),
|
||||
filesize=int(
|
||||
tag.find("./main:size", _NAMESPACES)
|
||||
.attrib.get("package", -1)
|
||||
),
|
||||
filename=tag.find(
|
||||
"./main:location", _NAMESPACES
|
||||
).attrib["href"],
|
||||
checksum=self._get_checksum(tag),
|
||||
mandatory=name in mandatory,
|
||||
requires=self._get_relations(tag, "requires"),
|
||||
obsoletes=self._get_relations(tag, "obsoletes"),
|
||||
provides=self._get_relations(tag, "provides")
|
||||
))
|
||||
except (ValueError, KeyError) as e:
|
||||
self.logger.error(
|
||||
"Malformed tag %s - %s: %s",
|
||||
repository, etree.tostring(tag), six.text_type(e)
|
||||
)
|
||||
raise
|
||||
counter += 1
|
||||
self.logger.info("loaded: %d packages from %s.", counter, repository)
|
||||
|
||||
def rebuild_repository(self, repository, packages):
|
||||
"""Overrides method of superclass."""
|
||||
basepath = utils.get_path_from_url(repository.url)
|
||||
self.logger.info("rebuild repository in %s", basepath)
|
||||
md_config = createrepo.MetaDataConfig()
|
||||
try:
|
||||
md_config.workers = multiprocessing.cpu_count()
|
||||
md_config.directory = str(basepath)
|
||||
md_config.update = True
|
||||
mdgen = createrepo.MetaDataGenerator(
|
||||
config_obj=md_config, callback=CreaterepoCallBack(self.logger)
|
||||
)
|
||||
mdgen.doPkgMetadata()
|
||||
mdgen.doRepoMetadata()
|
||||
mdgen.doFinalMove()
|
||||
except createrepo.MDError as e:
|
||||
err_msg = six.text_type(e)
|
||||
self.logger.exception(
|
||||
"failed to create yum repository in %s: %s",
|
||||
basepath,
|
||||
err_msg
|
||||
)
|
||||
shutil.rmtree(
|
||||
os.path.join(md_config.outputdir, md_config.tempdir),
|
||||
ignore_errors=True
|
||||
)
|
||||
raise RuntimeError(
|
||||
"Failed to create yum repository in {0}."
|
||||
.format(err_msg))
|
||||
|
||||
def fork_repository(self, connection, repository, destination,
|
||||
source=False, locale=False):
|
||||
# TODO(download gpk)
|
||||
# TODO(sources and locales)
|
||||
new_repo = copy.copy(repository)
|
||||
new_repo.url = utils.localize_repo_url(destination, repository.url)
|
||||
self.logger.info(
|
||||
"clone repository %s to %s", repository, new_repo.url
|
||||
)
|
||||
utils.ensure_dir_exist(new_repo.url)
|
||||
self.rebuild_repository(new_repo, set())
|
||||
return new_repo
|
||||
|
||||
def _load_db(self, connection, baseurl, repomd, *aliases):
|
||||
"""Loads database.
|
||||
|
||||
:param connection: the connection object
|
||||
:param baseurl: the base repository URL
|
||||
:param repomd: the parsed metadata of repository
|
||||
:param aliases: the aliases of database name
|
||||
:return: parsed database file or None if db does not exist
|
||||
"""
|
||||
|
||||
for dbname in aliases:
|
||||
self.logger.debug("loading %s database...", dbname)
|
||||
node = repomd.find(
|
||||
"./md:data[@type='{0}']".format(dbname), _NAMESPACES
|
||||
)
|
||||
if node is not None:
|
||||
break
|
||||
else:
|
||||
return
|
||||
|
||||
url = urljoin(
|
||||
baseurl,
|
||||
node.find("./md:location", _NAMESPACES).attrib["href"]
|
||||
)
|
||||
self.logger.debug("loading %s - %s...", dbname, url)
|
||||
stream = connection.open_stream(url)
|
||||
if url.endswith(".gz"):
|
||||
stream = GzipDecompress(stream)
|
||||
return etree.parse(stream)
|
||||
|
||||
def _get_mandatory_packages(self, groups_db):
|
||||
"""Get the set of mandatory package names.
|
||||
|
||||
:param groups_db: the parsed groups database
|
||||
"""
|
||||
package_names = set()
|
||||
if groups_db is None:
|
||||
return package_names
|
||||
count = 0
|
||||
for name in _CORE_GROUPS:
|
||||
result = groups_db.xpath("./group/id[text()='{0}']".format(name))
|
||||
if len(result) == 0:
|
||||
self.logger.warning("the group '%s' is not found.", name)
|
||||
continue
|
||||
group = result[0].getparent()
|
||||
for t in _MANDATORY_TYPES:
|
||||
xpath = "./packagelist/packagereq[@type='{0}']".format(t)
|
||||
for tag in group.iterfind(xpath):
|
||||
package_names.add(tag.text)
|
||||
count += 1
|
||||
self.logger.info("detected %d mandatory packages.", count)
|
||||
return package_names
|
||||
|
||||
def _get_relations(self, pkg_tag, name):
|
||||
"""Gets package relations by name from package tag.
|
||||
|
||||
:param pkg_tag: the xml-tag with package description
|
||||
:param name: the relations name
|
||||
:return: list of PackageRelation objects
|
||||
"""
|
||||
relations = list()
|
||||
append = relations.append
|
||||
tags_iter = pkg_tag.iterfind(
|
||||
"./main:format/rpm:%s/rpm:entry" % name,
|
||||
_NAMESPACES
|
||||
)
|
||||
for elem in tags_iter:
|
||||
append(PackageRelation.from_args(
|
||||
self._unparse_relation_attrs(elem.attrib)
|
||||
))
|
||||
|
||||
return relations
|
||||
|
||||
def _get_checksum(self, pkg_tag):
|
||||
"""Gets checksum from package tag."""
|
||||
checksum = dict.fromkeys(("md5", "sha1", "sha256"), None)
|
||||
checksum_tag = pkg_tag.find("./main:checksum", _NAMESPACES)
|
||||
checksum[checksum_tag.attrib["type"]] = checksum_tag.text
|
||||
return FileChecksum(**checksum)
|
||||
|
||||
def _unparse_relation_attrs(self, attrs):
|
||||
"""Gets the package relation from attributes.
|
||||
|
||||
:param attrs: the relation tag attributes
|
||||
:return tuple(name, version_op, version_edge)
|
||||
"""
|
||||
if "flags" not in attrs:
|
||||
return attrs['name'], None
|
||||
|
||||
return (
|
||||
attrs['name'],
|
||||
attrs["flags"].lower(),
|
||||
self._unparse_version_attrs(attrs)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _unparse_version_attrs(attrs):
|
||||
"""Gets the package version from attributes.
|
||||
|
||||
:param attrs: the relation tag attributes
|
||||
:return: the PackageVersion object
|
||||
"""
|
||||
|
||||
return PackageVersion(
|
||||
int(attrs.get("epoch", 0)),
|
||||
attrs.get("ver", "0.0").split("."),
|
||||
attrs.get("rel", "0").split(".")
|
||||
)
|
|
@ -1,21 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import eventlet
|
||||
|
||||
eventlet.monkey_patch()
|
|
@ -1,84 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import functools
|
||||
import hashlib
|
||||
|
||||
|
||||
class _HashComposite(object):
|
||||
"""Combines several hash methods."""
|
||||
|
||||
def __init__(self, hash_objects):
|
||||
self.hash_objects = hash_objects
|
||||
|
||||
def update(self, data):
|
||||
"""Updates the hash objects with the string arg.
|
||||
|
||||
For more details see doc of hashlib.update.
|
||||
"""
|
||||
for o in self.hash_objects:
|
||||
o.update(data)
|
||||
|
||||
def hexdigest(self):
|
||||
"""Returns the list of appropriate hexdigests of hash_objects.
|
||||
|
||||
For more details see doc of hashlib.hexdigest.
|
||||
"""
|
||||
return [o.hexdigest() for o in self.hash_objects]
|
||||
|
||||
|
||||
def _new_composite(methods):
|
||||
"""Creates new composite method."""
|
||||
|
||||
def wrapper():
|
||||
return _HashComposite([x() for x in methods])
|
||||
return wrapper
|
||||
|
||||
|
||||
def _checksum(method):
|
||||
"""Makes function to calculate checksum for stream."""
|
||||
@functools.wraps(method)
|
||||
def calculate(stream, chunksize=16 * 1024):
|
||||
"""Calculates checksum for binary stream.
|
||||
|
||||
:param stream: file-like object opened in binary mode.
|
||||
:return: the checksum of content in terms of method.
|
||||
"""
|
||||
|
||||
s = method()
|
||||
while True:
|
||||
chunk = stream.read(chunksize)
|
||||
if not chunk:
|
||||
break
|
||||
s.update(chunk)
|
||||
return s.hexdigest()
|
||||
return calculate
|
||||
|
||||
|
||||
md5 = _checksum(hashlib.md5)
|
||||
|
||||
sha1 = _checksum(hashlib.sha1)
|
||||
|
||||
sha256 = _checksum(hashlib.sha256)
|
||||
|
||||
|
||||
def composite(*methods):
|
||||
"""Calculate several checksum at one time."""
|
||||
return _checksum(_new_composite(
|
||||
[getattr(hashlib, x) for x in methods]
|
||||
))
|
|
@ -1,307 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import errno
|
||||
import logging
|
||||
import os
|
||||
import six
|
||||
import six.moves.http_client as http_client
|
||||
import six.moves.urllib.request as urllib
|
||||
import six.moves.urllib_error as urlerror
|
||||
import time
|
||||
|
||||
from packetary.library.streams import StreamWrapper
|
||||
from packetary.library.utils import ensure_dir_exist
|
||||
|
||||
|
||||
logger = logging.getLogger(__package__)
|
||||
|
||||
|
||||
RETRYABLE_ERRORS = (http_client.HTTPException, IOError)
|
||||
|
||||
|
||||
class RangeError(urlerror.URLError):
|
||||
pass
|
||||
|
||||
|
||||
class RetryableRequest(urllib.Request):
|
||||
MAX_TIMEOUT = 5
|
||||
|
||||
offset = 0
|
||||
retries_left = 1
|
||||
retry_interval = 0
|
||||
start_time = 0
|
||||
|
||||
def get_retry_interval(self):
|
||||
"""Calculates progressive retry interval in seconds.
|
||||
|
||||
:return: the time to wait before start retry
|
||||
"""
|
||||
# we uses progressive timeout between retries,
|
||||
# the greatest number of retry will have greatest timeout
|
||||
# but limited with max_delay
|
||||
coef = max(self.MAX_TIMEOUT - self.retries_left, 1)
|
||||
timeout = self.retry_interval * coef
|
||||
return min(timeout, self.MAX_TIMEOUT)
|
||||
|
||||
|
||||
class ResumableResponse(StreamWrapper):
|
||||
"""The http-response wrapper to add resume ability.
|
||||
|
||||
Allows to resume read from same position if connection is lost.
|
||||
"""
|
||||
|
||||
def __init__(self, request, response, opener):
|
||||
"""Initialises.
|
||||
|
||||
:param request: the original http request
|
||||
:param response: the original http response
|
||||
:param opener: the instance of urllib.OpenerDirector
|
||||
"""
|
||||
super(ResumableResponse, self).__init__(response)
|
||||
self.request = request
|
||||
self.opener = opener
|
||||
|
||||
def read_chunk(self, chunksize):
|
||||
"""Overrides super class method."""
|
||||
while 1:
|
||||
try:
|
||||
chunk = self.stream.read(chunksize)
|
||||
self.request.offset += len(chunk)
|
||||
return chunk
|
||||
except RETRYABLE_ERRORS as e:
|
||||
# TODO(check hashsums)
|
||||
response = self.opener.error(
|
||||
self.request.get_type(), self.request,
|
||||
self.stream, 502, six.text_type(e), self.stream.info()
|
||||
)
|
||||
self.stream = response.stream
|
||||
|
||||
|
||||
class RetryHandler(urllib.HTTPRedirectHandler):
|
||||
"""urllib Handler to add ability for retrying on server errors."""
|
||||
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
new_req = urllib.HTTPRedirectHandler.redirect_request(
|
||||
self, req, fp, code, msg, headers, newurl
|
||||
)
|
||||
if new_req is not None:
|
||||
# We use class assignment for casting new request to type
|
||||
# RetryableRequest
|
||||
new_req.__class__ = RetryableRequest
|
||||
new_req.retries_left = req.retries_left
|
||||
new_req.offset = req.offset
|
||||
new_req.start_time = req.start_time
|
||||
new_req.retry_interval = req.retry_interval
|
||||
return new_req
|
||||
|
||||
@staticmethod
|
||||
def http_request(request):
|
||||
"""Initialises http request.
|
||||
|
||||
:param request: the instance of RetryableRequest
|
||||
:return: the request
|
||||
"""
|
||||
logger.debug("start request: %s", request.get_full_url())
|
||||
if request.offset > 0:
|
||||
request.add_header('Range', 'bytes=%d-' % request.offset)
|
||||
request.start_time = time.time()
|
||||
return request
|
||||
|
||||
def http_response(self, request, response):
|
||||
"""Wraps response in a ResumableResponse.
|
||||
|
||||
Checks that partial request completed successfully.
|
||||
:param request: the instance of RetryableRequest
|
||||
:param response: the response object
|
||||
:return: ResumableResponse if success otherwise same response
|
||||
"""
|
||||
code, msg = response.getcode(), response.msg
|
||||
|
||||
if 300 <= code < 400:
|
||||
# the redirect group, pass to next handler as is
|
||||
return response
|
||||
|
||||
# the server should response partial content if range is specified
|
||||
if request.offset > 0 and code != 206:
|
||||
raise RangeError(msg)
|
||||
|
||||
if code >= 400:
|
||||
logger.error(
|
||||
"request failed: %s - %d(%s), retries left - %d.",
|
||||
request.get_full_url(), code, msg, request.retries_left - 1
|
||||
)
|
||||
if is_retryable_http_error(code) and request.retries_left > 0:
|
||||
time.sleep(request.get_retry_interval())
|
||||
request.retries_left -= 1
|
||||
response = self.parent.open(request)
|
||||
# pass response to next handler as is.
|
||||
return response
|
||||
|
||||
logger.debug(
|
||||
"request completed: %s - %d (%s), duration - %d ms.",
|
||||
request.get_full_url(), response.getcode(), response.msg,
|
||||
int((time.time() - request.start_time) * 1000)
|
||||
)
|
||||
|
||||
return ResumableResponse(request, response, self.parent)
|
||||
|
||||
https_request = http_request
|
||||
https_response = http_response
|
||||
|
||||
|
||||
def is_retryable_http_error(code):
|
||||
"""Checks that http error can be retried.
|
||||
|
||||
:param code: the HTTP_CODE
|
||||
:return: True if request can be retried otherwise False
|
||||
"""
|
||||
return code >= http_client.INTERNAL_SERVER_ERROR
|
||||
|
||||
|
||||
class ConnectionsManager(object):
|
||||
"""The connections manager."""
|
||||
|
||||
def __init__(self, proxy=None, secure_proxy=None,
|
||||
retries_num=0, retry_interval=0):
|
||||
"""Initialises.
|
||||
|
||||
:param proxy: the url of proxy for http-connections
|
||||
:param secure_proxy: the url of proxy for https-connections
|
||||
:param retries_num: the number of allowed retries
|
||||
:param retry_interval: the time between retries (in seconds)
|
||||
"""
|
||||
if proxy:
|
||||
proxies = {
|
||||
"http": proxy,
|
||||
"https": secure_proxy or proxy,
|
||||
}
|
||||
else:
|
||||
proxies = None
|
||||
|
||||
self.retries_num = retries_num
|
||||
self.retry_interval = retry_interval
|
||||
self.opener = urllib.build_opener(
|
||||
RetryHandler(),
|
||||
urllib.ProxyHandler(proxies)
|
||||
)
|
||||
|
||||
def make_request(self, url, offset=0):
|
||||
"""Makes new http request.
|
||||
|
||||
:param url: the remote file`s url
|
||||
:param offset: the number of bytes from the beginning,
|
||||
that will be skipped
|
||||
:return: The new http request
|
||||
"""
|
||||
|
||||
if url.startswith("/"):
|
||||
url = "file://" + url
|
||||
|
||||
request = RetryableRequest(url)
|
||||
request.retries_left = self.retries_num
|
||||
request.retry_interval = self.retry_interval
|
||||
request.offset = offset
|
||||
return request
|
||||
|
||||
def open_stream(self, url, offset=0):
|
||||
"""Opens remote file for streaming.
|
||||
|
||||
:param url: the remote file`s url
|
||||
:param offset: the number of bytes from the beginning,
|
||||
that will be skipped
|
||||
"""
|
||||
|
||||
request = self.make_request(url, offset)
|
||||
while 1:
|
||||
try:
|
||||
return self.opener.open(request)
|
||||
except (RangeError, urlerror.HTTPError):
|
||||
raise
|
||||
except RETRYABLE_ERRORS as e:
|
||||
if request.retries_left <= 0:
|
||||
raise
|
||||
request.retries_left -= 1
|
||||
logger.exception(
|
||||
"Failed to open url - %s: %s. retries left - %d.",
|
||||
url, six.text_type(e), request.retries_left
|
||||
)
|
||||
time.sleep(request.get_retry_interval())
|
||||
|
||||
def retrieve(self, url, filename, **attributes):
|
||||
"""Downloads remote file.
|
||||
|
||||
:param url: the remote file`s url
|
||||
:param filename: the target filename on local filesystem
|
||||
:param attributes: the file attributes, like size, hashsum, etc.
|
||||
:return: the count of actually copied bytes
|
||||
"""
|
||||
offset = 0
|
||||
try:
|
||||
stats = os.stat(filename)
|
||||
expected_size = attributes.get('size', -1)
|
||||
if expected_size == stats.st_size:
|
||||
# TODO(check hashsum)
|
||||
return 0
|
||||
|
||||
if stats.st_size < expected_size:
|
||||
offset = stats.st_size
|
||||
except OSError as e:
|
||||
if e.errno != errno.ENOENT:
|
||||
raise
|
||||
ensure_dir_exist(os.path.dirname(filename))
|
||||
|
||||
logger.info("download: %s from the offset: %d", url, offset)
|
||||
|
||||
fd = os.open(filename, os.O_CREAT | os.O_WRONLY)
|
||||
try:
|
||||
return self._copy_stream(fd, url, offset)
|
||||
except RangeError:
|
||||
if offset == 0:
|
||||
raise
|
||||
logger.warning(
|
||||
"Failed to resume download, starts from the beginning: %s",
|
||||
url
|
||||
)
|
||||
return self._copy_stream(fd, url, 0)
|
||||
finally:
|
||||
os.fsync(fd)
|
||||
os.close(fd)
|
||||
|
||||
def _copy_stream(self, fd, url, offset):
|
||||
"""Copies remote file to local.
|
||||
|
||||
:param fd: the file`s descriptor
|
||||
:param url: the remote file`s url
|
||||
:param offset: the number of bytes from the beginning,
|
||||
that will be skipped
|
||||
:return: the count of actually copied bytes
|
||||
"""
|
||||
|
||||
source = self.open_stream(url, offset)
|
||||
os.ftruncate(fd, offset)
|
||||
os.lseek(fd, offset, os.SEEK_SET)
|
||||
chunk_size = 16 * 1024
|
||||
size = 0
|
||||
while 1:
|
||||
chunk = source.read(chunk_size)
|
||||
if not chunk:
|
||||
break
|
||||
os.write(fd, chunk)
|
||||
size += len(chunk)
|
||||
return size
|
|
@ -1,93 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from __future__ import with_statement
|
||||
|
||||
import logging
|
||||
import sys
|
||||
|
||||
import six
|
||||
|
||||
from eventlet.greenpool import GreenPool
|
||||
|
||||
|
||||
logger = logging.getLogger(__package__)
|
||||
|
||||
|
||||
class AsynchronousSection(object):
|
||||
"""Allows calling function asynchronously with waiting on exit."""
|
||||
|
||||
MIN_POOL_SIZE = 1
|
||||
|
||||
def __init__(self, size=0, ignore_errors_num=0):
|
||||
"""Initialises.
|
||||
|
||||
:param size: the max number of parallel tasks
|
||||
:param ignore_errors_num:
|
||||
number of errors which does not stop the execution
|
||||
"""
|
||||
|
||||
self.executor = GreenPool(max(size, self.MIN_POOL_SIZE))
|
||||
self.ignore_errors_num = ignore_errors_num
|
||||
self.errors = []
|
||||
self.tasks = set()
|
||||
|
||||
def __enter__(self):
|
||||
self.errors[:] = []
|
||||
return self
|
||||
|
||||
def __exit__(self, etype, *_):
|
||||
self.wait(etype is not None)
|
||||
|
||||
def execute(self, func, *args, **kwargs):
|
||||
"""Calls function asynchronously."""
|
||||
if 0 <= self.ignore_errors_num < len(self.errors):
|
||||
raise RuntimeError("Too many errors.")
|
||||
|
||||
gt = self.executor.spawn(func, *args, **kwargs)
|
||||
self.tasks.add(gt)
|
||||
gt.link(self.on_complete)
|
||||
return gt
|
||||
|
||||
def on_complete(self, gt):
|
||||
"""Callback to handle task completion."""
|
||||
|
||||
try:
|
||||
gt.wait()
|
||||
except Exception as e:
|
||||
logger.error("Task failed: %s", six.text_type(e))
|
||||
self.errors.append(sys.exc_info())
|
||||
finally:
|
||||
self.tasks.discard(gt)
|
||||
|
||||
def wait(self, ignore_errors=False):
|
||||
"""Waits until all tasks will be completed.
|
||||
|
||||
Do not use directly, will be called from context manager.
|
||||
"""
|
||||
self.executor.waitall()
|
||||
if len(self.errors) > 0:
|
||||
for exc_info in self.errors:
|
||||
logger.exception("error details.", exc_info=exc_info)
|
||||
|
||||
self.errors[:] = []
|
||||
if not ignore_errors:
|
||||
raise RuntimeError(
|
||||
"Operations completed with errors.\n"
|
||||
"See log for more details."
|
||||
)
|
|
@ -1,133 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import zlib
|
||||
|
||||
|
||||
class StreamWrapper(object):
|
||||
"""Helper class to implement stream wrappers.
|
||||
|
||||
It is base-class for Streamers,
|
||||
that provides functionality to transform stream on the fly.
|
||||
The wrapped stream may return data more that required,
|
||||
the extra read data will be kept in the internal buffer till
|
||||
next read.
|
||||
"""
|
||||
|
||||
CHUNK_SIZE = 1024
|
||||
|
||||
def __init__(self, stream):
|
||||
"""Initializes.
|
||||
|
||||
:param stream: file-like object opened in binary mode.
|
||||
"""
|
||||
self.stream = stream
|
||||
self.unread_tail = b""
|
||||
|
||||
def __getattr__(self, item):
|
||||
return getattr(self.stream, item)
|
||||
|
||||
def _read_tail(self):
|
||||
tmp = self.unread_tail
|
||||
self.unread_tail = b""
|
||||
return tmp
|
||||
|
||||
def _align_chunk(self, chunk, size):
|
||||
self.unread_tail = chunk[size:]
|
||||
return chunk[:size]
|
||||
|
||||
def read_chunk(self, chunksize):
|
||||
"""Overrides this method to change default behaviour."""
|
||||
return self.stream.read(chunksize)
|
||||
|
||||
def read(self, size=-1):
|
||||
result = self._read_tail()
|
||||
if size < 0:
|
||||
while True:
|
||||
chunk = self.read_chunk(self.CHUNK_SIZE)
|
||||
if not chunk:
|
||||
break
|
||||
result += chunk
|
||||
else:
|
||||
if len(result) > size:
|
||||
result = self._align_chunk(result, size)
|
||||
size -= len(result)
|
||||
while size > 0:
|
||||
chunk = self.read_chunk(self.CHUNK_SIZE)
|
||||
if not chunk:
|
||||
break
|
||||
if len(chunk) > size:
|
||||
chunk = self._align_chunk(chunk, size)
|
||||
size -= len(chunk)
|
||||
result += chunk
|
||||
return result
|
||||
|
||||
def readline(self):
|
||||
pos = self.unread_tail.find(b"\n")
|
||||
if pos >= 0:
|
||||
line = self._align_chunk(self.unread_tail, pos + 1)
|
||||
else:
|
||||
line = self._read_tail()
|
||||
while True:
|
||||
chunk = self.read_chunk(self.CHUNK_SIZE)
|
||||
if not chunk:
|
||||
break
|
||||
pos = chunk.find(b"\n")
|
||||
if pos >= 0:
|
||||
line += self._align_chunk(chunk, pos + 1)
|
||||
break
|
||||
line += chunk
|
||||
return line
|
||||
|
||||
def readlines(self):
|
||||
while True:
|
||||
line = self.readline()
|
||||
if not line:
|
||||
break
|
||||
yield line
|
||||
|
||||
def __iter__(self):
|
||||
return self.readlines()
|
||||
|
||||
|
||||
class GzipDecompress(StreamWrapper):
|
||||
"""The decompress stream."""
|
||||
|
||||
def __init__(self, stream):
|
||||
super(GzipDecompress, self).__init__(stream)
|
||||
# Magic parameter makes zlib module understand gzip header
|
||||
# http://stackoverflow.com/questions/1838699/how-can-i-decompress-a-gzip-stream-with-zlib
|
||||
# This works on cpython and pypy, but not jython.
|
||||
self.decompress = zlib.decompressobj(16 + zlib.MAX_WBITS)
|
||||
|
||||
def read_chunk(self, chunksize):
|
||||
if self.decompress.unconsumed_tail:
|
||||
uncompressed = self.decompress.decompress(
|
||||
self.decompress.unconsumed_tail, chunksize
|
||||
)
|
||||
if uncompressed:
|
||||
return uncompressed
|
||||
|
||||
while True:
|
||||
chunk = self.stream.read(chunksize)
|
||||
if not chunk:
|
||||
break
|
||||
uncompressed = self.decompress.decompress(chunk, chunksize)
|
||||
if uncompressed:
|
||||
return uncompressed
|
||||
return self.decompress.flush()
|
|
@ -1,115 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from __future__ import with_statement
|
||||
|
||||
import errno
|
||||
import os
|
||||
|
||||
import six
|
||||
|
||||
urlparse = six.moves.urllib.parse.urlparse
|
||||
|
||||
|
||||
def append_token_to_string(tokens, token):
|
||||
"""Adds new token to space separated list of tokens.
|
||||
|
||||
:param tokens: the 'sep' separated list
|
||||
:param token: new item
|
||||
"""
|
||||
values = tokens.split()
|
||||
if token not in values:
|
||||
values.append(token)
|
||||
values.sort()
|
||||
return ' '.join(values)
|
||||
return tokens
|
||||
|
||||
|
||||
def composite_writer(*args):
|
||||
"""Makes helper, that writes into several files simultaneously.
|
||||
|
||||
:param args: the list of file objects
|
||||
:return: the callable object - writer
|
||||
"""
|
||||
def write(text):
|
||||
"""Writes simultaneously to all files with utf-8 encoding control.
|
||||
|
||||
:param text: the text, that needs to write
|
||||
"""
|
||||
if isinstance(text, six.text_type):
|
||||
text = text.encode("utf-8")
|
||||
for arg in args:
|
||||
arg.write(text)
|
||||
return write
|
||||
|
||||
|
||||
def get_size_and_checksum_for_files(files, checksum_algo):
|
||||
"""Gets the path, size and checksum for files.
|
||||
|
||||
:param files: the sequence of files
|
||||
:param checksum_algo: the checksum calculator
|
||||
:return the sequence of tuples(filename, size, checksum)
|
||||
"""
|
||||
|
||||
for filename in files:
|
||||
with open(filename, "rb") as fd:
|
||||
size = os.fstat(fd.fileno()).st_size
|
||||
checksum = checksum_algo(fd)
|
||||
yield filename, size, checksum
|
||||
|
||||
|
||||
def get_path_from_url(url, ensure_file=True):
|
||||
"""Get the path from the URL.
|
||||
|
||||
:param url: the URL
|
||||
:param ensure_file: If True, ensure that scheme is "file"
|
||||
:return: the path component from URL
|
||||
:raises ValueError
|
||||
"""
|
||||
|
||||
comps = urlparse(url, scheme="file")
|
||||
if ensure_file and comps.scheme != "file":
|
||||
raise ValueError(
|
||||
"The absolute path is expected, actual have: {0}.".format(url)
|
||||
)
|
||||
if os.sep != "/":
|
||||
return comps.path.replace("/", os.sep)
|
||||
return comps.path
|
||||
|
||||
|
||||
def localize_repo_url(localurl, repo_url):
|
||||
"""Gets local repository url.
|
||||
|
||||
:param localurl: the base local URL
|
||||
:param repo_url: the origin URL of repository
|
||||
:return: localurl + get_path_from_url(repo_url)
|
||||
"""
|
||||
return localurl.rstrip("/") + urlparse(repo_url).path
|
||||
|
||||
|
||||
def ensure_dir_exist(path):
|
||||
"""Creates directory if it does not exist.
|
||||
|
||||
:param path: the full path to directory
|
||||
"""
|
||||
|
||||
try:
|
||||
os.makedirs(path)
|
||||
except OSError as e:
|
||||
if e.errno != errno.EEXIST:
|
||||
raise
|
|
@ -1,38 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from packetary.objects.index import Index
|
||||
from packetary.objects.package import FileChecksum
|
||||
from packetary.objects.package import Package
|
||||
from packetary.objects.package_relation import PackageRelation
|
||||
from packetary.objects.package_relation import VersionRange
|
||||
from packetary.objects.package_version import PackageVersion
|
||||
from packetary.objects.packages_tree import PackagesTree
|
||||
from packetary.objects.repository import Repository
|
||||
|
||||
|
||||
__all__ = [
|
||||
"FileChecksum",
|
||||
"Index",
|
||||
"Package",
|
||||
"PackageRelation",
|
||||
"PackagesTree",
|
||||
"PackageVersion",
|
||||
"Repository",
|
||||
"VersionRange",
|
||||
]
|
|
@ -1,62 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import abc
|
||||
import six
|
||||
|
||||
|
||||
@six.add_metaclass(abc.ABCMeta)
|
||||
class ComparableObject(object):
|
||||
"""Superclass for objects, that should be comparable.
|
||||
|
||||
Note: because python3 does not support __cmp__ slot, use
|
||||
cmp method to implement all of compare methods.
|
||||
"""
|
||||
|
||||
@abc.abstractmethod
|
||||
def cmp(self, other):
|
||||
"""Compares with other object.
|
||||
|
||||
:return: value is negative if if self < other, zero if self == other
|
||||
strictly positive if x > y
|
||||
"""
|
||||
|
||||
def __lt__(self, other):
|
||||
return self.cmp(other) < 0
|
||||
|
||||
def __le__(self, other):
|
||||
return self.cmp(other) <= 0
|
||||
|
||||
def __gt__(self, other):
|
||||
return self.cmp(other) > 0
|
||||
|
||||
def __ge__(self, other):
|
||||
return self.cmp(other) >= 0
|
||||
|
||||
def __eq__(self, other):
|
||||
if other is self:
|
||||
return True
|
||||
return isinstance(other, type(self)) and self.cmp(other) == 0
|
||||
|
||||
def __ne__(self, other):
|
||||
if other is self:
|
||||
return False
|
||||
return not isinstance(other, type(self)) or self.cmp(other) != 0
|
||||
|
||||
def __cmp__(self, other):
|
||||
return self.cmp(other)
|
|
@ -1,208 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from bintrees import FastRBTree
|
||||
from collections import defaultdict
|
||||
import functools
|
||||
import operator
|
||||
import six
|
||||
|
||||
|
||||
def _make_operator(direction, op):
|
||||
"""Makes search operator from low-level operation and search direction."""
|
||||
return functools.partial(direction, condition=op)
|
||||
|
||||
|
||||
def _start_upperbound(versions, version, condition):
|
||||
"""Gets all versions from [start, version] that meet condition.
|
||||
|
||||
:param versions: the tree of versions.
|
||||
:param version: the required version
|
||||
:param condition: condition for search
|
||||
:return: the list of found versions
|
||||
"""
|
||||
|
||||
result = list(versions.value_slice(None, version))
|
||||
try:
|
||||
bound = versions.ceiling_item(version)
|
||||
if condition(bound[0], version):
|
||||
result.append(bound[1])
|
||||
except KeyError:
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def _lowerbound_end(versions, version, condition):
|
||||
"""Gets all versions from [version, end] that meet condition.
|
||||
|
||||
:param versions: the tree of versions.
|
||||
:param version: the required version
|
||||
:param condition: condition for search
|
||||
:return: the list of found versions
|
||||
"""
|
||||
result = []
|
||||
items = iter(versions.item_slice(version, None))
|
||||
bound = next(items, None)
|
||||
if bound is None:
|
||||
return result
|
||||
if condition(bound[0], version):
|
||||
result.append(bound[1])
|
||||
result.extend(x[1] for x in items)
|
||||
return result
|
||||
|
||||
|
||||
def _equal(tree, version):
|
||||
"""Gets the package with specified version."""
|
||||
if version in tree:
|
||||
return [tree[version]]
|
||||
return []
|
||||
|
||||
|
||||
def _any(tree, _):
|
||||
"""Gets the package with max version."""
|
||||
return list(tree.values())
|
||||
|
||||
|
||||
class Index(object):
|
||||
"""The search index for packages.
|
||||
|
||||
Builds three search-indexes:
|
||||
- index of packages with versions.
|
||||
- index of virtual packages (provides).
|
||||
- index of obsoleted packages (obsoletes).
|
||||
|
||||
Uses to find package by name and range of versions.
|
||||
"""
|
||||
|
||||
operators = {
|
||||
None: _any,
|
||||
"lt": _make_operator(_start_upperbound, operator.lt),
|
||||
"le": _make_operator(_start_upperbound, operator.le),
|
||||
"gt": _make_operator(_lowerbound_end, operator.gt),
|
||||
"ge": _make_operator(_lowerbound_end, operator.ge),
|
||||
"eq": _equal,
|
||||
}
|
||||
|
||||
def __init__(self):
|
||||
self.packages = defaultdict(FastRBTree)
|
||||
self.obsoletes = defaultdict(FastRBTree)
|
||||
self.provides = defaultdict(FastRBTree)
|
||||
|
||||
def __iter__(self):
|
||||
"""Iterates over all packages including versions."""
|
||||
return self.get_all()
|
||||
|
||||
def __len__(self, _reduce=six.functools.reduce):
|
||||
"""Returns the total number of packages with versions."""
|
||||
return _reduce(
|
||||
lambda x, y: x + len(y),
|
||||
six.itervalues(self.packages),
|
||||
0
|
||||
)
|
||||
|
||||
def get_all(self):
|
||||
"""Gets sequence from all of packages including versions."""
|
||||
|
||||
for versions in six.itervalues(self.packages):
|
||||
for version in versions.values():
|
||||
yield version
|
||||
|
||||
def find(self, name, version):
|
||||
"""Finds the package by name and range of versions.
|
||||
|
||||
:param name: the package`s name.
|
||||
:param version: the range of versions.
|
||||
:return: the package if it is found, otherwise None
|
||||
"""
|
||||
candidates = self.find_all(name, version)
|
||||
if len(candidates) > 0:
|
||||
return candidates[-1]
|
||||
return None
|
||||
|
||||
def find_all(self, name, version):
|
||||
"""Finds the packages by name and range of versions.
|
||||
|
||||
:param name: the package`s name.
|
||||
:param version: the range of versions.
|
||||
:return: the list of suitable packages
|
||||
"""
|
||||
|
||||
if name in self.packages:
|
||||
candidates = self._find_versions(
|
||||
self.packages[name], version
|
||||
)
|
||||
if len(candidates) > 0:
|
||||
return candidates
|
||||
|
||||
if name in self.obsoletes:
|
||||
return self._resolve_relation(
|
||||
self.obsoletes[name], version
|
||||
)
|
||||
|
||||
if name in self.provides:
|
||||
return self._resolve_relation(
|
||||
self.provides[name], version
|
||||
)
|
||||
return []
|
||||
|
||||
def add(self, package):
|
||||
"""Adds new package to indexes.
|
||||
|
||||
:param package: the package object.
|
||||
"""
|
||||
self.packages[package.name][package.version] = package
|
||||
key = package.name, package.version
|
||||
|
||||
for obsolete in package.obsoletes:
|
||||
self.obsoletes[obsolete.name][key] = obsolete
|
||||
|
||||
for provide in package.provides:
|
||||
self.provides[provide.name][key] = provide
|
||||
|
||||
def _resolve_relation(self, relations, version):
|
||||
"""Resolve relation according to relations index.
|
||||
|
||||
:param relations: the index of relations
|
||||
:param version: the range of versions
|
||||
:return: package if found, otherwise None
|
||||
"""
|
||||
for key, candidate in relations.iter_items(reverse=True):
|
||||
if candidate.version.has_intersection(version):
|
||||
return [self.packages[key[0]][key[1]]]
|
||||
return []
|
||||
|
||||
@staticmethod
|
||||
def _find_versions(versions, version):
|
||||
"""Searches accurate version.
|
||||
|
||||
Search for the highest version out of intersection
|
||||
of existing and required range of versions.
|
||||
|
||||
:param versions: the existing versions
|
||||
:param version: the required range of versions
|
||||
:return: package if found, otherwise None
|
||||
"""
|
||||
|
||||
try:
|
||||
op = Index.operators[version.op]
|
||||
except KeyError:
|
||||
raise ValueError(
|
||||
"Unsupported operation: {0}"
|
||||
.format(version.op)
|
||||
)
|
||||
return op(versions, version.edge)
|
|
@ -1,80 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from collections import namedtuple
|
||||
|
||||
from packetary.objects.base import ComparableObject
|
||||
|
||||
|
||||
FileChecksum = namedtuple("FileChecksum", ("md5", "sha1", "sha256"))
|
||||
|
||||
|
||||
class Package(ComparableObject):
|
||||
"""Structure to describe package object."""
|
||||
|
||||
def __init__(self, repository, name, version, filename,
|
||||
filesize, checksum, mandatory=False,
|
||||
requires=None, provides=None, obsoletes=None):
|
||||
"""Initialises.
|
||||
|
||||
:param name: the package`s name
|
||||
:param version: the package`s version
|
||||
:param filename: the package`s relative filename
|
||||
:param filesize: the package`s file size
|
||||
:param checksum: the package`s checksum
|
||||
:param requires: the package`s requirements(optional)
|
||||
:param provides: the package`s provides(optional)
|
||||
:param obsoletes: the package`s obsoletes(optional)
|
||||
:param mandatory: indicates that package is mandatory
|
||||
"""
|
||||
|
||||
self.repository = repository
|
||||
self.name = name
|
||||
self.version = version
|
||||
self.filename = filename
|
||||
self.checksum = checksum
|
||||
self.filesize = filesize
|
||||
self.requires = requires or []
|
||||
self.provides = provides or []
|
||||
self.obsoletes = obsoletes or []
|
||||
self.mandatory = mandatory
|
||||
|
||||
def __copy__(self):
|
||||
"""Creates shallow copy of package."""
|
||||
return Package(**self.__dict__)
|
||||
|
||||
def __str__(self):
|
||||
return "{0} {1}".format(self.name, self.version)
|
||||
|
||||
def __unicode__(self):
|
||||
return u"{0} {1}".format(self.name, self.version)
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.name, self.version))
|
||||
|
||||
def cmp(self, other):
|
||||
"""Compares with other Package object."""
|
||||
if self.name < other.name:
|
||||
return -1
|
||||
if self.name > other.name:
|
||||
return 1
|
||||
if self.version < other.version:
|
||||
return -1
|
||||
if self.version > other.version:
|
||||
return 1
|
||||
return 0
|
|
@ -1,157 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import operator
|
||||
|
||||
|
||||
class VersionRange(object):
|
||||
"""Describes the range of versions.
|
||||
|
||||
Range of version is compare operation and edge.
|
||||
the compare operation can be one of:
|
||||
equal, greater, less, greater or equal, less or equal.
|
||||
"""
|
||||
|
||||
__slots__ = ["op", "edge"]
|
||||
|
||||
def __init__(self, op=None, edge=None):
|
||||
"""Initialises.
|
||||
|
||||
:param op: the name of operator to compare.
|
||||
:param edge: the edge of versions.
|
||||
"""
|
||||
self.op = op
|
||||
self.edge = edge
|
||||
|
||||
def __contains__(self, point):
|
||||
return getattr(operator, self.op)(point, self.edge)
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.op, self.edge))
|
||||
|
||||
def __eq__(self, other):
|
||||
if not isinstance(other, VersionRange):
|
||||
return False
|
||||
|
||||
return self.op == other.op and \
|
||||
self.edge == other.edge
|
||||
|
||||
def __str__(self):
|
||||
if self.edge is not None:
|
||||
return "{0} {1}".format(self.op, self.edge)
|
||||
return "any"
|
||||
|
||||
def __unicode__(self):
|
||||
if self.edge is not None:
|
||||
return u"{0} {1}".format(self.op, self.edge)
|
||||
return u"any"
|
||||
|
||||
def has_intersection(self, other):
|
||||
"""Checks that 2 ranges has intersection.
|
||||
|
||||
:param other: the candidate to check
|
||||
:return: True if intersection exists, otherwise False
|
||||
:raise TypeError: when other does not instance of VersionRange
|
||||
"""
|
||||
|
||||
if not isinstance(other, VersionRange):
|
||||
raise TypeError(
|
||||
"Unorderable type <type 'VersionRange'> and {0}"
|
||||
.format(type(other))
|
||||
)
|
||||
|
||||
if self.op is None or other.op is None:
|
||||
return True
|
||||
|
||||
if self.op[0] == other.op[0]:
|
||||
if self.op == 'eq':
|
||||
return self.edge == other.edge
|
||||
# the intersection is -inf or +inf
|
||||
return True
|
||||
if self.edge == other.edge:
|
||||
# need to cover case < a and >= a
|
||||
return self.edge in other and other.edge in self
|
||||
# all other cases
|
||||
return self.edge in other or other.edge in self
|
||||
|
||||
|
||||
class PackageRelation(object):
|
||||
"""Describes the package`s relation.
|
||||
|
||||
Relation includes the name of required package
|
||||
and range of versions that satisfies requirement.
|
||||
"""
|
||||
|
||||
__slots__ = ["name", "version", "alternative"]
|
||||
|
||||
def __init__(self, name, version=None, alternative=None):
|
||||
"""Initialises.
|
||||
|
||||
:param name: the name of required package
|
||||
:param version: the version range of required package
|
||||
:param alternative: the alternative relation
|
||||
"""
|
||||
self.name = name
|
||||
self.version = VersionRange() if version is None else version
|
||||
self.alternative = alternative
|
||||
|
||||
@classmethod
|
||||
def from_args(cls, *args):
|
||||
"""Construct relation from list of arguments.
|
||||
|
||||
:param args: the list of tuples(name, [version_op, version_edge])
|
||||
"""
|
||||
if len(args) == 0:
|
||||
return None
|
||||
|
||||
head = args[0]
|
||||
name = head[0]
|
||||
version = VersionRange(*head[1:])
|
||||
alternative = cls.from_args(*args[1:])
|
||||
return cls(name, version, alternative)
|
||||
|
||||
def __iter__(self):
|
||||
"""Iterates over alternatives."""
|
||||
r = self
|
||||
while r is not None:
|
||||
yield r
|
||||
r = r.alternative
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.name, self.version))
|
||||
|
||||
def __eq__(self, other):
|
||||
if not isinstance(other, PackageRelation):
|
||||
return False
|
||||
|
||||
return self.name == other.name and \
|
||||
self.version == other.version
|
||||
|
||||
def __str__(self):
|
||||
if self.alternative is None:
|
||||
return "{0} ({1})".format(self.name, self.version)
|
||||
return "{0} ({1}) | {2}".format(
|
||||
self.name, self.version, self.alternative
|
||||
)
|
||||
|
||||
def __unicode__(self):
|
||||
if self.alternative is None:
|
||||
return u"{0} ({1})".format(self.name, self.version)
|
||||
return u"{0} ({1}) | {2}".format(
|
||||
self.name, self.version, self.alternative
|
||||
)
|
|
@ -1,127 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
from packetary.objects.base import ComparableObject
|
||||
|
||||
|
||||
class PackageVersion(ComparableObject):
|
||||
"""The Package version."""
|
||||
|
||||
__slots__ = ["epoch", "version", "release"]
|
||||
|
||||
def __init__(self, epoch, version, release):
|
||||
self.epoch = int(epoch)
|
||||
self.version = tuple(version)
|
||||
self.release = tuple(release)
|
||||
|
||||
@classmethod
|
||||
def from_string(cls, text):
|
||||
"""Constructs from string.
|
||||
|
||||
:param text: the version in format '[{epoch-}]-{version}-{release}'
|
||||
"""
|
||||
components = text.split("-")
|
||||
if len(components) > 2:
|
||||
epoch = components[0]
|
||||
components = components[1:]
|
||||
else:
|
||||
epoch = 0
|
||||
return cls(epoch, components[0].split("."), components[1].split("."))
|
||||
|
||||
def cmp(self, other):
|
||||
if not isinstance(other, PackageVersion):
|
||||
other = PackageVersion.from_string(str(other))
|
||||
|
||||
if not isinstance(other, PackageVersion):
|
||||
raise TypeError
|
||||
if self.epoch < other.epoch:
|
||||
return -1
|
||||
if self.epoch > other.epoch:
|
||||
return 1
|
||||
|
||||
res = self._cmp_version_part(self.version, other.version)
|
||||
if res != 0:
|
||||
return res
|
||||
return self._cmp_version_part(self.release, other.release)
|
||||
|
||||
def __eq__(self, other):
|
||||
if other is self:
|
||||
return True
|
||||
return self.cmp(other) == 0
|
||||
|
||||
def __str__(self):
|
||||
return "{0}-{1}-{2}".format(
|
||||
self.epoch,
|
||||
".".join(str(x) for x in self.version),
|
||||
".".join(str(x) for x in self.release)
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _order(cls, x):
|
||||
"""Return an integer value for character x"""
|
||||
if x.isdigit():
|
||||
return int(x) + 1
|
||||
if x.isalpha():
|
||||
return ord(x)
|
||||
return ord(x) + 256
|
||||
|
||||
@classmethod
|
||||
def _cmp_version_string(cls, version1, version2):
|
||||
"""Compares two versions as string."""
|
||||
la = [cls._order(x) for x in version1]
|
||||
lb = [cls._order(x) for x in version2]
|
||||
while la or lb:
|
||||
a = 0
|
||||
b = 0
|
||||
if la:
|
||||
a = la.pop(0)
|
||||
if lb:
|
||||
b = lb.pop(0)
|
||||
if a < b:
|
||||
return -1
|
||||
elif a > b:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
@classmethod
|
||||
def _cmp_version_part(cls, version1, version2):
|
||||
"""Compares two versions."""
|
||||
ver1_it = iter(version1)
|
||||
ver2_it = iter(version2)
|
||||
while True:
|
||||
v1 = next(ver1_it, None)
|
||||
v2 = next(ver2_it, None)
|
||||
|
||||
if v1 is None or v2 is None:
|
||||
if v1 is not None:
|
||||
return 1
|
||||
if v2 is not None:
|
||||
return -1
|
||||
return 0
|
||||
|
||||
if v1.isdigit() and v2.isdigit():
|
||||
a = int(v1)
|
||||
b = int(v2)
|
||||
if a < b:
|
||||
return -1
|
||||
if a > b:
|
||||
return 1
|
||||
else:
|
||||
r = cls._cmp_version_string(v1, v2)
|
||||
if r != 0:
|
||||
return r
|
|
@ -1,129 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import warnings
|
||||
|
||||
from packetary.objects.index import Index
|
||||
|
||||
|
||||
class UnresolvedWarning(UserWarning):
|
||||
"""Warning about unresolved depends."""
|
||||
pass
|
||||
|
||||
|
||||
class PackagesTree(Index):
|
||||
"""Helper class to deal with dependency graph."""
|
||||
|
||||
def __init__(self):
|
||||
super(PackagesTree, self).__init__()
|
||||
self.mandatory_packages = []
|
||||
|
||||
def add(self, package):
|
||||
super(PackagesTree, self).add(package)
|
||||
# store all mandatory packages in separated list for quick access
|
||||
if package.mandatory:
|
||||
self.mandatory_packages.append(package)
|
||||
|
||||
def get_unresolved_dependencies(self, base=None):
|
||||
"""Gets the set of unresolved dependencies.
|
||||
|
||||
:param base: the base index to resolve dependencies
|
||||
:return: the set of unresolved depends.
|
||||
"""
|
||||
external = self.__get_unresolved_dependencies(self)
|
||||
if base is None:
|
||||
return external
|
||||
|
||||
unresolved = set()
|
||||
for relation in external:
|
||||
for rel in relation:
|
||||
if base.find(rel.name, rel.version) is not None:
|
||||
break
|
||||
else:
|
||||
unresolved.add(relation)
|
||||
return unresolved
|
||||
|
||||
def get_minimal_subset(self, main, requirements):
|
||||
"""Gets the minimal work subset.
|
||||
|
||||
:param main: the main index, to complete requirements.
|
||||
:param requirements: additional requirements.
|
||||
:return: The set of resolved depends.
|
||||
"""
|
||||
|
||||
unresolved = set()
|
||||
resolved = set()
|
||||
if main is None:
|
||||
def pkg_filter(*_):
|
||||
pass
|
||||
else:
|
||||
pkg_filter = main.find
|
||||
self.__get_unresolved_dependencies(main, requirements)
|
||||
|
||||
stack = list()
|
||||
stack.append(requirements)
|
||||
|
||||
# add all mandatory packages
|
||||
for pkg in self.mandatory_packages:
|
||||
resolved.add(pkg)
|
||||
stack.append(pkg.requires)
|
||||
|
||||
while len(stack) > 0:
|
||||
required = stack.pop()
|
||||
for require in required:
|
||||
for rel in require:
|
||||
if rel not in unresolved:
|
||||
if pkg_filter(rel.name, rel.version) is not None:
|
||||
break
|
||||
# use all packages that meets depends
|
||||
candidates = self.find_all(rel.name, rel.version)
|
||||
for cand in candidates:
|
||||
if cand not in resolved:
|
||||
resolved.add(cand)
|
||||
stack.append(cand.requires)
|
||||
if len(candidates) > 0:
|
||||
break
|
||||
else:
|
||||
unresolved.add(require)
|
||||
msg = "Unresolved depends: {0}".format(require)
|
||||
warnings.warn(UnresolvedWarning(msg))
|
||||
|
||||
return resolved
|
||||
|
||||
@staticmethod
|
||||
def __get_unresolved_dependencies(index, unresolved=None):
|
||||
"""Gets the set of unresolved dependencies.
|
||||
|
||||
:param index: the search index.
|
||||
:param unresolved: the known list of unresolved packages.
|
||||
:return: the set of unresolved depends.
|
||||
"""
|
||||
|
||||
if unresolved is None:
|
||||
unresolved = set()
|
||||
|
||||
for pkg in index:
|
||||
for require in pkg.requires:
|
||||
for rel in require:
|
||||
if rel not in unresolved:
|
||||
candidate = index.find(rel.name, rel.version)
|
||||
if candidate is not None and candidate != pkg:
|
||||
break
|
||||
else:
|
||||
unresolved.add(require)
|
||||
return unresolved
|
|
@ -1,48 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
|
||||
class Repository(object):
|
||||
"""Structure to describe repository object."""
|
||||
|
||||
def __init__(self, name, url, architecture, origin):
|
||||
"""Initialises.
|
||||
|
||||
:param name: the repository`s name, may be tuple of strings
|
||||
:param url: the repository`s URL
|
||||
:param architecture: the repository`s architecture
|
||||
:param origin: the repository`s origin
|
||||
"""
|
||||
self.name = name
|
||||
self.url = url
|
||||
self.architecture = architecture
|
||||
self.origin = origin
|
||||
|
||||
def __str__(self):
|
||||
if isinstance(self.name, tuple):
|
||||
return ".".join(self.name)
|
||||
return self.name or self.url
|
||||
|
||||
def __unicode__(self):
|
||||
if isinstance(self.name, tuple):
|
||||
return u".".join(self.name)
|
||||
return self.name or self.url
|
||||
|
||||
def __copy__(self):
|
||||
"""Creates shallow copy of package."""
|
||||
return Repository(**self.__dict__)
|
|
@ -1,47 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
import copy
|
||||
|
||||
|
||||
class CopyStatistics(object):
|
||||
"""The statistics of packages copying"""
|
||||
def __init__(self):
|
||||
# the number of copied packages
|
||||
self.copied = 0
|
||||
# the number of total packages
|
||||
self.total = 0
|
||||
|
||||
def on_package_copied(self, bytes_copied):
|
||||
"""Proceed next copied package."""
|
||||
if bytes_copied > 0:
|
||||
self.copied += 1
|
||||
self.total += 1
|
||||
|
||||
def __iadd__(self, other):
|
||||
if not isinstance(other, CopyStatistics):
|
||||
raise TypeError
|
||||
|
||||
self.copied += other.copied
|
||||
self.total += other.total
|
||||
return self
|
||||
|
||||
def __add__(self, other):
|
||||
result = copy.copy(self)
|
||||
result += other
|
||||
return result
|
|
@ -1,27 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
|
||||
try:
|
||||
import unittest2 as unittest
|
||||
except ImportError:
|
||||
import unittest
|
||||
|
||||
|
||||
class TestCase(unittest.TestCase):
|
||||
|
||||
"""Test case base class for all unit tests."""
|
|
@ -1,19 +0,0 @@
|
|||
Package: test
|
||||
Source: test.src
|
||||
Version: 1.1.1-1~u14.04+test
|
||||
Architecture: all
|
||||
Maintainer: Test
|
||||
Installed-Size: 3509
|
||||
Homepage: http://localhost/
|
||||
Priority: required
|
||||
Section: web
|
||||
Filename: pool/main/t/test.deb
|
||||
Size: 100
|
||||
Depends: test2 (>= 0.8.16~exp9)|tes2-old, test3
|
||||
Pre-Depends: test-main
|
||||
Provides: file
|
||||
Replaces: test-old
|
||||
SHA256: 14d6e308d8699b7f9ba2fe1ef778c0e38cf295614d308039d687b6b097d50859
|
||||
SHA1: 402bd18c145ae3b5344edf07f246be159397fd40
|
||||
MD5sum: 1ae09f80109f40dfbfaf3ba423c8625a
|
||||
Description: test package
|
|
@ -1,6 +0,0 @@
|
|||
Archive: trusty
|
||||
Version: 14.04
|
||||
Component: main
|
||||
Origin: Ubuntu
|
||||
Label: Ubuntu
|
||||
Architecture: amd64
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue