diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst deleted file mode 100644 index 0e9946c..0000000 --- a/CONTRIBUTING.rst +++ /dev/null @@ -1,17 +0,0 @@ -If you would like to contribute to the development of OpenStack, you must -follow the steps in this page: - - http://docs.openstack.org/infra/manual/developers.html - -If you already have a good understanding of how the system works and your -OpenStack accounts are set up, you can skip to the development workflow -section of this documentation to learn how changes to OpenStack should be -submitted for review via the Gerrit tool: - - http://docs.openstack.org/infra/manual/developers.html#development-workflow - -Pull requests submitted through GitHub will be ignored. - -Bugs should be filed on Launchpad, not GitHub: - - https://bugs.launchpad.net/fuel diff --git a/HACKING.rst b/HACKING.rst deleted file mode 100644 index 50a409c..0000000 --- a/HACKING.rst +++ /dev/null @@ -1,4 +0,0 @@ -fuel-mirror Style Commandments -============================== - -Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/ diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 8cdb845..0000000 --- a/LICENSE +++ /dev/null @@ -1,340 +0,0 @@ - GNU GENERAL PUBLIC LICENSE - Version 2, June 1991 - - Copyright (C) 1989, 1991 Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA - Everyone is permitted to copy and distribute verbatim copies - of this license document, but changing it is not allowed. - - Preamble - - The licenses for most software are designed to take away your -freedom to share and change it. By contrast, the GNU General Public -License is intended to guarantee your freedom to share and change free -software--to make sure the software is free for all its users. This -General Public License applies to most of the Free Software -Foundation's software and to any other program whose authors commit to -using it. (Some other Free Software Foundation software is covered by -the GNU Lesser General Public License instead.) You can apply it to -your programs, too. - - When we speak of free software, we are referring to freedom, not -price. Our General Public Licenses are designed to make sure that you -have the freedom to distribute copies of free software (and charge for -this service if you wish), that you receive source code or can get it -if you want it, that you can change the software or use pieces of it -in new free programs; and that you know you can do these things. - - To protect your rights, we need to make restrictions that forbid -anyone to deny you these rights or to ask you to surrender the rights. -These restrictions translate to certain responsibilities for you if you -distribute copies of the software, or if you modify it. - - For example, if you distribute copies of such a program, whether -gratis or for a fee, you must give the recipients all the rights that -you have. You must make sure that they, too, receive or can get the -source code. And you must show them these terms so they know their -rights. - - We protect your rights with two steps: (1) copyright the software, and -(2) offer you this license which gives you legal permission to copy, -distribute and/or modify the software. - - Also, for each author's protection and ours, we want to make certain -that everyone understands that there is no warranty for this free -software. If the software is modified by someone else and passed on, we -want its recipients to know that what they have is not the original, so -that any problems introduced by others will not reflect on the original -authors' reputations. - - Finally, any free program is threatened constantly by software -patents. We wish to avoid the danger that redistributors of a free -program will individually obtain patent licenses, in effect making the -program proprietary. To prevent this, we have made it clear that any -patent must be licensed for everyone's free use or not licensed at all. - - The precise terms and conditions for copying, distribution and -modification follow. - - GNU GENERAL PUBLIC LICENSE - TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION - - 0. This License applies to any program or other work which contains -a notice placed by the copyright holder saying it may be distributed -under the terms of this General Public License. The "Program", below, -refers to any such program or work, and a "work based on the Program" -means either the Program or any derivative work under copyright law: -that is to say, a work containing the Program or a portion of it, -either verbatim or with modifications and/or translated into another -language. (Hereinafter, translation is included without limitation in -the term "modification".) Each licensee is addressed as "you". - -Activities other than copying, distribution and modification are not -covered by this License; they are outside its scope. The act of -running the Program is not restricted, and the output from the Program -is covered only if its contents constitute a work based on the -Program (independent of having been made by running the Program). -Whether that is true depends on what the Program does. - - 1. You may copy and distribute verbatim copies of the Program's -source code as you receive it, in any medium, provided that you -conspicuously and appropriately publish on each copy an appropriate -copyright notice and disclaimer of warranty; keep intact all the -notices that refer to this License and to the absence of any warranty; -and give any other recipients of the Program a copy of this License -along with the Program. - -You may charge a fee for the physical act of transferring a copy, and -you may at your option offer warranty protection in exchange for a fee. - - 2. You may modify your copy or copies of the Program or any portion -of it, thus forming a work based on the Program, and copy and -distribute such modifications or work under the terms of Section 1 -above, provided that you also meet all of these conditions: - - a) You must cause the modified files to carry prominent notices - stating that you changed the files and the date of any change. - - b) You must cause any work that you distribute or publish, that in - whole or in part contains or is derived from the Program or any - part thereof, to be licensed as a whole at no charge to all third - parties under the terms of this License. - - c) If the modified program normally reads commands interactively - when run, you must cause it, when started running for such - interactive use in the most ordinary way, to print or display an - announcement including an appropriate copyright notice and a - notice that there is no warranty (or else, saying that you provide - a warranty) and that users may redistribute the program under - these conditions, and telling the user how to view a copy of this - License. (Exception: if the Program itself is interactive but - does not normally print such an announcement, your work based on - the Program is not required to print an announcement.) - -These requirements apply to the modified work as a whole. If -identifiable sections of that work are not derived from the Program, -and can be reasonably considered independent and separate works in -themselves, then this License, and its terms, do not apply to those -sections when you distribute them as separate works. But when you -distribute the same sections as part of a whole which is a work based -on the Program, the distribution of the whole must be on the terms of -this License, whose permissions for other licensees extend to the -entire whole, and thus to each and every part regardless of who wrote it. - -Thus, it is not the intent of this section to claim rights or contest -your rights to work written entirely by you; rather, the intent is to -exercise the right to control the distribution of derivative or -collective works based on the Program. - -In addition, mere aggregation of another work not based on the Program -with the Program (or with a work based on the Program) on a volume of -a storage or distribution medium does not bring the other work under -the scope of this License. - - 3. You may copy and distribute the Program (or a work based on it, -under Section 2) in object code or executable form under the terms of -Sections 1 and 2 above provided that you also do one of the following: - - a) Accompany it with the complete corresponding machine-readable - source code, which must be distributed under the terms of Sections - 1 and 2 above on a medium customarily used for software interchange; or, - - b) Accompany it with a written offer, valid for at least three - years, to give any third party, for a charge no more than your - cost of physically performing source distribution, a complete - machine-readable copy of the corresponding source code, to be - distributed under the terms of Sections 1 and 2 above on a medium - customarily used for software interchange; or, - - c) Accompany it with the information you received as to the offer - to distribute corresponding source code. (This alternative is - allowed only for noncommercial distribution and only if you - received the program in object code or executable form with such - an offer, in accord with Subsection b above.) - -The source code for a work means the preferred form of the work for -making modifications to it. For an executable work, complete source -code means all the source code for all modules it contains, plus any -associated interface definition files, plus the scripts used to -control compilation and installation of the executable. However, as a -special exception, the source code distributed need not include -anything that is normally distributed (in either source or binary -form) with the major components (compiler, kernel, and so on) of the -operating system on which the executable runs, unless that component -itself accompanies the executable. - -If distribution of executable or object code is made by offering -access to copy from a designated place, then offering equivalent -access to copy the source code from the same place counts as -distribution of the source code, even though third parties are not -compelled to copy the source along with the object code. - - 4. You may not copy, modify, sublicense, or distribute the Program -except as expressly provided under this License. Any attempt -otherwise to copy, modify, sublicense or distribute the Program is -void, and will automatically terminate your rights under this License. -However, parties who have received copies, or rights, from you under -this License will not have their licenses terminated so long as such -parties remain in full compliance. - - 5. You are not required to accept this License, since you have not -signed it. However, nothing else grants you permission to modify or -distribute the Program or its derivative works. These actions are -prohibited by law if you do not accept this License. Therefore, by -modifying or distributing the Program (or any work based on the -Program), you indicate your acceptance of this License to do so, and -all its terms and conditions for copying, distributing or modifying -the Program or works based on it. - - 6. Each time you redistribute the Program (or any work based on the -Program), the recipient automatically receives a license from the -original licensor to copy, distribute or modify the Program subject to -these terms and conditions. You may not impose any further -restrictions on the recipients' exercise of the rights granted herein. -You are not responsible for enforcing compliance by third parties to -this License. - - 7. If, as a consequence of a court judgment or allegation of patent -infringement or for any other reason (not limited to patent issues), -conditions are imposed on you (whether by court order, agreement or -otherwise) that contradict the conditions of this License, they do not -excuse you from the conditions of this License. If you cannot -distribute so as to satisfy simultaneously your obligations under this -License and any other pertinent obligations, then as a consequence you -may not distribute the Program at all. For example, if a patent -license would not permit royalty-free redistribution of the Program by -all those who receive copies directly or indirectly through you, then -the only way you could satisfy both it and this License would be to -refrain entirely from distribution of the Program. - -If any portion of this section is held invalid or unenforceable under -any particular circumstance, the balance of the section is intended to -apply and the section as a whole is intended to apply in other -circumstances. - -It is not the purpose of this section to induce you to infringe any -patents or other property right claims or to contest validity of any -such claims; this section has the sole purpose of protecting the -integrity of the free software distribution system, which is -implemented by public license practices. Many people have made -generous contributions to the wide range of software distributed -through that system in reliance on consistent application of that -system; it is up to the author/donor to decide if he or she is willing -to distribute software through any other system and a licensee cannot -impose that choice. - -This section is intended to make thoroughly clear what is believed to -be a consequence of the rest of this License. - - 8. If the distribution and/or use of the Program is restricted in -certain countries either by patents or by copyrighted interfaces, the -original copyright holder who places the Program under this License -may add an explicit geographical distribution limitation excluding -those countries, so that distribution is permitted only in or among -countries not thus excluded. In such case, this License incorporates -the limitation as if written in the body of this License. - - 9. The Free Software Foundation may publish revised and/or new versions -of the General Public License from time to time. Such new versions will -be similar in spirit to the present version, but may differ in detail to -address new problems or concerns. - -Each version is given a distinguishing version number. If the Program -specifies a version number of this License which applies to it and "any -later version", you have the option of following the terms and conditions -either of that version or of any later version published by the Free -Software Foundation. If the Program does not specify a version number of -this License, you may choose any version ever published by the Free Software -Foundation. - - 10. If you wish to incorporate parts of the Program into other free -programs whose distribution conditions are different, write to the author -to ask for permission. For software which is copyrighted by the Free -Software Foundation, write to the Free Software Foundation; we sometimes -make exceptions for this. Our decision will be guided by the two goals -of preserving the free status of all derivatives of our free software and -of promoting the sharing and reuse of software generally. - - NO WARRANTY - - 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY -FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN -OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES -PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED -OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF -MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS -TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE -PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, -REPAIR OR CORRECTION. - - 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING -WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR -REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, -INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING -OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED -TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY -YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER -PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE -POSSIBILITY OF SUCH DAMAGES. - - END OF TERMS AND CONDITIONS - - How to Apply These Terms to Your New Programs - - If you develop a new program, and you want it to be of the greatest -possible use to the public, the best way to achieve this is to make it -free software which everyone can redistribute and change under these terms. - - To do so, attach the following notices to the program. It is safest -to attach them to the start of each source file to most effectively -convey the exclusion of warranty; and each file should have at least -the "copyright" line and a pointer to where the full notice is found. - - {description} - Copyright (C) {year} {fullname} - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License along - with this program; if not, write to the Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -Also add information on how to contact you by electronic and paper mail. - -If the program is interactive, make it output a short notice like this -when it starts in an interactive mode: - - Gnomovision version 69, Copyright (C) year name of author - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. - This is free software, and you are welcome to redistribute it - under certain conditions; type `show c' for details. - -The hypothetical commands `show w' and `show c' should show the appropriate -parts of the General Public License. Of course, the commands you use may -be called something other than `show w' and `show c'; they could even be -mouse-clicks or menu items--whatever suits your program. - -You should also get your employer (if you work as a programmer) or your -school, if any, to sign a "copyright disclaimer" for the program, if -necessary. Here is a sample; alter the names: - - Yoyodyne, Inc., hereby disclaims all copyright interest in the program - `Gnomovision' (which makes passes at compilers) written by James Hacker. - - {signature of Ty Coon}, 1 April 1989 - Ty Coon, President of Vice - -This General Public License does not permit incorporating your program into -proprietary programs. If your program is a subroutine library, you may -consider it more useful to permit linking proprietary applications with the -library. If this is what you want to do, use the GNU Lesser General -Public License instead of this License. - diff --git a/MAINTAINERS b/MAINTAINERS index fd5b24e..98a968b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -26,49 +26,6 @@ description: maintainers: -- contrib/: - - name: Bulat Gaifullin - email: bgaifullin@mirantis.com - IRC: bgaifullin - - - name: Vladimir Kozhukalov - email: vkozhukalov@mirantis.com - IRC: kozhukalov - -- debian/: &packaging_team - - name: Mikhail Ivanov - email: mivanov@mirantis.com - IRC: mivanov - - - name: Artem Silenkov - email: asilenkov@mirantis.com - IRC: asilenkov - - - name: Alexander Tsamutali - email: atsamutali@mirantis.com - IRC: astsmtl - - - name: Daniil Trishkin - email: dtrishkin@mirantis.com - IRC: dtrishkin - - - name: Ivan Udovichenko - email: iudovichenko@mirantis.com - IRC: tlbr - - - name: Igor Yozhikov - email: iyozhikov@mirantis.com - IRC: IgorYozhikov - -- packetary/: - - name: Bulat Gaifullin - email: bgaifullin@mirantis.com - IRC: bgaifullin - - - name: Vladimir Kozhukalov - email: vkozhukalov@mirantis.com - IRC: kozhukalov - - perestroika/: &build_team - name: Dmitry Burmistrov email: dburmistrov@mirantis.com @@ -77,5 +34,3 @@ maintainers: - name: Sergey Kulanov email: skulanov@mirantis.com IRC: SergK - -- specs/: *packaging_team diff --git a/MANIFEST.in b/MANIFEST.in deleted file mode 100644 index 3f60bfe..0000000 --- a/MANIFEST.in +++ /dev/null @@ -1,5 +0,0 @@ -include AUTHORS -include ChangeLog -include *requirements.txt - -global-exclude *.pyc diff --git a/README.rst b/README.rst index 6608033..6c8611a 100644 --- a/README.rst +++ b/README.rst @@ -2,38 +2,6 @@ Repository structure ==================== -* contrib/fuel_mirror - It is a command line utility that provides the same functionality - and user interface as deprecated fuel-createmirror. It provides - two major features: - * clone/build mirror (full or partial) - * update repository configuration in nailgun - First one is a matter of packetary while second one should be left - totally up to fuelclient. So this module is to be deprecated soon - in favor of packetary and fuelclient. - - WARNING: It is not designed to be used on 'live' repositories - that are available to clients during synchronization. That means - repositories will be inconsistent during the update. Please use these - scripts in conjunction with snapshots, on inactive repos, etc. - -* debian - Specs for DEB packages. - -* doc - Documentation for packetary module. - -* packetary - It is a Python library and command line utilty that allows - one to clone and build rpm/deb repositories. - Features: - * Common interface for different package-managers. - * Utility to build dependency graph for package(s). - * Utility to create mirror of repository according to dependency graph. - * perestroika It is a set shell/python script that are used to build DEB/RPM packages. These scripts are widely used by Fuel Packaging CI. - -* specs - Specs for RPM packages. diff --git a/babel.cfg b/babel.cfg deleted file mode 100644 index 15cd6cb..0000000 --- a/babel.cfg +++ /dev/null @@ -1,2 +0,0 @@ -[python: **.py] - diff --git a/contrib/fuel_mirror/MANIFEST.in b/contrib/fuel_mirror/MANIFEST.in deleted file mode 100644 index 5594774..0000000 --- a/contrib/fuel_mirror/MANIFEST.in +++ /dev/null @@ -1,8 +0,0 @@ -include AUTHORS -include ChangeLog -recursive-include etc * - -exclude .gitignore -exclude .gitreview - -global-exclude *.pyc diff --git a/contrib/fuel_mirror/README.rst b/contrib/fuel_mirror/README.rst deleted file mode 100644 index 7f67615..0000000 --- a/contrib/fuel_mirror/README.rst +++ /dev/null @@ -1,16 +0,0 @@ -=========== -fuel_mirror -=========== - -The fuel-mirror is utility, that allows to create local repositories -with packages are required for the OpenStack deployment. - -* Free software: Apache license -* Documentation: http://docs.openstack.org/developer/fuel-mirror -* Source: http://git.openstack.org/cgit/openstack/fuel-mirror/ -* Bugs: http://bugs.launchpad.net/fuel - -Features --------- - -* TODO diff --git a/contrib/fuel_mirror/babel.cfg b/contrib/fuel_mirror/babel.cfg deleted file mode 100644 index 15cd6cb..0000000 --- a/contrib/fuel_mirror/babel.cfg +++ /dev/null @@ -1,2 +0,0 @@ -[python: **.py] - diff --git a/contrib/fuel_mirror/data/centos.yaml b/contrib/fuel_mirror/data/centos.yaml deleted file mode 100644 index 93c11da..0000000 --- a/contrib/fuel_mirror/data/centos.yaml +++ /dev/null @@ -1,55 +0,0 @@ -fuel_release_match: - version: $openstack_version - operating_system: CentOS - -repos: - - ¢os - name: "centos" - uri: "http://vault.centos.org/7.1.1503/os/x86_64/" - type: "rpm" - priority: null - - - ¢os_updates - name: "centos-updates" - uri: "http://vault.centos.org/7.1.1503/updates/x86_64/" - type: "rpm" - priority: null - - - &mos - name: "mos" - uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/os/x86_64/" - type: "rpm" - priority: null - - - &mos_updates - name: "mos-updates" - uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/updates/x86_64/" - type: "rpm" - priority: null - - - &mos_security - name: "mos-security" - uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/security/x86_64/" - type: "rpm" - priority: null - - - &mos_holdback - name: "mos-holdback" - uri: "http://mirror.fuel-infra.org/mos-repos/centos/mos$mos_version-centos7/holdback/x86_64/" - type: "rpm" - priority: null - -groups: - mos: - - *mos - - *mos_updates - - *mos_security - - *mos_holdback - - centos: - - *centos - - *centos_updates - - -inheritance: - centos: mos diff --git a/contrib/fuel_mirror/data/ubuntu.yaml b/contrib/fuel_mirror/data/ubuntu.yaml deleted file mode 100644 index 6c51b9b..0000000 --- a/contrib/fuel_mirror/data/ubuntu.yaml +++ /dev/null @@ -1,158 +0,0 @@ -# GLOBAL variables -ubuntu_baseurl: &ubuntu_baseurl http://archive.ubuntu.com/ubuntu -mos_baseurl: &mos_baseurl http://mirror.fuel-infra.org/mos-repos/ubuntu/$mos_version - -fuel_release_match: - version: $openstack_version - operating_system: Ubuntu - -# Main is a required parameter which defines what repository will be used -# for images creation and that mirror should contain all packages for minimal -# system creation. -repos: - - &ubuntu - main: true - name: "ubuntu" - uri: *ubuntu_baseurl - suite: "trusty" - section: "main multiverse restricted universe" - type: "deb" - priority: null - - - &ubuntu_updates - name: "ubuntu-updates" - uri: *ubuntu_baseurl - suite: "trusty-updates" - section: "main multiverse restricted universe" - type: "deb" - priority: null - - - &ubuntu_security - name: "ubuntu-security" - uri: *ubuntu_baseurl - suite: "trusty-security" - section: "main multiverse restricted universe" - type: "deb" - priority: null - - - &mos - name: "mos" - uri: *mos_baseurl - suite: "mos$mos_version" - section: "main restricted" - type: "deb" - priority: 1000 - - - &mos_updates - name: "mos-updates" - uri: *mos_baseurl - suite: "mos$mos_version-updates" - section: "main restricted" - type: "deb" - priority: 1000 - - - &mos_security - name: "mos-security" - uri: *mos_baseurl - suite: "mos$mos_version-security" - section: "main restricted" - type: "deb" - priority: 1000 - - - &mos_holdback - name: "mos-holdback" - uri: *mos_baseurl - suite: "mos$mos_version-holdback" - section: "main restricted" - type: "deb" - priority: 1000 - -# Packages are required to build bootstrap images for a system. -# The mirror should contiain such packages in addition to local mirror. -packages: &packages - - "acpi-support" - - "anacron" - - "aptitude" - - "atop" - - "acct" - - "bash-completion" - - "bc" - - "build-essential" - - "cloud-init" - - "conntrackd" - - "cpu-checker" - - "cpufrequtils" - - "debconf-utils" - - "devscripts" - - "fping" - - "git" - - "grub-pc" - - "htop" - - "hwloc" - - "ifenslave" - - "iperf" - - "iptables-persistent" - - "irqbalance" - - "language-pack-en" - - "libapache2-mod-fastcgi" - - "libnss3-tools" - - "linux-firmware-nonfree" - - "linux-headers-generic-lts-trusty" - - "linux-image-generic-lts-trusty" - - "live-boot" - - "livecd-rootfs" - - "mc" - - "memcached" - - "monit" - - "msmtp-mta" - - "multipath-tools" - - "multipath-tools-boot" - - "nginx" - - "ntp" - - "openssh-server" - - "percona-toolkit" - - "percona-xtrabackup" - - "pm-utils" - - "puppet" - - "python-lesscpy" - - "python-pip" - - "rsyslog-gnutls" - - "rsyslog-relp" - - "screen" - - "squashfs-tools" - - "swift-plugin-s3" - - "sysfsutils" - - "sysstat" - - "telnet" - - "tmux" - - "traceroute" - - "ubuntu-standard" - - "vim" - - "virt-what" - - "xinetd" - - "xmlstarlet" - - "tftpd-hpa" - - "syslinux" - - -groups: - mos: - - *mos - - *mos_updates - - *mos_security - - *mos_holdback - - ubuntu: - - *ubuntu - - *ubuntu_updates - - *ubuntu_security - - -inheritance: - ubuntu: mos - -osnames: - mos: ubuntu - -requirements: - ubuntu: *packages diff --git a/contrib/fuel_mirror/doc/source/conf.py b/contrib/fuel_mirror/doc/source/conf.py deleted file mode 100644 index 348029b..0000000 --- a/contrib/fuel_mirror/doc/source/conf.py +++ /dev/null @@ -1,79 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import os -import sys - -sys.path.insert(0, os.path.abspath('../..')) -# -- General configuration ---------------------------------------------------- - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = [ - 'sphinx.ext.autodoc', - #'sphinx.ext.intersphinx', - 'oslosphinx' -] - -# autodoc generation is a bit aggressive and a nuisance when doing heavy -# text edit cycles. -# execute "export SPHINX_DEBUG=1" in your terminal to disable - -# The suffix of source filenames. -source_suffix = '.rst' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'fuel_mirror' -copyright = u'2015, Mirantis, Inc' - -# If true, '()' will be appended to :func: etc. cross-reference text. -add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -add_module_names = True - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# -- Options for HTML output -------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. Major themes that come with -# Sphinx are currently 'default' and 'sphinxdoc'. -# html_theme_path = ["."] -# html_theme = '_theme' -# html_static_path = ['static'] - -# Output file base name for HTML help builder. -htmlhelp_basename = '%sdoc' % project - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, author, documentclass -# [howto/manual]). -latex_documents = [ - ('index', - '%s.tex' % project, - u'%s Documentation' % project, - u'OpenStack Foundation', 'manual'), -] - -# Example configuration for intersphinx: refer to the Python standard library. -#intersphinx_mapping = {'http://docs.python.org/': None} diff --git a/contrib/fuel_mirror/doc/source/contributing.rst b/contrib/fuel_mirror/doc/source/contributing.rst deleted file mode 100644 index 36570d8..0000000 --- a/contrib/fuel_mirror/doc/source/contributing.rst +++ /dev/null @@ -1,4 +0,0 @@ -============ -Contributing -============ -.. include:: ../../../../CONTRIBUTING.rst diff --git a/contrib/fuel_mirror/doc/source/index.rst b/contrib/fuel_mirror/doc/source/index.rst deleted file mode 100644 index 2be2378..0000000 --- a/contrib/fuel_mirror/doc/source/index.rst +++ /dev/null @@ -1,25 +0,0 @@ -.. fuel_mirror documentation master file, created by - sphinx-quickstart on Tue Jul 9 22:26:36 2013. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to fuel_mirror's documentation! -======================================================== - -Contents: - -.. toctree:: - :maxdepth: 2 - - readme - installation - usage - contributing - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - diff --git a/contrib/fuel_mirror/doc/source/installation.rst b/contrib/fuel_mirror/doc/source/installation.rst deleted file mode 100644 index be36bdf..0000000 --- a/contrib/fuel_mirror/doc/source/installation.rst +++ /dev/null @@ -1,12 +0,0 @@ -============ -Installation -============ - -At the command line:: - - $ pip install fuel_mirror - -Or, if you have virtualenvwrapper installed:: - - $ mkvirtualenv fuel_mirror - $ pip install fuel_mirror diff --git a/contrib/fuel_mirror/doc/source/readme.rst b/contrib/fuel_mirror/doc/source/readme.rst deleted file mode 100644 index a6210d3..0000000 --- a/contrib/fuel_mirror/doc/source/readme.rst +++ /dev/null @@ -1 +0,0 @@ -.. include:: ../../README.rst diff --git a/contrib/fuel_mirror/doc/source/usage.rst b/contrib/fuel_mirror/doc/source/usage.rst deleted file mode 100644 index a1879eb..0000000 --- a/contrib/fuel_mirror/doc/source/usage.rst +++ /dev/null @@ -1,7 +0,0 @@ -======== -Usage -======== - -To use fuel_mirror in a project:: - - import fuel_createmirror diff --git a/contrib/fuel_mirror/etc/config.yaml b/contrib/fuel_mirror/etc/config.yaml deleted file mode 100644 index e6e9576..0000000 --- a/contrib/fuel_mirror/etc/config.yaml +++ /dev/null @@ -1,11 +0,0 @@ -threads_num: 10 -ignore_errors_num: 2 -retries_num: 3 -target_dir: "/var/www/nailgun/mirrors" -pattern_dir: "/usr/share/fuel-mirror" -base_url: "http://{FUEL_SERVER_IP}:8080/mirrors/" - -# uncomment if need -# http_proxy: null -# https_proxy: null -# fuel_server: 10.20.0.2 diff --git a/contrib/fuel_mirror/fuel_mirror/__init__.py b/contrib/fuel_mirror/fuel_mirror/__init__.py deleted file mode 100644 index 566a696..0000000 --- a/contrib/fuel_mirror/fuel_mirror/__init__.py +++ /dev/null @@ -1,26 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import pbr.version - -try: - __version__ = pbr.version.VersionInfo( - 'fuel_mirror').version_string() -except Exception as e: - print("ERROR", e) - __version__ = "0.0.0-test" diff --git a/contrib/fuel_mirror/fuel_mirror/app.py b/contrib/fuel_mirror/fuel_mirror/app.py deleted file mode 100644 index fca7be0..0000000 --- a/contrib/fuel_mirror/fuel_mirror/app.py +++ /dev/null @@ -1,157 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from cliff import app -from cliff.commandmanager import CommandManager -import yaml - - -import fuel_mirror -from fuel_mirror.common import accessors -from fuel_mirror.common import utils - - -class Application(app.App): - """Main cliff application class. - - Performs initialization of the command manager and - configuration of basic engines. - """ - - config = None - fuel = None - repo_manager_accessor = None - sources = None - versions = None - - def build_option_parser(self, description, version, argparse_kwargs=None): - """Specifies common cmdline arguments.""" - p_inst = super(Application, self) - parser = p_inst.build_option_parser(description=description, - version=version, - argparse_kwargs=argparse_kwargs) - - parser.add_argument( - "--config", - default="/etc/fuel-mirror/config.yaml", - metavar="PATH", - help="Path to config file." - ) - parser.add_argument( - "-S", "--fuel-server", - metavar="FUEL-SERVER", - help="The public address of Fuel Master." - ) - parser.add_argument( - "--fuel-user", - help="Fuel Master admin login." - " Alternatively, use env var KEYSTONE_USER)." - ) - parser.add_argument( - "--fuel-password", - help="Fuel Master admin password." - " Alternatively, use env var KEYSTONE_PASSWORD)." - ) - return parser - - def initialize_app(self, argv): - """Initialises common options.""" - with open(self.options.config, "r") as stream: - self.config = yaml.load(stream) - - self._initialize_fuel_accessor() - self._initialize_repo_manager() - - def _initialize_repo_manager(self): - self.repo_manager_accessor = accessors.get_packetary_accessor( - threads_num=int(self.config.get('threads_num', 0)), - retries_num=int(self.config.get('retries_num', 0)), - ignore_errors_num=int(self.config.get('ignore_errors_num', 0)), - http_proxy=self.config.get('http_proxy'), - https_proxy=self.config.get('https_proxy'), - ) - - def _initialize_fuel_accessor(self): - fuel_default = utils.get_fuel_settings() - - fuel_server = utils.first( - self.options.fuel_server, - self.config.get("fuel_server"), - fuel_default.get("server") - ) - fuel_user = utils.first( - self.options.fuel_user, - fuel_default.get("user") - ) - fuel_password = utils.first( - self.options.fuel_password, - fuel_default.get("password") - ) - - if not fuel_server: - for option in ("mos_version", "openstack_version"): - if not self.config.setdefault(option, ''): - self.LOG.warning( - "The option '{0}' is not defined." - "Please specify the option 'fuel-server' or '{0}'." - .format(option) - ) - return - - self.config["base_url"] = self.config["base_url"].format( - FUEL_SERVER_IP=fuel_server.split(":", 1)[0] - ) - - self.fuel = accessors.get_fuel_api_accessor( - fuel_server, - fuel_user, - fuel_password - ) - fuel_ver = self.fuel.FuelVersion.get_all_data() - self.config.setdefault( - 'mos_version', fuel_ver['release'] - ) - self.config.setdefault( - 'openstack_version', fuel_ver['openstack_version'] - ) - - -def main(argv=None): - """Entry point.""" - return Application( - description="The utility to create local mirrors.", - version=fuel_mirror.__version__, - command_manager=CommandManager("fuel_mirror", convert_underscores=True) - ).run(argv) - - -def debug(name, cmd_class, argv=None): - """Helps to debug command.""" - import sys - - if argv is None: - argv = sys.argv[1:] - - argv = [name] + argv + ["-v", "-v", "--debug"] - cmd_mgr = CommandManager("test_fuel_mirror", convert_underscores=True) - cmd_mgr.add_command(name, cmd_class) - return Application( - description="The fuel mirror utility test.", - version="0.0.1", - command_manager=cmd_mgr - ).run(argv) diff --git a/contrib/fuel_mirror/fuel_mirror/commands/__init__.py b/contrib/fuel_mirror/fuel_mirror/commands/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/contrib/fuel_mirror/fuel_mirror/commands/apply.py b/contrib/fuel_mirror/fuel_mirror/commands/apply.py deleted file mode 100644 index 634e6b4..0000000 --- a/contrib/fuel_mirror/fuel_mirror/commands/apply.py +++ /dev/null @@ -1,203 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import six - -from packetary.library.utils import localize_repo_url - -from fuel_mirror.commands.base import BaseCommand -from fuel_mirror.common.utils import is_subdict -from fuel_mirror.common.utils import lists_merge - - -class ApplyCommand(BaseCommand): - """Applies local mirrors for Fuel-environments.""" - - def get_parser(self, prog_name): - parser = super(ApplyCommand, self).get_parser(prog_name) - parser.add_argument( - "--default", - dest="set_default", - action="store_true", - default=False, - help="Set as default repository." - ) - parser.add_argument( - "--replace", - dest="replace", - action="store_true", - default=False, - help="Replace default repos with generated mirrors." - ) - parser.add_argument( - "-e", "--env", - dest="env", nargs="+", - help="Fuel environment ID to update, " - "by default applies for all environments." - ) - - return parser - - def take_action(self, parsed_args): - if self.app.fuel is None: - raise ValueError("Please specify the fuel-server option.") - - data = self.load_data(parsed_args) - base_url = self.app.config["base_url"] - release_match = data["fuel_release_match"] - replace_repos = parsed_args.replace - - localized_repos = [] - for _, repos in self.get_groups(parsed_args, data): - for repo_data in repos: - new_data = repo_data.copy() - new_data['uri'] = localize_repo_url( - base_url, repo_data['uri'] - ) - localized_repos.append(new_data) - - localized_repos.sort(key=lambda x: not x.pop('main', False)) - - self.update_clusters( - parsed_args.env, - localized_repos, - release_match, - replace_repos=replace_repos) - - if parsed_args.set_default: - self.update_release_repos( - localized_repos, - release_match, - replace_repos=replace_repos) - - self.app.stdout.write( - "Operations have been completed successfully.\n" - ) - - def update_clusters(self, - ids, - repositories, - release_match, - replace_repos=False): - """Applies repositories for existing clusters. - - :param ids: the cluster ids. - :param repositories: the meta information of repositories - :param release_match: The pattern to check Fuel Release - """ - self.app.stdout.write("Updating the Cluster repositories...\n") - - if ids: - clusters = self.app.fuel.Environment.get_by_ids(ids) - else: - clusters = self.app.fuel.Environment.get_all() - - for cluster in clusters: - releases = six.moves.filter( - lambda x: is_subdict(release_match, x.data), - self.app.fuel.Release.get_by_ids([cluster.data["release_id"]]) - ) - if next(releases, None) is None: - continue - - modified = self._update_repository_settings( - cluster.get_settings_data(), - repositories, - replace_repos=replace_repos) - - if modified: - self.app.LOG.info( - "Try to update the Cluster '%s'", - cluster.data['name'] - ) - self.app.LOG.debug( - "The modified cluster attributes: %s", - modified - ) - cluster.set_settings_data(modified) - - def update_release_repos(self, - repositories, - release_match, - replace_repos=False): - """Applies repositories for existing default settings. - - :param repositories: the meta information of repositories - :param release_match: The pattern to check Fuel Release - """ - self.app.stdout.write("Updating the release repositories...\n") - releases = six.moves.filter( - lambda x: is_subdict(release_match, x.data), - self.app.fuel.Release.get_all() - ) - for release in releases: - modified = self._update_repository_settings( - release.data["attributes_metadata"], - repositories, - replace_repos=replace_repos) - if modified: - release.data["attributes_metadata"] = modified - self.app.LOG.info( - "Try to update the Release '%s'", - release.data['name'] - ) - self.app.LOG.debug( - "The modified release attributes: %s", - release.data - ) - # TODO(need to add method for release object) - release.connection.put_request( - release.instance_api_path.format(release.id), - release.data - ) - - def _update_repository_settings(self, - settings, - repositories, - replace_repos=False): - """Updates repository settings. - - :param settings: the target settings - :param repositories: the meta of repositories - """ - editable = settings["editable"] - if 'repo_setup' not in editable: - self.app.LOG.info('Attributes are read-only.') - return - - repos_attr = editable["repo_setup"]["repos"] - if replace_repos: - repos_attr['value'] = repositories - else: - lists_merge(repos_attr['value'], repositories, "name") - - # NOTE(akostrikov) That assignment is only for informational purpose. - settings["editable"]["repo_setup"]["repos"] = repos_attr - - return settings - - -def debug(argv=None): - """Helper for debugging Apply command.""" - from fuel_mirror.app import debug - - return debug("apply", ApplyCommand, argv) - - -if __name__ == "__main__": - debug() diff --git a/contrib/fuel_mirror/fuel_mirror/commands/base.py b/contrib/fuel_mirror/fuel_mirror/commands/base.py deleted file mode 100644 index 613d177..0000000 --- a/contrib/fuel_mirror/fuel_mirror/commands/base.py +++ /dev/null @@ -1,125 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import os.path - -from cliff import command -from jsonschema import validate -from jsonschema import ValidationError -import six - -from fuel_mirror.common.utils import load_input_data -from fuel_mirror.schemas.input_data_schema import SCHEMA - - -class BaseCommand(command.Command): - - """The Base command for fuel-mirror.""" - REPO_ARCH = "x86_64" - - @property - def stdout(self): - """Shortcut for self.app.stdout.""" - return self.app.stdout - - def get_parser(self, prog_name): - """Specifies common options.""" - parser = super(BaseCommand, self).get_parser(prog_name) - - input_group = parser.add_mutually_exclusive_group(required=True) - input_group.add_argument( - '-I', '--input-file', - metavar='PATH', - help='The path to file with input data.') - - input_group.add_argument( - '-P', '--pattern', - metavar='NAME', - help='The builtin input file name.' - ) - - parser.add_argument( - "-G", "--group", - dest="groups", - required=True, - nargs='+', - help="The name of repository groups." - ) - return parser - - def resolve_input_pattern(self, pattern): - """Gets the full path to input file by pattern. - - :param pattern: the config file name without ext - :return: the full path - """ - return os.path.join( - self.app.config['pattern_dir'], pattern + ".yaml" - ) - - @staticmethod - def validate_data(data, schema): - """Validate the input data using jsonschema validation. - - :param data: a data to validate represented as a dict - :param schema: a schema to validate represented as a dict; - must be in JSON Schema Draft 4 format. - """ - try: - validate(data, schema) - except ValidationError as ex: - if len(ex.path) > 0: - join_ex_path = '.'.join(six.text_type(x) for x in ex.path) - detail = ("Invalid input for field/attribute {0}." - " Value: {1}. {2}").format(join_ex_path, - ex.instance, ex.message) - else: - detail = ex.message - raise ValidationError(detail) - - def load_data(self, parsed_args): - """Load the input data. - - :param parsed_args: the command-line arguments - :return: the input data - """ - if parsed_args.pattern: - input_file = self.resolve_input_pattern(parsed_args.pattern) - else: - input_file = parsed_args.input_file - - data = load_input_data( - input_file, - mos_version=self.app.config["mos_version"], - openstack_version=self.app.config["openstack_version"] - ) - self.validate_data(data, SCHEMA) - return data - - @classmethod - def get_groups(cls, parsed_args, data): - """Gets repository groups from input data. - - :param parsed_args: the command-line arguments - :param data: the input data - :return: the sequence of pairs (group_name, repositories) - """ - all_groups = data['groups'] - return ( - (x, all_groups[x]) for x in parsed_args.groups if x in all_groups - ) diff --git a/contrib/fuel_mirror/fuel_mirror/commands/create.py b/contrib/fuel_mirror/fuel_mirror/commands/create.py deleted file mode 100644 index 484d601..0000000 --- a/contrib/fuel_mirror/fuel_mirror/commands/create.py +++ /dev/null @@ -1,79 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from fuel_mirror.commands.base import BaseCommand -from fuel_mirror.common.url_builder import get_url_builder - - -class CreateCommand(BaseCommand): - """Creates a new local mirrors.""" - - def take_action(self, parsed_args): - """See the Command.take_action.""" - data = self.load_data(parsed_args) - repos_reqs = data.get('requirements', {}) - inheritance = data.get('inheritance', {}) - target_dir = self.app.config["target_dir"] - - total_stats = None - for group_name, repos in self.get_groups(parsed_args, data): - url_builder = get_url_builder(repos[0]["type"]) - repo_manager = self.app.repo_manager_accessor( - repos[0]["type"], self.REPO_ARCH - ) - if group_name in inheritance: - child_group = inheritance[group_name] - dependencies = [ - url_builder.get_repo_url(x) - for x in data['groups'][child_group] - ] - else: - dependencies = None - - stat = repo_manager.clone_repositories( - [url_builder.get_repo_url(x) for x in repos], - target_dir, - dependencies, - repos_reqs.get(group_name) - ) - - if total_stats is None: - total_stats = stat - else: - total_stats += stat - - if total_stats is not None: - self.stdout.write( - "Packages processed: {0.copied}/{0.total}\n" - .format(total_stats) - ) - else: - self.stdout.write( - "No packages.\n" - ) - - -def debug(argv=None): - """Helper for debugging Create command.""" - from fuel_mirror.app import debug - - return debug("create", CreateCommand, argv) - - -if __name__ == "__main__": - debug() diff --git a/contrib/fuel_mirror/fuel_mirror/common/__init__.py b/contrib/fuel_mirror/fuel_mirror/common/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/contrib/fuel_mirror/fuel_mirror/common/accessors.py b/contrib/fuel_mirror/fuel_mirror/common/accessors.py deleted file mode 100644 index 35425f6..0000000 --- a/contrib/fuel_mirror/fuel_mirror/common/accessors.py +++ /dev/null @@ -1,64 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import functools -import os - - -def get_packetary_accessor(**kwargs): - """Gets the configured repository manager. - - :param kwargs: The packetary configuration parameters. - """ - - import packetary - - return functools.partial( - packetary.RepositoryApi.create, - packetary.Context(packetary.Configuration(**kwargs)) - ) - - -def get_fuel_api_accessor(address=None, user=None, password=None): - """Gets the fuel client api accessor. - - :param address: The address of Fuel Master node. - :param user: The username to access to the Fuel Master node. - :param user: The password to access to the Fuel Master node. - """ - if address: - host_and_port = address.split(":") - os.environ["SERVER_ADDRESS"] = host_and_port[0] - if len(host_and_port) > 1: - os.environ["LISTEN_PORT"] = host_and_port[1] - - if user is not None: - os.environ["KEYSTONE_USER"] = user - if password is not None: - os.environ["KEYSTONE_PASS"] = password - - # import fuelclient.ClientAPI after configuring - # environment variables - try: - from fuelclient import objects - except ImportError: - raise RuntimeError( - "fuelclient module seems not installed. " - "This action requires it to be available." - ) - return objects diff --git a/contrib/fuel_mirror/fuel_mirror/common/url_builder.py b/contrib/fuel_mirror/fuel_mirror/common/url_builder.py deleted file mode 100644 index be13a8d..0000000 --- a/contrib/fuel_mirror/fuel_mirror/common/url_builder.py +++ /dev/null @@ -1,59 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - -def get_url_builder(repotype): - """Gets the instance of RepoUrlBuilder. - - :param repotype: the type of repository: rpm|deb - :return: the RepoBuilder implementation - """ - return { - "deb": AptRepoUrlBuilder, - "rpm": YumRepoUrlBuilder - }[repotype] - - -class RepoUrlBuilder(object): - REPO_FOLDER = "mirror" - - @classmethod - def get_repo_url(cls, repo_data): - """Gets the url with replaced variable holders. - - :param repo_data: the repositories`s meta data - :return: the full repository`s url - """ - - -class AptRepoUrlBuilder(RepoUrlBuilder): - """URL builder for apt-repository(es).""" - - @classmethod - def get_repo_url(cls, repo_data): - return " ".join( - repo_data[x] for x in ("uri", "suite", "section") - ) - - -class YumRepoUrlBuilder(RepoUrlBuilder): - """URL builder for Yum repository(es).""" - - @classmethod - def get_repo_url(cls, repo_data): - return repo_data["uri"] diff --git a/contrib/fuel_mirror/fuel_mirror/common/utils.py b/contrib/fuel_mirror/fuel_mirror/common/utils.py deleted file mode 100644 index 855ef00..0000000 --- a/contrib/fuel_mirror/fuel_mirror/common/utils.py +++ /dev/null @@ -1,106 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -from string import Template - -import six -import yaml - - -def lists_merge(main, patch, key): - """Merges the list of dicts with same keys. - - >>> lists_merge([{"a": 1, "c": 2}], [{"a": 1, "c": 3}], key="a") - [{'a': 1, 'c': 3}] - - :param main: the main list - :type main: list - :param patch: the list of additional elements - :type patch: list - :param key: the key for compare - """ - main_idx = dict( - (x[key], i) for i, x in enumerate(main) - ) - - patch_idx = dict( - (x[key], i) for i, x in enumerate(patch) - ) - - for k in sorted(patch_idx): - if k in main_idx: - main[main_idx[k]].update(patch[patch_idx[k]]) - else: - main.append(patch[patch_idx[k]]) - return main - - -def is_subdict(dict1, dict2): - """Checks that dict1 is subdict of dict2. - - >>> is_subdict({"a": 1}, {'a': 1, 'b': 1}) - True - - :param dict1: the candidate - :param dict2: the super dict - :return: True if all keys from dict1 are present - and has same value in dict2 otherwise False - """ - for k, v in six.iteritems(dict1): - if k not in dict2 or dict2[k] != v: - return False - return True - - -def first(*args): - """Get first not empty value. - - >>> first(0, 1) == next(iter(filter(None, [0, 1]))) - True - - :param args: the list of arguments - :return first value that bool(v) is True, None if not found. - """ - for arg in args: - if arg: - return arg - - -def get_fuel_settings(): - """Gets the fuel settings from astute container, if it is available.""" - - try: - with open("/etc/fuel/astute.yaml", "r") as fd: - settings = yaml.load(fd) - return { - "server": settings.get("ADMIN_NETWORK", {}).get("ipaddress"), - "user": settings.get("FUEL_ACCESS", {}).get("user"), - "password": settings.get("FUEL_ACCESS", {}).get("password") - } - except (OSError, IOError): - return {} - - -def load_input_data(input_file, **kwargs): - """Load yaml file and parse it to dict with replacement by kwargs. - - :param input_file: name of file to parse fuel mirror template - :param kwargs: arguments to substitute template - :return: processed from yaml file dict. - """ - with open(input_file, "r") as fd: - return yaml.load(Template(fd.read()).safe_substitute(**kwargs)) diff --git a/contrib/fuel_mirror/fuel_mirror/schemas/__init__.py b/contrib/fuel_mirror/fuel_mirror/schemas/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/contrib/fuel_mirror/fuel_mirror/schemas/input_data_schema.py b/contrib/fuel_mirror/fuel_mirror/schemas/input_data_schema.py deleted file mode 100644 index 0cf52a1..0000000 --- a/contrib/fuel_mirror/fuel_mirror/schemas/input_data_schema.py +++ /dev/null @@ -1,120 +0,0 @@ -SCHEMA = { - "$schema": "http://json-schema.org/draft-04/schema#", - "definitions": { - "DEB_REPO_SCHEMA": { - "type": "object", - "required": [ - "name", - "uri", - "suite", - "section" - ], - "properties": { - "name": { - "type": "string" - }, - "type": { - "type": "string", - "enum": ["deb"] - }, - "uri": { - "type": "string" - }, - "priority": { - "anyOf": [ - { - "type": "integer" - }, - { - "type": "null" - } - ] - }, - "suite": { - "type": "string" - }, - "section": { - "type": "string" - }, - } - }, - "RPM_REPO_SCHEMA": { - "type": "object", - "required": [ - "name", - "uri", - ], - "properties": { - "name": { - "type": "string" - }, - "type": { - "type": "string", - "enum": ["rpm"] - }, - "uri": { - "type": "string" - }, - "priority": { - "anyOf": [ - { - "type": "integer" - }, - { - "type": "null" - } - ] - }, - } - }, - "REPO_SCHEMA": { - "anyOf": - [ - {"$ref": "#/definitions/DEB_REPO_SCHEMA"}, - {"$ref": "#/definitions/RPM_REPO_SCHEMA"} - ] - }, - - "REPOS_SCHEMA": { - "type": "array", "items": {"$ref": "#/definitions/REPO_SCHEMA"} - } - }, - "type": "object", - "required": [ - "groups", - ], - "properties": { - "fuel_release_match": { - "type": "object", - "properties": { - "operating_system": { - "type": "string" - } - }, - "required": [ - "operating_system" - ] - }, - "requirements": { - "type": "object", - "patternProperties": { - "^[0-9a-z_-]+$": {"type": "array"} - }, - "additionalProperties": False, - }, - "groups": { - "type": "object", - "patternProperties": { - "^[0-9a-z_-]+$": {"$ref": "#/definitions/REPOS_SCHEMA"} - }, - "additionalProperties": False, - }, - "inheritance": { - "type": "object", - "patternProperties": { - "^[0-9a-z_-]+$": {"type": "string"} - }, - "additionalProperties": False, - } - } -} diff --git a/contrib/fuel_mirror/fuel_mirror/tests/__init__.py b/contrib/fuel_mirror/fuel_mirror/tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/contrib/fuel_mirror/fuel_mirror/tests/base.py b/contrib/fuel_mirror/fuel_mirror/tests/base.py deleted file mode 100644 index 3e5d0c9..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/base.py +++ /dev/null @@ -1,33 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -try: - import unittest2 as unittest -except ImportError: - import unittest - - -class TestCase(unittest.TestCase): - - """Test case base class for all unit tests.""" - - def assertNotRaises(self, exception, method, *args, **kwargs): - try: - method(*args, **kwargs) - except exception as e: - self.fail("Unexpected error: {0}".format(e)) diff --git a/contrib/fuel_mirror/fuel_mirror/tests/data/test_centos.yaml b/contrib/fuel_mirror/fuel_mirror/tests/data/test_centos.yaml deleted file mode 100644 index 11f69bf..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/data/test_centos.yaml +++ /dev/null @@ -1,22 +0,0 @@ -fuel_release_match: - operating_system: CentOS - -inheritance: - centos: mos - -groups: - mos: - - name: "mos" - type: "rpm" - uri: "http://localhost/mos$mos_version/x86_64" - priority: 10 - - centos: - - name: "centos" - type: "rpm" - uri: "http://localhost/centos/os/x86_64" - priority: 5 - -requirements: - centos: - - "package_rpm" diff --git a/contrib/fuel_mirror/fuel_mirror/tests/data/test_config.yaml b/contrib/fuel_mirror/fuel_mirror/tests/data/test_config.yaml deleted file mode 100644 index 3ba5cd1..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/data/test_config.yaml +++ /dev/null @@ -1,7 +0,0 @@ -threads_num: 1 -ignore_errors_num: 2 -retries_num: 3 -http_proxy: "http://localhost" -https_proxy: "https://localhost" -target_dir: "/var/www/" -base_url: "http://{FUEL_SERVER_IP}:8080/" diff --git a/contrib/fuel_mirror/fuel_mirror/tests/data/test_invalid_ubuntu.yaml b/contrib/fuel_mirror/fuel_mirror/tests/data/test_invalid_ubuntu.yaml deleted file mode 100644 index a5071f4..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/data/test_invalid_ubuntu.yaml +++ /dev/null @@ -1,9 +0,0 @@ -fuel_release_match: - operating_system: Ubuntu - -inheritance: - ubuntu: mos - -requirements: - ubuntu: - - "package_deb" diff --git a/contrib/fuel_mirror/fuel_mirror/tests/data/test_ubuntu.yaml b/contrib/fuel_mirror/fuel_mirror/tests/data/test_ubuntu.yaml deleted file mode 100644 index 9554ef0..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/data/test_ubuntu.yaml +++ /dev/null @@ -1,27 +0,0 @@ -fuel_release_match: - operating_system: Ubuntu - -inheritance: - ubuntu: mos - -groups: - mos: - - name: "mos" - type: "deb" - uri: "http://localhost/mos" - suite: "mos$mos_version" - section: "main restricted" - priority: 1000 - - ubuntu: - - name: "ubuntu" - type: "deb" - main: true - uri: "http://localhost/ubuntu" - suite: "trusty" - section: "main multiverse restricted universe" - priority: 500 - -requirements: - ubuntu: - - "package_deb" diff --git a/contrib/fuel_mirror/fuel_mirror/tests/test_accessors.py b/contrib/fuel_mirror/fuel_mirror/tests/test_accessors.py deleted file mode 100644 index cbc4ea9..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/test_accessors.py +++ /dev/null @@ -1,92 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock - -from fuel_mirror.common import accessors -from fuel_mirror.tests import base - - -class TestAccessors(base.TestCase): - def test_get_packetary_accessor(self): - packetary = mock.MagicMock() - with mock.patch.dict("sys.modules", packetary=packetary): - accessor = accessors.get_packetary_accessor( - http_proxy="http://localhost", - https_proxy="https://localhost", - retries_num=1, - threads_num=2, - ignore_errors_num=3 - ) - accessor("deb") - accessor("yum") - packetary.Configuration.assert_called_once_with( - http_proxy="http://localhost", - https_proxy="https://localhost", - retries_num=1, - threads_num=2, - ignore_errors_num=3 - ) - packetary.Context.assert_called_once_with( - packetary.Configuration() - ) - self.assertEqual(2, packetary.RepositoryApi.create.call_count) - packetary.RepositoryApi.create.assert_any_call( - packetary.Context(), "deb" - ) - packetary.RepositoryApi.create.assert_any_call( - packetary.Context(), "yum" - ) - - @mock.patch("fuel_mirror.common.accessors.os") - def test_get_fuel_api_accessor(self, os): - fuelclient = mock.MagicMock() - patch = { - "fuelclient": fuelclient, - "fuelclient.objects": fuelclient.objects - } - with mock.patch.dict("sys.modules", patch): - accessor = accessors.get_fuel_api_accessor( - "localhost:8080", "guest", "123" - ) - accessor.Environment.get_all() - - os.environ.__setitem__.asseert_any_call( - "SERVER_ADDRESS", "localhost" - ) - os.environ.__setitem__.asseert_any_call( - "LISTEN_PORT", "8080" - ) - os.environ.__setitem__.asseert_any_call( - "KEYSTONE_USER", "guest" - ) - os.environ.__setitem__.asseert_any_call( - "KEYSTONE_PASS", "123" - ) - fuelclient.objects.Environment.get_all.assert_called_once_with() - - @mock.patch("fuel_mirror.common.accessors.os") - def test_get_fuel_api_accessor_with_default_parameters(self, os): - fuelclient = mock.MagicMock() - patch = { - "fuelclient": fuelclient, - "fuelclient.objects": fuelclient.objects - } - with mock.patch.dict("sys.modules", patch): - accessors.get_fuel_api_accessor() - os.environ.__setitem__.assert_not_called() diff --git a/contrib/fuel_mirror/fuel_mirror/tests/test_cli_commands.py b/contrib/fuel_mirror/fuel_mirror/tests/test_cli_commands.py deleted file mode 100644 index ab2bd0b..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/test_cli_commands.py +++ /dev/null @@ -1,499 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock -import os.path -import subprocess - -from jsonschema import ValidationError - -# The cmd2 does not work with python3.5 -# because it tries to get access to the property mswindows, -# that was removed in 3.5 -subprocess.mswindows = False - -from fuel_mirror.commands import apply -from fuel_mirror.commands import create -from fuel_mirror.common.utils import load_input_data -from fuel_mirror.tests import base - - -CONFIG_PATH = os.path.join( - os.path.dirname(__file__), "data", "test_config.yaml" -) - -UBUNTU_PATH = os.path.join( - os.path.dirname(__file__), "data", "test_ubuntu.yaml" -) - -CENTOS_PATH = os.path.join( - os.path.dirname(__file__), "data", "test_centos.yaml" -) - -INVALID_DATA_PATH = os.path.join( - os.path.dirname(__file__), "data", "test_invalid_ubuntu.yaml" -) - - -# TODO(akostrikov) lists_merge we are using is not stable so we have to use -# different local repos in cases with merge and without it. -# We pass sorted by priority list, but in lists_merge we sort it by key. -# As we are aiming to use existing repos as primary source - it is not issue. -def local_repos(mirror_host='10.25.0.10:8080', name_postfix='', reverse=True): - mirror_lists = load_input_data(UBUNTU_PATH, mos_version=1) - sorted_repos = reduce(lambda x, y: x + y, mirror_lists['groups'].values()) - sorted_repos.sort(key=lambda x: x['priority'], reverse=reverse) - for repo in sorted_repos: - repo.pop('main', None) - repo['name'] = repo['name'] + name_postfix - repo['uri'] = repo['uri'].replace('localhost', mirror_host) - return sorted_repos - - -def mirror_repos(): - return local_repos(mirror_host='mirror.com:8080', name_postfix='-mirror') - - -@mock.patch.multiple( - "fuel_mirror.app", - accessors=mock.DEFAULT -) -class TestCliCommands(base.TestCase): - common_argv = [ - "--config", CONFIG_PATH, - "--fuel-server=10.25.0.10", - "--fuel-user=test", - "--fuel-password=test1" - ] - - def start_cmd(self, cmd, argv, data_file): - cmd.debug( - argv + self.common_argv + ["--input-file", data_file] - ) - - def _setup_fuel_versions(self, fuel_mock): - fuel_mock.FuelVersion.get_all_data.return_value = { - "release": "1", - "openstack_version": "2" - } - - def _create_fuel_release(self, fuel_mock, osname, repos=None): - if repos is None: - repos = [] - release = mock.MagicMock(data={ - "name": "test release", - "operating_system": osname, - "attributes_metadata": { - "editable": {"repo_setup": {"repos": {"value": repos}}} - } - }) - - fuel_mock.Release.get_by_ids.return_value = [release] - fuel_mock.Release.get_all.return_value = [release] - return release - - def _create_fuel_env(self, fuel_mock, repos=None): - if repos is None: - repos = [] - env = mock.MagicMock(data={ - "name": "test", - "release_id": 1 - }) - env.get_settings_data.return_value = { - "editable": {"repo_setup": {"repos": {"value": repos}}} - } - fuel_mock.Environment.get_by_ids.return_value = [env] - fuel_mock.Environment.get_all.return_value = [env] - return env - - def test_create_mos_ubuntu(self, accessors): - self._setup_fuel_versions(accessors.get_fuel_api_accessor()) - packetary = accessors.get_packetary_accessor() - - self.start_cmd(create, ["--group", "mos"], UBUNTU_PATH) - accessors.get_packetary_accessor.assert_called_with( - threads_num=1, - ignore_errors_num=2, - retries_num=3, - http_proxy="http://localhost", - https_proxy="https://localhost", - ) - packetary.assert_called_with("deb", "x86_64") - api = packetary() - api.clone_repositories.assert_called_once_with( - ['http://localhost/mos mos1 main restricted'], - '/var/www/', - None, None - ) - - def test_create_partial_ubuntu(self, accessors): - self._setup_fuel_versions(accessors.get_fuel_api_accessor()) - packetary = accessors.get_packetary_accessor() - - self.start_cmd(create, ["--group", "ubuntu"], UBUNTU_PATH) - accessors.get_packetary_accessor.assert_called_with( - threads_num=1, - ignore_errors_num=2, - retries_num=3, - http_proxy="http://localhost", - https_proxy="https://localhost", - ) - packetary.assert_called_with("deb", "x86_64") - api = packetary() - api.clone_repositories.assert_called_once_with( - ['http://localhost/ubuntu trusty ' - 'main multiverse restricted universe'], - '/var/www/', - ['http://localhost/mos mos1 main restricted'], - ['package_deb'] - ) - - def test_create_mos_centos(self, accessors): - self._setup_fuel_versions(accessors.get_fuel_api_accessor()) - packetary = accessors.get_packetary_accessor() - - self.start_cmd(create, ["--group", "mos"], CENTOS_PATH) - accessors.get_packetary_accessor.assert_called_with( - threads_num=1, - ignore_errors_num=2, - retries_num=3, - http_proxy="http://localhost", - https_proxy="https://localhost", - ) - packetary.assert_called_with("rpm", "x86_64") - api = packetary() - api.clone_repositories.assert_called_once_with( - ['http://localhost/mos1/x86_64'], - '/var/www/', - None, None - ) - - def test_create_partial_centos(self, accessors): - self._setup_fuel_versions(accessors.get_fuel_api_accessor()) - packetary = accessors.get_packetary_accessor() - - self.start_cmd(create, ["--group", "centos"], CENTOS_PATH) - accessors.get_packetary_accessor.assert_called_with( - threads_num=1, - ignore_errors_num=2, - retries_num=3, - http_proxy="http://localhost", - https_proxy="https://localhost", - ) - packetary.assert_called_with("rpm", "x86_64") - api = packetary() - api.clone_repositories.assert_called_once_with( - ['http://localhost/centos/os/x86_64'], - '/var/www/', - ['http://localhost/mos1/x86_64'], - ["package_rpm"] - ) - - def test_apply_for_ubuntu_based_env(self, accessors): - fuel = accessors.get_fuel_api_accessor() - self._setup_fuel_versions(fuel) - env = self._create_fuel_env(fuel) - self._create_fuel_release(fuel, "Ubuntu") - self.start_cmd( - apply, ['--group', 'mos', 'ubuntu', '--env', '1'], - UBUNTU_PATH - ) - accessors.get_fuel_api_accessor.assert_called_with( - "10.25.0.10", "test", "test1" - ) - fuel.FuelVersion.get_all_data.assert_called_once_with() - env.set_settings_data.assert_called_with( - { - 'editable': { - 'repo_setup': { - 'repos': {'value': local_repos()} - } - } - } - ) - - def test_with_existing_mirrors(self, accessors): - fuel = accessors.get_fuel_api_accessor() - self._setup_fuel_versions(fuel) - env = self._create_fuel_env(fuel, repos=mirror_repos()) - self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos()) - self.start_cmd( - apply, ['--group', 'mos', 'ubuntu', '--env', '1'], - UBUNTU_PATH - ) - accessors.get_fuel_api_accessor.assert_called_with( - "10.25.0.10", "test", "test1" - ) - fuel.FuelVersion.get_all_data.assert_called_once_with() - env.set_settings_data.assert_called_with( - { - 'editable': { - 'repo_setup': { - 'repos': {'value': mirror_repos() + local_repos()} - } - } - } - ) - - def test_replace_existing_mirrors_with_local(self, accessors): - fuel = accessors.get_fuel_api_accessor() - self._setup_fuel_versions(fuel) - env = self._create_fuel_env(fuel, repos=mirror_repos()) - self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos()) - self.start_cmd( - apply, ['--group', 'mos', 'ubuntu', '--env', '1', '--replace'], - UBUNTU_PATH - ) - accessors.get_fuel_api_accessor.assert_called_with( - "10.25.0.10", "test", "test1" - ) - fuel.FuelVersion.get_all_data.assert_called_once_with() - env.set_settings_data.assert_called_with( - { - 'editable': { - 'repo_setup': { - 'repos': {'value': local_repos(reverse=False)} - } - } - } - ) - - def test_apply_for_centos_based_env(self, accessors): - fuel = accessors.get_fuel_api_accessor() - self._setup_fuel_versions(fuel) - env = self._create_fuel_env(fuel) - self._create_fuel_release(fuel, "CentOS") - self.start_cmd( - apply, ['--group', 'mos', 'centos', '--env', '1'], - CENTOS_PATH - ) - accessors.get_fuel_api_accessor.assert_called_with( - "10.25.0.10", "test", "test1" - ) - fuel.FuelVersion.get_all_data.assert_called_once_with() - env.set_settings_data.assert_called_with( - {'editable': {'repo_setup': {'repos': {'value': [ - { - 'priority': 5, - 'name': 'centos', - 'type': 'rpm', - 'uri': 'http://10.25.0.10:8080/centos/os/x86_64' - }, - { - 'priority': 10, - 'name': 'mos', - 'type': 'rpm', - 'uri': 'http://10.25.0.10:8080/mos1/x86_64' - }] - }}}} - ) - - def test_apply_for_ubuntu_release(self, accessors): - fuel = accessors.get_fuel_api_accessor() - self._setup_fuel_versions(fuel) - env = self._create_fuel_env(fuel) - release = self._create_fuel_release(fuel, "Ubuntu") - self.start_cmd( - apply, ['--group', 'mos', 'ubuntu', '--default'], - UBUNTU_PATH - ) - accessors.get_fuel_api_accessor.assert_called_with( - "10.25.0.10", "test", "test1" - ) - fuel.FuelVersion.get_all_data.assert_called_once_with() - self.assertEqual(1, env.set_settings_data.call_count) - release.connection.put_request.assert_called_once_with( - release.instance_api_path.format(), - { - 'name': "test release", - 'operating_system': 'Ubuntu', - 'attributes_metadata': { - 'editable': {'repo_setup': {'repos': {'value': [ - { - 'name': 'mos', - 'priority': 1000, - 'suite': 'mos1', - 'section': 'main restricted', - 'type': 'deb', - 'uri': 'http://10.25.0.10:8080/mos' - }, - { - 'name': 'ubuntu', - 'priority': 500, - 'suite': 'trusty', - 'section': 'main multiverse restricted universe', - 'type': 'deb', - 'uri': 'http://10.25.0.10:8080/ubuntu' - } - ]}}} - } - } - ) - - def test_update_release_and_cluster_repositories(self, accessors): - fuel = accessors.get_fuel_api_accessor() - self._setup_fuel_versions(fuel) - env = self._create_fuel_env(fuel, repos=mirror_repos()) - release = \ - self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos()) - self.start_cmd( - apply, ['--group', 'mos', 'ubuntu', '--default'], - UBUNTU_PATH - ) - accessors.get_fuel_api_accessor.assert_called_with( - "10.25.0.10", "test", "test1" - ) - fuel.FuelVersion.get_all_data.assert_called_once_with() - expected_repos = mirror_repos() + local_repos() - env.set_settings_data.assert_called_with( - { - 'editable': { - 'repo_setup': { - 'repos': {'value': expected_repos} - } - } - } - ) - release.connection.put_request.assert_called_once_with( - release.instance_api_path.format(), - { - 'name': "test release", - 'operating_system': 'Ubuntu', - 'attributes_metadata': { - 'editable': {'repo_setup': {'repos': { - 'value': expected_repos - }}} - } - } - ) - - def test_replace_release_and_cluster_repositories(self, accessors): - fuel = accessors.get_fuel_api_accessor() - self._setup_fuel_versions(fuel) - env = self._create_fuel_env(fuel, repos=mirror_repos()) - release = \ - self._create_fuel_release(fuel, "Ubuntu", repos=mirror_repos()) - self.start_cmd( - apply, ['--group', 'mos', 'ubuntu', '--default', '--replace'], - UBUNTU_PATH - ) - accessors.get_fuel_api_accessor.assert_called_with( - "10.25.0.10", "test", "test1" - ) - fuel.FuelVersion.get_all_data.assert_called_once_with() - expected_repos = local_repos(reverse=False) - env.set_settings_data.assert_called_with( - { - 'editable': { - 'repo_setup': { - 'repos': {'value': expected_repos} - } - } - } - ) - release.connection.put_request.assert_called_once_with( - release.instance_api_path.format(), - { - 'name': "test release", - 'operating_system': 'Ubuntu', - 'attributes_metadata': { - 'editable': {'repo_setup': {'repos': { - 'value': expected_repos - }}} - } - } - ) - - def test_apply_for_centos_release(self, accessors): - fuel = accessors.get_fuel_api_accessor() - self._setup_fuel_versions(fuel) - env = self._create_fuel_env(fuel) - release = self._create_fuel_release(fuel, "CentOS") - self.start_cmd( - apply, ['--group', 'mos', 'centos', '--default'], - CENTOS_PATH - ) - accessors.get_fuel_api_accessor.assert_called_with( - "10.25.0.10", "test", "test1" - ) - fuel.FuelVersion.get_all_data.assert_called_once_with() - self.assertEqual(1, env.set_settings_data.call_count) - release.connection.put_request.assert_called_once_with( - release.instance_api_path.format(), - { - 'name': "test release", - 'operating_system': 'CentOS', - 'attributes_metadata': { - 'editable': {'repo_setup': {'repos': {'value': [ - { - 'name': 'centos', - 'priority': 5, - 'type': 'rpm', - 'uri': 'http://10.25.0.10:8080/centos/os/x86_64' - }, - { - 'name': 'mos', - 'priority': 10, - 'type': 'rpm', - 'uri': 'http://10.25.0.10:8080/mos1/x86_64' - }, - ]}}} - } - } - ) - - @mock.patch("fuel_mirror.app.utils.get_fuel_settings") - def test_apply_fail_if_no_fuel_address(self, m_get_settings, accessors): - m_get_settings.return_value = {} - with self.assertRaisesRegexp( - ValueError, "Please specify the fuel-server option"): - apply.debug( - ["--config", CONFIG_PATH, "-G", "mos", "-I", UBUNTU_PATH] - ) - self.assertFalse(accessors.get_fuel_api_accessor.called) - - @mock.patch("fuel_mirror.app.utils.get_fuel_settings") - def test_create_without_fuel_address(self, m_get_settings, accessors): - m_get_settings.return_value = {} - packetary = accessors.get_packetary_accessor() - create.debug( - ["--config", CONFIG_PATH, "-G", "mos", "-I", UBUNTU_PATH] - ) - self.assertFalse(accessors.get_fuel_api_accessor.called) - accessors.get_packetary_accessor.assert_called_with( - threads_num=1, - ignore_errors_num=2, - retries_num=3, - http_proxy="http://localhost", - https_proxy="https://localhost", - ) - packetary.assert_called_with("deb", "x86_64") - api = packetary() - api.clone_repositories.assert_called_once_with( - ['http://localhost/mos mos main restricted'], - '/var/www/', - None, - None - ) - - @mock.patch("fuel_mirror.app.utils.get_fuel_settings") - def test_create_with_invalid_data(self, m_get_settings, accessors): - self.assertRaises( - ValidationError, create.debug, ["--config", CONFIG_PATH, "-G", - "mos", "-I", INVALID_DATA_PATH] - ) diff --git a/contrib/fuel_mirror/fuel_mirror/tests/test_url_builder.py b/contrib/fuel_mirror/fuel_mirror/tests/test_url_builder.py deleted file mode 100644 index aad93b9..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/test_url_builder.py +++ /dev/null @@ -1,70 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from fuel_mirror.common import url_builder -from fuel_mirror.tests import base - - -class TestUrlBuilder(base.TestCase): - def test_get_url_builder(self): - self.assertTrue(issubclass( - url_builder.get_url_builder("deb"), - url_builder.AptRepoUrlBuilder - )) - self.assertTrue(issubclass( - url_builder.get_url_builder("rpm"), - url_builder.YumRepoUrlBuilder - )) - with self.assertRaises(KeyError): - url_builder.get_url_builder("unknown") - - -class TestAptUrlBuilder(base.TestCase): - @classmethod - def setUpClass(cls): - cls.builder = url_builder.get_url_builder("deb") - cls.repo_data = { - "name": "ubuntu", - "suite": "trusty", - "section": "main restricted", - "type": "deb", - "uri": "http://localhost/ubuntu" - } - - def test_get_repo_url(self): - self.assertEqual( - "http://localhost/ubuntu trusty main restricted", - self.builder.get_repo_url(self.repo_data) - ) - - -class TestYumUrlBuilder(base.TestCase): - @classmethod - def setUpClass(cls): - cls.builder = url_builder.get_url_builder("rpm") - cls.repo_data = { - "name": "centos", - "type": "rpm", - "uri": "http://localhost/os/x86_64" - } - - def test_get_repo_url(self): - self.assertEqual( - "http://localhost/os/x86_64", - self.builder.get_repo_url(self.repo_data) - ) diff --git a/contrib/fuel_mirror/fuel_mirror/tests/test_utils.py b/contrib/fuel_mirror/fuel_mirror/tests/test_utils.py deleted file mode 100644 index 1a48ebb..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/test_utils.py +++ /dev/null @@ -1,114 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock -import six - -from fuel_mirror.common import utils -from fuel_mirror.tests import base - - -class DictAsObj(object): - def __init__(self, d): - self.__dict__.update(d) - - def __eq__(self, other): - return self.__dict__ == other.__dict__ - - -class TestUtils(base.TestCase): - def test_lists_merge(self): - main = [{"a": 1, "b": 2, "c": 0}, {"a": 2, "b": 3, "c": 1}] - patch = [{"a": 2, "b": 4}, {"a": 3, "b": 5}] - utils.lists_merge( - main, - patch, - key="a" - ) - self.assertItemsEqual( - [{"a": 1, "b": 2, "c": 0}, - {"a": 2, "b": 4, "c": 1}, - {"a": 3, "b": 5}], - main - ) - - def test_first(self): - self.assertEqual( - 1, - utils.first(0, 1, 0), - ) - self.assertEqual( - 1, - utils.first(None, [], '', 1), - ) - self.assertIsNone( - utils.first(None, [], 0, ''), - ) - self.assertIsNone( - utils.first(), - ) - - def test_is_subdict(self): - self.assertFalse(utils.is_subdict({"c": 1}, {"a": 1, "b": 1})) - self.assertFalse(utils.is_subdict({"a": 1, "b": 2}, {"a": 1, "b": 1})) - self.assertFalse( - utils.is_subdict({"a": 1, "b": 1, "c": 2}, {"a": 1, "b": 1}) - ) - self.assertFalse( - utils.is_subdict({"a": 1, "b": None}, {"a": 1}) - ) - self.assertTrue(utils.is_subdict({}, {"a": 1})) - self.assertTrue(utils.is_subdict({"a": 1}, {"a": 1, "b": 1})) - self.assertTrue(utils.is_subdict({"a": 1, "b": 1}, {"a": 1, "b": 1})) - - @mock.patch("fuel_mirror.common.utils.open") - def test_get_fuel_settings(self, m_open): - m_open().__enter__.side_effect = [ - six.StringIO( - 'ADMIN_NETWORK:\n' - ' ipaddress: "10.20.0.4"\n' - 'FUEL_ACCESS:\n' - ' user: "test"\n' - ' password: "test_pwd"\n', - ), - OSError - ] - - self.assertEqual( - { - "server": "10.20.0.4", - "user": "test", - "password": "test_pwd", - }, - utils.get_fuel_settings() - ) - - self.assertEqual( - {}, - utils.get_fuel_settings() - ) - - @mock.patch("fuel_mirror.common.utils.yaml") - @mock.patch("fuel_mirror.common.utils.open") - def test_load_input_data(self, open_mock, yaml_mock): - data = "$param1: $param2" - open_mock().__enter__().read.return_value = data - v = utils.load_input_data("data.yaml", param1="key", param2="value") - open_mock.assert_called_with("data.yaml", "r") - yaml_mock.load.assert_called_once_with("key: value") - self.assertIs(yaml_mock.load(), v) diff --git a/contrib/fuel_mirror/fuel_mirror/tests/test_validate_config.py b/contrib/fuel_mirror/fuel_mirror/tests/test_validate_config.py deleted file mode 100644 index 6093367..0000000 --- a/contrib/fuel_mirror/fuel_mirror/tests/test_validate_config.py +++ /dev/null @@ -1,158 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import os.path - -from jsonschema import validate -from jsonschema import ValidationError -import yaml - -from fuel_mirror.schemas.input_data_schema import SCHEMA -from fuel_mirror.tests import base - - -DATA_DIR = os.path.join(os.path.dirname(__file__), "..", "..", "data") - - -class TestValidateConfigs(base.TestCase): - - def test_validate_data_files(self): - for f in os.listdir(DATA_DIR): - with open(os.path.join(DATA_DIR, f), "r") as fd: - data = yaml.load(fd) - self.assertNotRaises(ValidationError, validate, data, SCHEMA) - self.assertIn("groups", data) - self.assertIn("fuel_release_match", data) - - def test_validate_fail_with_empty_data(self): - self.assertRaises(ValidationError, validate, {}, SCHEMA) - - def test_validate_fail_without_groups(self): - invalid_data = { - "requirements": { - "ubuntu": ["package_deb"] - } - } - self.assertRaisesRegexp( - ValidationError, "'groups' is a required property", validate, - invalid_data, SCHEMA) - - def test_invalid_requirements_in_pattern_properies(self): - invalid_data = { - "requirements": { - "ubun.tu": ["package_deb"] - }, - "groups": { - } - } - self.assertRaisesRegexp( - ValidationError, "'ubun.tu' was unexpected", validate, - invalid_data, SCHEMA) - - def test_invalid_requirements_type_array(self): - invalid_data = { - "requirements": { - "ubuntu": "package_deb" - }, - "groups": { - } - } - self.assertRaisesRegexp( - ValidationError, "'package_deb' is not of type 'array'", validate, - invalid_data, SCHEMA) - - def test_invalid_inheritens_in_pattern_properies(self): - invalid_data = { - "inheritance": { - "ubun.tu": "mos" - }, - "groups": { - } - } - self.assertRaisesRegexp( - ValidationError, "'ubun.tu' was unexpected", validate, - invalid_data, SCHEMA) - - def test_invalid_inheritens_type_string(self): - invalid_data = { - "inheritance": { - "ubuntu": 123 - }, - "groups": { - } - } - self.assertRaisesRegexp( - ValidationError, "123 is not of type 'string'", validate, - invalid_data, SCHEMA) - - def test_invalid_groups_in_pattern_properies(self): - invalid_data = { - "groups": { - "mo.s": [] - } - } - self.assertRaisesRegexp( - ValidationError, "'mo.s' was unexpected", validate, - invalid_data, SCHEMA) - - def test_invalid_groups_type_array(self): - invalid_data = { - "groups": { - "mos": "string" - } - } - self.assertRaisesRegexp( - ValidationError, "'string' is not of type 'array'", validate, - invalid_data, SCHEMA) - - def test_without_name_in_groups_array(self): - invalid_data = { - "groups": { - "mos": [ - { - 'type': 'deb', - 'uri': 'http://localhost/mos', - 'priority': None, - 'suite': 'mos$mos_version', - 'section': 'main restricted' - } - ] - } - } - self.assertRaisesRegexp( - ValidationError, "is not valid under any of the given schemas", - validate, invalid_data, SCHEMA) - - def test_with_invalid_type_in_groups_array(self): - invalid_data = { - "groups": { - "mos": [ - { - 'name': 'mos', - 'type': 'adf', - 'uri': 'http://localhost/mos', - 'priority': None, - 'suite': 'mos$mos_version', - 'section': 'main restricted' - } - ] - } - } - self.assertRaisesRegexp( - ValidationError, "is not valid under any of the given schemas", - validate, invalid_data, SCHEMA) diff --git a/contrib/fuel_mirror/requirements.txt b/contrib/fuel_mirror/requirements.txt deleted file mode 100644 index a69fea2..0000000 --- a/contrib/fuel_mirror/requirements.txt +++ /dev/null @@ -1,12 +0,0 @@ -# The order of packages is significant, because pip processes them in the order -# of appearance. Changing the order has an impact on the overall integration -# process, which may cause wedges in the gate later. - -pbr>=0.8 -Babel>=1.3 -cliff>=1.7.0 -six>=1.5.2 -PyYAML>=3.10 -packetary>=0.1.0 -python-fuelclient>=7.0.0 -jsonschema>=2.3.0 diff --git a/contrib/fuel_mirror/scripts/fuel-createmirror b/contrib/fuel_mirror/scripts/fuel-createmirror deleted file mode 100755 index bc68dbb..0000000 --- a/contrib/fuel_mirror/scripts/fuel-createmirror +++ /dev/null @@ -1,80 +0,0 @@ -#!/bin/bash - -echo "This script is DEPRECATED. Please use fuel-mirror utility!" - -# This shell script was wraps the fuel-mirror utility to provide backward compatibility -# with previous version of tool. - -usage() { -cat <=0.10.0 - -coverage>=3.6 -discover -python-subunit>=0.0.18 -sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2 -oslosphinx>=2.5.0 # Apache-2.0 -oslotest>=1.10.0 # Apache-2.0 -testrepository>=0.0.18 -testscenarios>=0.4 -testtools>=1.4.0 -cliff>=1.7.0 -six>=1.5.2 diff --git a/contrib/fuel_mirror/tox.ini b/contrib/fuel_mirror/tox.ini deleted file mode 100644 index 0a57b39..0000000 --- a/contrib/fuel_mirror/tox.ini +++ /dev/null @@ -1,35 +0,0 @@ -[tox] -minversion = 1.6 -envlist = py34,py27,py26,pep8 -skipsdist = True - -[testenv] -usedevelop = True -install_command = pip install -U {opts} {packages} -setenv = - VIRTUAL_ENV={envdir} -deps = -r{toxinidir}/test-requirements.txt -commands = python setup.py test --slowest --testr-args='{posargs:fuel_mirror}' - -[testenv:pep8] -commands = flake8 - -[testenv:venv] -commands = {posargs} - -[testenv:cover] -commands = python setup.py test --coverage --testr-args='{posargs:fuel_mirror}' - -[testenv:docs] -commands = python setup.py build_sphinx - -[testenv:debug] -commands = oslo_debug_helper {posargs} - -[flake8] -# E123, E125 skipped as they are invalid PEP-8. - -show-source = True -ignore = E123,E125 -builtins = _ -exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build diff --git a/debian/changelog b/debian/changelog deleted file mode 100644 index fb1f193..0000000 --- a/debian/changelog +++ /dev/null @@ -1,17 +0,0 @@ -fuel-mirror (10.0.0-1) experimental; urgency=medium - - * Bump version to 10.0 - - -- Sergey Kulanov Mon, 21 Mar 2016 13:29:18 +0200 - -fuel-mirror (9.0.0-1) experimental; urgency=low - - * Bump version to 9.0 - - -- Sergey Kulanov Thu, 17 Dec 2015 16:39:55 +0200 - -fuel-mirror (8.0.0-1) experimental; urgency=low - - * Initial release. - - -- bgaifullin Fri, 27 Nov 2015 00:28:26 +0300 diff --git a/debian/compat b/debian/compat deleted file mode 100644 index ec63514..0000000 --- a/debian/compat +++ /dev/null @@ -1 +0,0 @@ -9 diff --git a/debian/control b/debian/control deleted file mode 100644 index 4598767..0000000 --- a/debian/control +++ /dev/null @@ -1,54 +0,0 @@ -Source: fuel-mirror -Section: Utilities -Priority: extra -Maintainer: Mirantis Product -Build-Depends: debhelper (>= 9), - dh-python, - openstack-pkg-tools (>= 23~), - python-all, - python-pbr (>= 0.8), - python-setuptools -Standards-Version: 3.9.6 -Homepage: mirantis.com - -Package: fuel-mirror -Architecture: all -Section: python -Depends: python-babel, - python-cliff (>= 1.7.0), - python-packetary (= ${binary:Version}), - python-pbr (>= 0.8), - python-six, - python-yaml, - python-tz, - ${python:Depends} -Recommends: python-fuelclient (>= 7.0.0) -Description: Utility to create RPM and DEB mirror - Provides two commands fuel-mirror and fuel-createmirror. - Second one is for backward compatibility with the previous - generation of the utility. These commands could be used - to create local copies of MOS and upstream deb and rpm - repositories. - -Package: python-packetary -Architecture: all -Depends: createrepo, - python-babel, - python-bintrees (>= 2.0.2), - python-chardet, - python-cliff (>= 1.7.0), - python-debian (>= 0.1.21), - python-eventlet (>= 0.15), - python-lxml, - python-pbr (>= 0.8), - python-six, - python-stevedore (>= 1.1.0), - python-tz, - ${python:Depends} -Description: Library allows to build and clone deb and rpm repos - Provides object model and API for dealing with deb - and rpm repositories. One can use this framework to - implement operations like building repository - from a set of packages, clone repository, find package - dependencies, mix repositories, pull out a subset of - packages into a separate repository, etc. diff --git a/debian/copyright b/debian/copyright deleted file mode 100644 index 2fa4ac7..0000000 --- a/debian/copyright +++ /dev/null @@ -1,43 +0,0 @@ -Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ -Upstream-Name: fuel-mirror -Source: git://github.com/openstack/fuel-mirror.git - -Files: debian/* -Copyright: (c) 2014, Mirantis -License: GPL-2 - -Files: * -Copyright: (c) 2014, Mirantis -License: Apache-2 - -License: Apache-2 - Licensed under the Apache License, Version 2.0 (the "License"); you may not - use this file except in compliance with the License. You may obtain a copy of - the License at: - . - http://www.apache.org/licenses/LICENSE-2.0 - . - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - License for the specific language governing permissions and limitations under - the License. - . - On Debian-based systems the full text of the Apache version 2.0 license can be - found in /usr/share/common-licenses/Apache-2.0. - -License: GPL-2 - Licensed under the GPL License, Version 2.0 (the "License"); you may not - use this file except in compliance with the License. You may obtain a copy of - the License at: - . - http://www.opensource.org/licenses/GPL-2.0 - . - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - License for the specific language governing permissions and limitations under - the License. - . - On Debian-based systems the full text of the GPL version 2.0 license can be - found in /usr/share/common-licenses/GPL-2. diff --git a/debian/fuel-mirror.install b/debian/fuel-mirror.install deleted file mode 100644 index e8eabae..0000000 --- a/debian/fuel-mirror.install +++ /dev/null @@ -1,2 +0,0 @@ -contrib/fuel_mirror/etc/config.yaml /etc/fuel-mirror -contrib/fuel_mirror/scripts/fuel-createmirror /usr/bin/ diff --git a/debian/rules b/debian/rules deleted file mode 100755 index 21c7f69..0000000 --- a/debian/rules +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/make -f - -PYTHONS:=$(shell pyversions -vr) - -include /usr/share/openstack-pkg-tools/pkgos.make -export OSLO_PACKAGE_VERSION=$(shell dpkg-parsechangelog | grep Version: | cut -d' ' -f2 | sed -e 's/^[[:digit:]]*://' -e 's/[-].*//' -e 's/~/.0/' | head -n 1) - -%: - dh $@ --buildsystem=python_distutils --with python2 - -override_dh_clean: - rm -rf build - dh_clean -O--buildsystem=python_distutils - - -override_dh_auto_install: - set -e ; for pyvers in $(PYTHONS); do \ - python$$pyvers setup.py install --install-layout=deb \ - --root $(CURDIR)/debian/python-packetary; \ - done - set -e ; cd contrib/fuel_mirror/; \ - for pyvers in $(PYTHONS); do \ - python$$pyvers ./setup.py install --install-layout=deb \ - --root $(CURDIR)/debian/fuel-mirror; \ - done - -override_dh_fixperms: - set -e; chmod 755 $(CURDIR)/debian/fuel-mirror/usr/bin/fuel-createmirror - -override_dh_python2: - dh_python2 --no-guessing-deps - -override_dh_installcatalogs: -override_dh_installemacsen override_dh_installifupdown: -override_dh_installinfo override_dh_installmenu override_dh_installmime: -override_dh_installmodules override_dh_installlogcheck: -override_dh_installpam override_dh_installppp override_dh_installudev override_dh_installwm: -override_dh_installxfonts override_dh_gconf override_dh_icons override_dh_perl override_dh_usrlocal: -override_dh_installgsettings: \ No newline at end of file diff --git a/debian/source/format b/debian/source/format deleted file mode 100644 index 46ebe02..0000000 --- a/debian/source/format +++ /dev/null @@ -1 +0,0 @@ -3.0 (quilt) \ No newline at end of file diff --git a/doc/__init__.py b/doc/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/doc/source/__init__.py b/doc/source/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/doc/source/conf.py b/doc/source/conf.py deleted file mode 100755 index 12fc8a7..0000000 --- a/doc/source/conf.py +++ /dev/null @@ -1,79 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import os -import sys - -sys.path.insert(0, os.path.abspath('../..')) -# -- General configuration ---------------------------------------------------- - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = [ - 'sphinx.ext.autodoc', - #'sphinx.ext.intersphinx', - 'oslosphinx' -] - -# autodoc generation is a bit aggressive and a nuisance when doing heavy -# text edit cycles. -# execute "export SPHINX_DEBUG=1" in your terminal to disable - -# The suffix of source filenames. -source_suffix = '.rst' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'packetary' -copyright = u'2015, Mirantis Inc.' - -# If true, '()' will be appended to :func: etc. cross-reference text. -add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -add_module_names = True - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# -- Options for HTML output -------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. Major themes that come with -# Sphinx are currently 'default' and 'sphinxdoc'. -# html_theme_path = ["."] -# html_theme = '_theme' -# html_static_path = ['static'] - -# Output file base name for HTML help builder. -htmlhelp_basename = '%sdoc' % project - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, author, documentclass -# [howto/manual]). -latex_documents = [ - ('index', - '%s.tex' % project, - u'%s Documentation' % project, - u'OpenStack Foundation', 'manual'), -] - -# Example configuration for intersphinx: refer to the Python standard library. -#intersphinx_mapping = {'http://docs.python.org/': None} diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst deleted file mode 100644 index 1728a61..0000000 --- a/doc/source/contributing.rst +++ /dev/null @@ -1,4 +0,0 @@ -============ -Contributing -============ -.. include:: ../../CONTRIBUTING.rst diff --git a/doc/source/index.rst b/doc/source/index.rst deleted file mode 100644 index 46d6225..0000000 --- a/doc/source/index.rst +++ /dev/null @@ -1,25 +0,0 @@ -.. packetary documentation master file, created by - sphinx-quickstart on Tue Jul 9 22:26:36 2013. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to packetary's documentation! -======================================================== - -Contents: - -.. toctree:: - :maxdepth: 2 - - readme - installation - usage - contributing - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - diff --git a/doc/source/installation.rst b/doc/source/installation.rst deleted file mode 100644 index 532b3f1..0000000 --- a/doc/source/installation.rst +++ /dev/null @@ -1,12 +0,0 @@ -============ -Installation -============ - -At the command line:: - - $ pip install packetary - -Or, if you have virtualenvwrapper installed:: - - $ mkvirtualenv packetary - $ pip install packetary diff --git a/doc/source/readme.rst b/doc/source/readme.rst deleted file mode 100644 index a6210d3..0000000 --- a/doc/source/readme.rst +++ /dev/null @@ -1 +0,0 @@ -.. include:: ../../README.rst diff --git a/doc/source/usage.rst b/doc/source/usage.rst deleted file mode 100644 index c061172..0000000 --- a/doc/source/usage.rst +++ /dev/null @@ -1,7 +0,0 @@ -======== -Usage -======== - -To use packetary in a project:: - - import packetary diff --git a/openstack-common.conf b/openstack-common.conf deleted file mode 100644 index c0bb65e..0000000 --- a/openstack-common.conf +++ /dev/null @@ -1,6 +0,0 @@ -[DEFAULT] - -# The list of modules to copy from oslo-incubator.git - -# The base module to hold the copy of openstack.common -base=packetary diff --git a/packetary/__init__.py b/packetary/__init__.py deleted file mode 100644 index 9ee18e2..0000000 --- a/packetary/__init__.py +++ /dev/null @@ -1,33 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import pbr.version - -from packetary.api import Configuration -from packetary.api import Context -from packetary.api import RepositoryApi - - -__all__ = [ - "Configuration", - "Context", - "RepositoryApi", -] - -__version__ = pbr.version.VersionInfo( - 'packetary').version_string() diff --git a/packetary/api.py b/packetary/api.py deleted file mode 100644 index dd7d98d..0000000 --- a/packetary/api.py +++ /dev/null @@ -1,233 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import logging - -import six - -from packetary.controllers import RepositoryController -from packetary.library.connections import ConnectionsManager -from packetary.library.executor import AsynchronousSection -from packetary.objects import Index -from packetary.objects import PackageRelation -from packetary.objects import PackagesTree -from packetary.objects.statistics import CopyStatistics - - -logger = logging.getLogger(__package__) - - -class Configuration(object): - """The configuration holder.""" - - def __init__(self, http_proxy=None, https_proxy=None, - retries_num=0, threads_num=0, - ignore_errors_num=0, retry_interval=0): - """Initialises. - - :param http_proxy: the url of proxy for connections over http, - no-proxy will be used if it is not specified - :param https_proxy: the url of proxy for connections over https, - no-proxy will be used if it is not specified - :param retries_num: the number of retries on errors - :param retry_interval: the time between retries (in seconds) - :param threads_num: the max number of active threads - :param ignore_errors_num: the number of errors that may occurs - before stop processing - """ - - self.http_proxy = http_proxy - self.https_proxy = https_proxy - self.ignore_errors_num = ignore_errors_num - self.retries_num = retries_num - self.retry_interval = retry_interval - self.threads_num = threads_num - - -class Context(object): - """The infra-objects holder.""" - - def __init__(self, config): - """Initialises. - - :param config: the configuration - """ - self._connection = ConnectionsManager( - proxy=config.http_proxy, - secure_proxy=config.https_proxy, - retries_num=config.retries_num, - retry_interval=config.retry_interval - ) - self._threads_num = config.threads_num - self._ignore_errors_num = config.ignore_errors_num - - @property - def connection(self): - """Gets the connection.""" - return self._connection - - def async_section(self, ignore_errors_num=None): - """Gets the execution scope. - - :param ignore_errors_num: custom value for ignore_errors_num, - the class value is used if omitted. - """ - if ignore_errors_num is None: - ignore_errors_num = self._ignore_errors_num - - return AsynchronousSection(self._threads_num, ignore_errors_num) - - -class RepositoryApi(object): - """Provides high-level API to operate with repositories.""" - - def __init__(self, controller): - """Initialises. - - :param controller: the repository controller. - """ - self.controller = controller - - @classmethod - def create(cls, config, repotype, repoarch): - """Creates the repository API instance. - - :param config: the configuration - :param repotype: the kind of repository(deb, yum, etc) - :param repoarch: the architecture of repository (x86_64 or i386) - """ - context = config if isinstance(config, Context) else Context(config) - return cls(RepositoryController.load(context, repotype, repoarch)) - - def get_packages(self, origin, debs=None, requirements=None): - """Gets the list of packages from repository(es). - - :param origin: The list of repository`s URLs - :param debs: the list of repository`s URL to calculate list of - dependencies, that will be used to filter packages. - :param requirements: the list of package relations, - to resolve the list of mandatory packages. - :return: the set of packages - """ - repositories = self._get_repositories(origin) - return self._get_packages(repositories, debs, requirements) - - def clone_repositories(self, origin, destination, debs=None, - requirements=None, keep_existing=True, - include_source=False, include_locale=False): - """Creates the clones of specified repositories in local folder. - - :param origin: The list of repository`s URLs - :param destination: the destination folder path - :param debs: the list of repository`s URL to calculate list of - dependencies, that will be used to filter packages. - :param requirements: the list of package relations, - to resolve the list of mandatory packages. - :param keep_existing: If False - local packages that does not exist - in original repo will be removed. - :param include_source: if True, the source packages - will be copied as well. - :param include_locale: if True, the locales - will be copied as well. - :return: count of copied and total packages. - """ - repositories = self._get_repositories(origin) - packages = self._get_packages(repositories, debs, requirements) - mirrors = self.controller.clone_repositories( - repositories, destination, include_source, include_locale - ) - - package_groups = dict((x, set()) for x in repositories) - for pkg in packages: - package_groups[pkg.repository].add(pkg) - - stat = CopyStatistics() - for repo, packages in six.iteritems(package_groups): - mirror = mirrors[repo] - logger.info("copy packages from - %s", repo) - self.controller.copy_packages( - mirror, packages, keep_existing, stat.on_package_copied - ) - return stat - - def get_unresolved_dependencies(self, origin, main=None): - """Gets list of unresolved dependencies for repository(es). - - :param origin: The list of repository`s URLs - :param main: The main repository(es) URL - :return: list of unresolved dependencies - """ - packages = PackagesTree() - self.controller.load_packages( - self._get_repositories(origin), - packages.add - ) - - if main is not None: - base = Index() - self.controller.load_packages( - self._get_repositories(main), - base.add - ) - else: - base = None - - return packages.get_unresolved_dependencies(base) - - def _get_repositories(self, urls): - """Gets the set of repositories by url.""" - repositories = set() - self.controller.load_repositories(urls, repositories.add) - return repositories - - def _get_packages(self, repositories, master, requirements): - """Gets the list of packages according to master and requirements.""" - if master is None and requirements is None: - packages = set() - self.controller.load_packages(repositories, packages.add) - return packages - - packages = PackagesTree() - self.controller.load_packages(repositories, packages.add) - if master is not None: - main_index = Index() - self.controller.load_packages( - self._get_repositories(master), - main_index.add - ) - else: - main_index = None - - return packages.get_minimal_subset( - main_index, - self._parse_requirements(requirements) - ) - - @staticmethod - def _parse_requirements(requirements): - """Gets the list of relations from requirements. - - :param requirements: the list of requirement in next format: - 'name [cmp version]|[alt [cmp version]]' - """ - if requirements is not None: - return set( - PackageRelation.from_args( - *(x.split() for x in r.split("|"))) for r in requirements - ) - return set() diff --git a/packetary/cli/__init__.py b/packetary/cli/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/packetary/cli/app.py b/packetary/cli/app.py deleted file mode 100644 index 341f6ff..0000000 --- a/packetary/cli/app.py +++ /dev/null @@ -1,104 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from cliff import app -from cliff.commandmanager import CommandManager - -import packetary - - -class Application(app.App): - """Main cliff application class. - - Performs initialization of the command manager and - configuration of basic engines. - """ - - def build_option_parser(self, description, version, argparse_kwargs=None): - """Specifies global options.""" - p_inst = super(Application, self) - parser = p_inst.build_option_parser(description=description, - version=version, - argparse_kwargs=argparse_kwargs) - - parser.add_argument( - "--ignore-errors-num", - type=int, - default=2, - metavar="NUMBER", - help="The number of errors that can be ignored." - ) - parser.add_argument( - "--retries-num", - type=int, - default=5, - metavar="NUMBER", - help="The number of retries." - ) - parser.add_argument( - "--retry-interval", - type=int, - default=2, - metavar="SECONDS", - help="The minimal time between retries in seconds." - ) - parser.add_argument( - "--threads-num", - default=3, - type=int, - metavar="NUMBER", - help="The number of threads." - ) - parser.add_argument( - "--http-proxy", - default=None, - metavar="http://username:password@proxy_host:proxy_port", - help="The URL of http proxy." - ) - parser.add_argument( - "--https-proxy", - default=None, - metavar="https://username:password@proxy_host:proxy_port", - help="The URL of https proxy." - ) - return parser - - -def main(argv=None): - return Application( - description="The utility manages packages and repositories.", - version=packetary.__version__, - command_manager=CommandManager("packetary", convert_underscores=True) - ).run(argv) - - -def debug(name, cmd_class, argv=None): - """Helper for debugging single command without package installation.""" - import sys - - if argv is None: - argv = sys.argv[1:] - - argv = [name] + argv + ["-v", "-v", "--debug"] - cmd_mgr = CommandManager("test_packetary", convert_underscores=True) - cmd_mgr.add_command(name, cmd_class) - return Application( - description="The utility manages packages and repositories.", - version="0.0.1", - command_manager=cmd_mgr - ).run(argv) diff --git a/packetary/cli/commands/__init__.py b/packetary/cli/commands/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/packetary/cli/commands/base.py b/packetary/cli/commands/base.py deleted file mode 100644 index bb672cf..0000000 --- a/packetary/cli/commands/base.py +++ /dev/null @@ -1,185 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import abc - -from cliff import command -import six - -from packetary.cli.commands.utils import make_display_attr_getter -from packetary.cli.commands.utils import read_lines_from_file -from packetary import RepositoryApi - - -@six.add_metaclass(abc.ABCMeta) -class BaseRepoCommand(command.Command): - """Super class for packetary commands.""" - - @property - def stdout(self): - """Shortcut for self.app.stdout.""" - return self.app.stdout - - def get_parser(self, prog_name): - """Specifies common options.""" - parser = super(BaseRepoCommand, self).get_parser(prog_name) - parser.add_argument( - '-t', - '--type', - type=str, - choices=['deb', 'rpm'], - metavar='TYPE', - default='deb', - help='The type of repository.') - - parser.add_argument( - '-a', - '--arch', - type=str, - choices=["x86_64", "i386"], - metavar='ARCHITECTURE', - default="x86_64", - help='The target architecture.') - - origin_gr = parser.add_mutually_exclusive_group(required=True) - origin_gr.add_argument( - '-o', '--origin-url', - nargs="+", - dest='origins', - type=six.text_type, - metavar='URL', - help='Space separated list of URLs of origin repositories.') - - origin_gr.add_argument( - '-O', '--origin-file', - type=read_lines_from_file, - dest='origins', - metavar='FILENAME', - help='The path to file with URLs of origin repositories.') - - return parser - - def take_action(self, parsed_args): - """See the Command.take_action. - - :param parsed_args: the command-line arguments - :return: the result of take_repo_action - :rtype: object - """ - return self.take_repo_action( - RepositoryApi.create( - self.app_args, parsed_args.type, parsed_args.arch - ), - parsed_args - ) - - @abc.abstractmethod - def take_repo_action(self, api, parsed_args): - """Takes action on repository. - - :param api: the RepositoryApi instance - :param parsed_args: the command-line arguments - :return: the action result - """ - - -class BaseProduceOutputCommand(BaseRepoCommand): - columns = None - - def get_parser(self, prog_name): - parser = super(BaseProduceOutputCommand, self).get_parser(prog_name) - - group = parser.add_argument_group( - title='output formatter', - description='output formatter options', - ) - group.add_argument( - '-c', '--column', - nargs='+', - choices=self.columns, - dest='columns', - metavar='COLUMN', - default=[], - help='Space separated list of columns to include.', - ) - group.add_argument( - '-s', - '--sort-columns', - type=str, - nargs='+', - choices=self.columns, - metavar='SORT_COLUMN', - default=[self.columns[0]], - help='Space separated list of keys for sorting ' - 'the data.' - ) - group.add_argument( - '--sep', - type=six.text_type, - metavar='ROW SEPARATOR', - default=six.text_type('; '), - help='The row separator.' - ) - - return parser - - def produce_output(self, parsed_args, data): - indexes = dict( - (c, i) for i, c in enumerate(self.columns) - ) - sort_index = [indexes[c] for c in parsed_args.sort_columns] - if isinstance(data, list): - data.sort(key=lambda x: [x[i] for i in sort_index]) - else: - data = sorted(data, key=lambda x: [x[i] for i in sort_index]) - - if parsed_args.columns: - include_index = [ - indexes[c] for c in parsed_args.columns - ] - data = ((row[i] for i in include_index) for row in data) - columns = parsed_args.columns - else: - columns = self.columns - - stdout = self.stdout - sep = parsed_args.sep - - # header - stdout.write("# ") - stdout.write(sep.join(columns)) - stdout.write("\n") - - for row in data: - stdout.write(sep.join(row)) - stdout.write("\n") - - def run(self, parsed_args): - # Use custom output producer. - # cliff.lister with default formatters does not work - # with large arrays of data, because it does not support streaming - # TODO(implement custom formatter) - - formatter = make_display_attr_getter(self.columns) - data = six.moves.map(formatter, self.take_action(parsed_args)) - self.produce_output(parsed_args, data) - return 0 - - @abc.abstractmethod - def take_repo_action(self, driver, parsed_args): - """See Command.take_repo_action.""" diff --git a/packetary/cli/commands/clone.py b/packetary/cli/commands/clone.py deleted file mode 100644 index fb9c3dc..0000000 --- a/packetary/cli/commands/clone.py +++ /dev/null @@ -1,115 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from packetary.cli.commands.base import BaseRepoCommand -from packetary.cli.commands.utils import read_lines_from_file - - -class CloneCommand(BaseRepoCommand): - """Clones the specified repository to local folder.""" - - def get_parser(self, prog_name): - parser = super(CloneCommand, self).get_parser(prog_name) - - parser.add_argument( - "-d", "--destination", - required=True, - help="The path to the destination folder." - ) - parser.add_argument( - "--clean", - dest="keep_existing", - action='store_false', - default=True, - help="Remove packages that does not exist in origin repo." - ) - - parser.add_argument( - "--sources", - action='store_true', - default=False, - help="Also copy source packages." - ) - - parser.add_argument( - "--locales", - action='store_true', - default=False, - help="Also copy localisation files." - ) - - bootstrap_group = parser.add_mutually_exclusive_group(required=False) - bootstrap_group.add_argument( - "-b", "--bootstrap", - nargs='+', - dest='bootstrap', - metavar='PACKAGE [OP VERSION]', - help="Space separated list of package relations, " - "to resolve the list of mandatory packages." - ) - bootstrap_group.add_argument( - "-B", "--bootstrap-file", - type=read_lines_from_file, - dest='bootstrap', - metavar='FILENAME', - help="Path to the file with list of package relations, " - "to resolve the list of mandatory packages." - ) - - requires_group = parser.add_mutually_exclusive_group(required=False) - requires_group.add_argument( - '-r', '--requires-url', - nargs="+", - dest='requires', - metavar='URL', - help="Space separated list of repository`s URL to calculate list " - "of dependencies, that will be used to filter packages") - - requires_group.add_argument( - '-R', '--requires-file', - type=read_lines_from_file, - dest='requires', - metavar='FILENAME', - help="The path to the file with list of repository`s URL " - "to calculate list of dependencies, " - "that will be used to filter packages") - return parser - - def take_repo_action(self, api, parsed_args): - stat = api.clone_repositories( - parsed_args.origins, - parsed_args.destination, - parsed_args.requires, - parsed_args.bootstrap, - parsed_args.keep_existing, - parsed_args.sources, - parsed_args.locales - ) - self.stdout.write( - "Packages copied: {0.copied}/{0.total}.\n".format(stat) - ) - - -def debug(argv=None): - """Helper to debug the Clone command.""" - from packetary.cli.app import debug - debug("clone", CloneCommand, argv) - - -if __name__ == "__main__": - debug() diff --git a/packetary/cli/commands/packages.py b/packetary/cli/commands/packages.py deleted file mode 100644 index 7f02bd7..0000000 --- a/packetary/cli/commands/packages.py +++ /dev/null @@ -1,93 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from packetary.cli.commands.base import BaseProduceOutputCommand -from packetary.cli.commands.utils import read_lines_from_file - - -class ListOfPackages(BaseProduceOutputCommand): - """Gets the list of packages from repository(es).""" - - columns = ( - "name", - "repository", - "version", - "filename", - "filesize", - "checksum", - "obsoletes", - "provides", - "requires", - ) - - def get_parser(self, prog_name): - parser = super(ListOfPackages, self).get_parser(prog_name) - - bootstrap_group = parser.add_mutually_exclusive_group(required=False) - bootstrap_group.add_argument( - "-b", "--bootstrap", - nargs='+', - dest='bootstrap', - metavar='PACKAGE [OP VERSION]', - help="Space separated list of package relations, " - "to resolve the list of mandatory packages." - ) - bootstrap_group.add_argument( - "-B", "--bootstrap-file", - type=read_lines_from_file, - dest='bootstrap', - metavar='FILENAME', - help="Path to the file with list of package relations, " - "to resolve the list of mandatory packages." - ) - - requires_group = parser.add_mutually_exclusive_group(required=False) - requires_group.add_argument( - '-r', '--requires-url', - nargs="+", - dest='requires', - metavar='URL', - help="Space separated list of repository`s URL to calculate list " - "of dependencies, that will be used to filter packages") - - requires_group.add_argument( - '-R', '--requires-file', - type=read_lines_from_file, - dest='requires', - metavar='FILENAME', - help="The path to the file with list of repository`s URL " - "to calculate list of dependencies, " - "that will be used to filter packages") - return parser - - def take_repo_action(self, api, parsed_args): - return api.get_packages( - parsed_args.origins, - parsed_args.requires, - parsed_args.bootstrap, - ) - - -def debug(argv=None): - """Helper to debug the ListOfPackages command.""" - from packetary.cli.app import debug - debug("packages", ListOfPackages, argv) - - -if __name__ == "__main__": - debug() diff --git a/packetary/cli/commands/unresolved.py b/packetary/cli/commands/unresolved.py deleted file mode 100644 index a4ce6f3..0000000 --- a/packetary/cli/commands/unresolved.py +++ /dev/null @@ -1,68 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from packetary.cli.commands.base import BaseProduceOutputCommand -from packetary.cli.commands.utils import read_lines_from_file - - -class ListOfUnresolved(BaseProduceOutputCommand): - """Gets the list of external dependencies for repository(es).""" - - columns = ( - "name", - "version", - "alternative", - ) - - def get_parser(self, prog_name): - parser = super(ListOfUnresolved, self).get_parser(prog_name) - main_group = parser.add_mutually_exclusive_group(required=False) - main_group.add_argument( - '-m', '--main-url', - nargs="+", - dest='main', - metavar='URL', - help='Space separated list of URLs of repository(es) ' - ' that are used to resolve dependencies.') - - main_group.add_argument( - '-M', '--main-file', - type=read_lines_from_file, - dest='main', - metavar='FILENAME', - help='The path to the file, that contains ' - 'list of URLs of repository(es) ' - ' that are used to resolve dependencies.') - return parser - - def take_repo_action(self, api, parsed_args): - return api.get_unresolved_dependencies( - parsed_args.origins, - parsed_args.main, - ) - - -def debug(argv=None): - """Helper to debug the ListOfUnresolved command.""" - - from packetary.cli.app import debug - debug("unresolved", ListOfUnresolved, argv) - - -if __name__ == "__main__": - debug() diff --git a/packetary/cli/commands/utils.py b/packetary/cli/commands/utils.py deleted file mode 100644 index de89158..0000000 --- a/packetary/cli/commands/utils.py +++ /dev/null @@ -1,72 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import operator - -import six - - -def read_lines_from_file(filename): - """Reads lines from file. - - Note: the line starts with '#' will be skipped. - - :param filename: the path of target file - :return: the list of lines from file - """ - with open(filename, 'r') as f: - return [ - x - for x in six.moves.map(operator.methodcaller("strip"), f) - if x and not x.startswith("#") - ] - - -def get_object_attrs(obj, attrs): - """Gets object attributes as list. - - :param obj: the target object - :param attrs: the list of attributes - :return: list of values from specified attributes. - """ - return [getattr(obj, f) for f in attrs] - - -def get_display_value(value): - """Get the displayable string for value. - - :param value: the target value - :return: the displayable string for value - """ - if value is None: - return u"-" - - if isinstance(value, list): - return u", ".join(six.text_type(x) for x in value) - return six.text_type(value) - - -def make_display_attr_getter(attrs): - """Gets formatter to convert attributes of object in displayable format. - - :param attrs: the list of attributes - :return: the formatter (callable object) - """ - return lambda x: [ - get_display_value(v) for v in get_object_attrs(x, attrs) - ] diff --git a/packetary/controllers/__init__.py b/packetary/controllers/__init__.py deleted file mode 100644 index a34146c..0000000 --- a/packetary/controllers/__init__.py +++ /dev/null @@ -1,23 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from packetary.controllers.repository import RepositoryController - -__all__ = [ - "RepositoryController" -] diff --git a/packetary/controllers/repository.py b/packetary/controllers/repository.py deleted file mode 100644 index da67e2a..0000000 --- a/packetary/controllers/repository.py +++ /dev/null @@ -1,171 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import logging -import os - -import six -import stevedore - - -logger = logging.getLogger(__package__) - -urljoin = six.moves.urllib.parse.urljoin - - -class RepositoryController(object): - """Implements low-level functionality to communicate with drivers.""" - - _drivers = None - - def __init__(self, context, driver, arch): - self.context = context - self.driver = driver - self.arch = arch - - @classmethod - def load(cls, context, driver_name, repoarch): - """Creates the repository manager. - - :param context: the context - :param driver_name: the name of required driver - :param repoarch: the architecture of repository (x86_64 or i386) - """ - if cls._drivers is None: - cls._drivers = stevedore.ExtensionManager( - "packetary.drivers", invoke_on_load=True - ) - try: - driver = cls._drivers[driver_name].obj - except KeyError: - raise NotImplementedError( - "The driver {0} is not supported yet.".format(driver_name) - ) - return cls(context, driver, repoarch) - - def load_repositories(self, urls, consumer): - """Loads the repository objects from url. - - :param urls: the list of repository urls. - :param consumer: the callback to consume objects - """ - if isinstance(urls, six.string_types): - urls = [urls] - - connection = self.context.connection - for parsed_url in self.driver.parse_urls(urls): - self.driver.get_repository( - connection, parsed_url, self.arch, consumer - ) - - def load_packages(self, repositories, consumer): - """Loads packages from repository. - - :param repositories: the repository object - :param consumer: the callback to consume objects - """ - connection = self.context.connection - for r in repositories: - self.driver.get_packages(connection, r, consumer) - - def assign_packages(self, repository, packages, keep_existing=True): - """Assigns new packages to the repository. - - It replaces the current repository`s packages. - - :param repository: the target repository - :param packages: the set of new packages - :param keep_existing: - if True, all existing packages will be kept as is. - if False, all existing packages, that are not included - to new packages will be removed. - """ - - if not isinstance(packages, set): - packages = set(packages) - else: - packages = packages.copy() - - if keep_existing: - consume_exist = packages.add - else: - def consume_exist(package): - if package not in packages: - filepath = os.path.join( - package.repository.url, package.filename - ) - logger.info("remove package - %s.", filepath) - os.remove(filepath) - - self.driver.get_packages( - self.context.connection, repository, consume_exist - ) - self.driver.rebuild_repository(repository, packages) - - def copy_packages(self, repository, packages, keep_existing, observer): - """Copies packages to repository. - - :param repository: the target repository - :param packages: the set of packages - :param keep_existing: see assign_packages for more details - :param observer: the package copying process observer - """ - with self.context.async_section() as section: - for package in packages: - section.execute( - self._copy_package, repository, package, observer - ) - self.assign_packages(repository, packages, keep_existing) - - def clone_repositories(self, repositories, destination, - source=False, locale=False): - """Creates copy of repositories. - - :param repositories: the origin repositories - :param destination: the target folder - :param source: If True, the source packages will be copied too. - :param locale: If True, the localisation will be copied too. - :return: the mapping origin to cloned repository. - """ - mirros = dict() - destination = os.path.abspath(destination) - with self.context.async_section(0) as section: - for r in repositories: - section.execute( - self._fork_repository, - r, destination, source, locale, mirros - ) - return mirros - - def _fork_repository(self, r, destination, source, locale, mirrors): - """Creates clone of repository and stores it in mirrors.""" - new_repository = self.driver.fork_repository( - self.context.connection, r, destination, source, locale - ) - mirrors[r] = new_repository - - def _copy_package(self, target, package, observer): - """Synchronises remote file to local fs.""" - dst_path = os.path.join(target.url, package.filename) - src_path = urljoin(package.repository.url, package.filename) - bytes_copied = self.context.connection.retrieve( - src_path, dst_path, size=package.filesize - ) - if package.filesize < 0: - package.filesize = bytes_copied - observer(bytes_copied) diff --git a/packetary/drivers/__init__.py b/packetary/drivers/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/packetary/drivers/base.py b/packetary/drivers/base.py deleted file mode 100644 index 3175acb..0000000 --- a/packetary/drivers/base.py +++ /dev/null @@ -1,82 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import abc -import logging - -import six - - -@six.add_metaclass(abc.ABCMeta) -class RepositoryDriverBase(object): - """The super class for Repository Drivers. - - For implementing support of new type of repository: - - inherit this class - - implement all abstract methods - - register implementation in 'packetary.drivers' namespace - """ - def __init__(self): - self.logger = logging.getLogger(__package__) - - @abc.abstractmethod - def parse_urls(self, urls): - """Parses the repository url. - - :return: the sequence of parsed urls - """ - - @abc.abstractmethod - def get_repository(self, connection, url, arch, consumer): - """Loads the repository meta information from URL. - - :param connection: the connection manager instance - :param url: the repository`s url - :param arch: the repository`s architecture - :param consumer: the callback to consume result - """ - - @abc.abstractmethod - def get_packages(self, connection, repository, consumer): - """Loads packages from repository. - - :param connection: the connection manager instance - :param repository: the repository object - :param consumer: the callback to consume result - """ - - @abc.abstractmethod - def fork_repository(self, connection, repository, destination, - source=False, locale=False): - """Creates the new repository with same metadata. - - :param connection: the connection manager instance - :param repository: the source repository - :param destination: the destination folder - :param source: copy source files - :param locale: copy localisation - :return: The copy of repository - """ - - @abc.abstractmethod - def rebuild_repository(self, repository, packages): - """Re-builds the repository. - - :param repository: the target repository - :param packages: the set of packages - """ diff --git a/packetary/drivers/deb_driver.py b/packetary/drivers/deb_driver.py deleted file mode 100644 index 030f318..0000000 --- a/packetary/drivers/deb_driver.py +++ /dev/null @@ -1,374 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from contextlib import closing -import copy -import datetime -import fcntl -import gzip -import os - -from debian import deb822 -from debian import debfile -from debian.debian_support import Version -import six - -from packetary.drivers.base import RepositoryDriverBase -from packetary.library.checksum import composite as checksum_composite -from packetary.library.streams import GzipDecompress -from packetary.library import utils -from packetary.objects import FileChecksum -from packetary.objects import Package -from packetary.objects import PackageRelation -from packetary.objects import Repository - - -_OPERATORS_MAPPING = { - '>>': 'gt', - '<<': 'lt', - '=': 'eq', - '>=': 'ge', - '<=': 'le', -} - -_ARCHITECTURES = { - "x86_64": "amd64", - "i386": "i386", - "source": "Source", - "amd64": "x86_64", -} - -_PRIORITIES = { - "required": 1, - "important": 2, - "standard": 3, - "optional": 4, - "extra": 5 -} - -# Order is important -_REPOSITORY_FILES = [ - "Packages", - "Release", - "Packages.gz" -] - -# TODO(should be configurable) -_MANDATORY_PRIORITY = 3 - -_CHECKSUM_METHODS = ( - "MD5Sum", - "SHA1", - "SHA256" -) - -_checksum_collector = checksum_composite('md5', 'sha1', 'sha256') - - -class DebRepositoryDriver(RepositoryDriverBase): - def parse_urls(self, urls): - """Overrides method of superclass.""" - for url in urls: - try: - tokens = iter(x for x in url.split(" ") if x) - base, suite = next(tokens), next(tokens) - components = list(tokens) - except StopIteration: - raise ValueError("Invalid url: {0}".format(url)) - - base = base.rstrip("/") - if base.endswith("/dists"): - base = base[:-6] - - # TODO(Flat Repository Format[1]) - # [1] https://wiki.debian.org/RepositoryFormat - for component in components: - yield (base, suite, component) - - def get_repository(self, connection, url, arch, consumer): - """Overrides method of superclass.""" - - base, suite, component = url - release = self._get_url_of_metafile( - (base, suite, component, arch), "Release" - ) - deb_release = deb822.Release(connection.open_stream(release)) - consumer(Repository( - name=(deb_release["Archive"], deb_release["Component"]), - architecture=arch, - origin=deb_release["origin"], - url=base + "/" - )) - - def get_packages(self, connection, repository, consumer): - """Overrides method of superclass.""" - index = self._get_url_of_metafile(repository, "Packages.gz") - stream = GzipDecompress(connection.open_stream(index)) - self.logger.info("loading packages from %s ...", repository) - pkg_iter = deb822.Packages.iter_paragraphs(stream) - counter = 0 - for dpkg in pkg_iter: - try: - consumer(Package( - repository=repository, - name=dpkg["package"], - version=Version(dpkg['version']), - filesize=int(dpkg.get('size', -1)), - filename=dpkg["filename"], - checksum=FileChecksum( - md5=dpkg.get("md5sum"), - sha1=dpkg.get("sha1"), - sha256=dpkg.get("sha256"), - ), - mandatory=self._is_mandatory(dpkg), - # Recommends are installed by default (since Lucid) - requires=self._get_relations( - dpkg, "depends", "pre-depends", "recommends" - ), - # The deb does not have obsoletes section - obsoletes=[], - provides=self._get_relations(dpkg, "provides"), - )) - except KeyError as e: - self.logger.error( - "Malformed index %s - %s: %s", - repository, six.text_type(dpkg), six.text_type(e) - ) - raise - counter += 1 - - self.logger.info("loaded: %d packages from %s.", counter, repository) - - def rebuild_repository(self, repository, packages): - """Overrides method of superclass.""" - basedir = utils.get_path_from_url(repository.url) - index_file = utils.get_path_from_url( - self._get_url_of_metafile(repository, "Packages") - ) - utils.ensure_dir_exist(os.path.dirname(index_file)) - index_gz = index_file + ".gz" - count = 0 - with open(index_file, "wb") as fd1: - with closing(gzip.open(index_gz, "wb")) as fd2: - writer = utils.composite_writer(fd1, fd2) - for pkg in packages: - filename = os.path.join(basedir, pkg.filename) - with closing(debfile.DebFile(filename)) as deb: - debcontrol = deb.debcontrol() - debcontrol.setdefault("Origin", repository.origin) - debcontrol["Size"] = str(pkg.filesize) - debcontrol["Filename"] = pkg.filename - for k, v in six.moves.zip(_CHECKSUM_METHODS, pkg.checksum): - debcontrol[k] = v - writer(debcontrol.dump()) - writer("\n") - count += 1 - self.logger.info("saved %d packages in %s", count, repository) - self._update_suite_index(repository) - - def fork_repository(self, connection, repository, destination, - source=False, locale=False): - # TODO(download gpk) - # TODO(sources and locales) - new_repo = copy.copy(repository) - new_repo.url = utils.localize_repo_url(destination, repository.url) - packages_file = utils.get_path_from_url( - self._get_url_of_metafile(new_repo, "Packages") - ) - release_file = utils.get_path_from_url( - self._get_url_of_metafile(new_repo, "Release") - ) - self.logger.info( - "clone repository %s to %s", repository, new_repo.url - ) - utils.ensure_dir_exist(os.path.dirname(release_file)) - - release = deb822.Release() - release["Origin"] = repository.origin - release["Label"] = repository.origin - release["Archive"] = repository.name[0] - release["Component"] = repository.name[1] - release["Architecture"] = _ARCHITECTURES[repository.architecture] - with open(release_file, "wb") as fd: - release.dump(fd) - - open(packages_file, "ab").close() - gzip.open(packages_file + ".gz", "ab").close() - return new_repo - - def _update_suite_index(self, repository): - """Updates the Release file in the suite.""" - path = os.path.join( - utils.get_path_from_url(repository.url), - "dists", repository.name[0] - ) - release_path = os.path.join(path, "Release") - self.logger.info( - "added repository suite release file: %s", release_path - ) - with open(release_path, "a+b") as fd: - fcntl.flock(fd.fileno(), fcntl.LOCK_EX) - try: - fd.seek(0) - release = deb822.Release(fd) - self._add_to_release(release, repository) - for m in _CHECKSUM_METHODS: - release.setdefault(m, []) - - self._add_files_to_release( - release, path, self._get_metafiles(repository) - ) - - fd.truncate(0) - release.dump(fd) - finally: - fcntl.flock(fd.fileno(), fcntl.LOCK_UN) - - def _get_relations(self, dpkg, *names): - """Gets the package relations. - - :param dpkg: the debian-package object - :type dpkg: deb822.Packages - :param names: the relation names - :return: the list of PackageRelation objects - """ - relations = list() - for name in names: - for variants in dpkg.relations[name]: - relation = PackageRelation.from_args( - *(self._unparse_relation(v) for v in variants) - ) - if relation is not None: - relations.append(relation) - return relations - - def _get_metafiles(self, repository): - """Gets the sequence of metafiles for repository.""" - return ( - utils.get_path_from_url( - self._get_url_of_metafile(repository, filename) - ) - for filename in _REPOSITORY_FILES - - ) - - @staticmethod - def _unparse_relation(relation): - """Gets the relation parameters. - - :param relation: the deb822.Releation object - :return: tuple(name, version_compare, version_edge) - """ - name = relation['name'] - version = relation.get("version") - if version is None: - return name, None - else: - return name, _OPERATORS_MAPPING[version[0]], version[1] - - @staticmethod - def _is_mandatory(dpkg): - """Checks that package is mandatory. - - :param dpkg: the debian-package object - :type dpkg: deb822.Packages - """ - if dpkg.get("essential") == "yes": - return True - - return _PRIORITIES.get( - dpkg.get("priority"), _MANDATORY_PRIORITY + 1 - ) < _MANDATORY_PRIORITY - - @staticmethod - def _get_url_of_metafile(repo_or_comps, filename): - """Gets the URL of meta-file. - - :param repo_or_comps: the repository object or - tuple(baseurl, suite, component, architecture) - :param filename: the name of meta-file - """ - if isinstance(repo_or_comps, Repository): - baseurl = repo_or_comps.url - suite, component = repo_or_comps.name - arch = repo_or_comps.architecture - else: - baseurl, suite, component, arch = repo_or_comps - - return "/".join(( - baseurl.rstrip("/"), "dists", suite, component, - "binary-" + _ARCHITECTURES[arch], - filename - )) - - @staticmethod - def _add_to_release(release, repository): - """Adds repository information to debian release. - - :param release: the deb822.Release instance - :param repository: the repository object - """ - - # reset the date - release["Date"] = datetime.datetime.now().strftime( - "%a, %d %b %Y %H:%M:%S %Z" - ) - release.setdefault("Origin", repository.origin) - release.setdefault("Label", repository.origin) - release.setdefault("Suite", repository.name[0]) - release.setdefault("Codename", repository.name[0].split("-", 1)[0]) - release.setdefault("Description", "The packages repository.") - - keys = ("Architectures", "Components") - values = (repository.architecture, repository.name[1]) - for key, value in six.moves.zip(keys, values): - if key in release: - release[key] = utils.append_token_to_string( - release[key], - value - ) - else: - release[key] = value - - @staticmethod - def _add_files_to_release(release, basepath, files): - """Adds information about meta files to debian release. - - :param release: the deb822.Release instance - :param basepath: the suite folder path - :param files: the sequence of files - """ - - files_info = utils.get_size_and_checksum_for_files( - files, _checksum_collector - ) - for filepath, size, cs in files_info: - fname = filepath[len(basepath) + 1:] - size = six.text_type(size) - for m, checksum in six.moves.zip(_CHECKSUM_METHODS, cs): - for v in release[m]: - if v["name"] == fname: - v[m] = checksum - v["size"] = size - break - else: - release[m].append(deb822.Deb822Dict({ - m: checksum, - "size": size, - "name": fname - })) diff --git a/packetary/drivers/rpm_driver.py b/packetary/drivers/rpm_driver.py deleted file mode 100644 index 8453e6e..0000000 --- a/packetary/drivers/rpm_driver.py +++ /dev/null @@ -1,283 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import copy -import multiprocessing -import os -import shutil - -import createrepo -import lxml.etree as etree -import six - -from packetary.drivers.base import RepositoryDriverBase -from packetary.library.streams import GzipDecompress -from packetary.library import utils -from packetary.objects import FileChecksum -from packetary.objects import Package -from packetary.objects import PackageRelation -from packetary.objects import PackageVersion -from packetary.objects import Repository - - -urljoin = six.moves.urllib.parse.urljoin - -# TODO(configurable option for drivers) -_CORE_GROUPS = ("core", "base") - -_MANDATORY_TYPES = ("mandatory", "default") - -# The namespaces are used in metadata xml of repository -_NAMESPACES = { - "main": "http://linux.duke.edu/metadata/common", - "md": "http://linux.duke.edu/metadata/repo", - "rpm": "http://linux.duke.edu/metadata/rpm" -} - - -class CreaterepoCallBack(object): - """Callback object for createrepo""" - def __init__(self, logger): - self.logger = logger - - def errorlog(self, msg): - """Error log output.""" - self.logger.error(msg) - - def log(self, msg): - """Logs message.""" - self.logger.info(msg) - - def progress(self, item, current, total): - """"Progress bar.""" - pass - - -class RpmRepositoryDriver(RepositoryDriverBase): - def parse_urls(self, urls): - """Overrides method of superclass.""" - return (url.rstrip("/") for url in urls) - - def get_repository(self, connection, url, arch, consumer): - name = utils.get_path_from_url(url, False) - consumer(Repository( - name=name, - url=url + "/", - architecture=arch, - origin="" - )) - - def get_packages(self, connection, repository, consumer): - """Overrides method of superclass.""" - baseurl = repository.url - repomd = urljoin(baseurl, "repodata/repomd.xml") - self.logger.debug("repomd: %s", repomd) - - repomd_tree = etree.parse(connection.open_stream(repomd)) - mandatory = self._get_mandatory_packages( - self._load_db( - connection, baseurl, repomd_tree, "group_gz", "group" - ) - ) - primary_db = self._load_db(connection, baseurl, repomd_tree, "primary") - if primary_db is None: - raise ValueError("Malformed repository: {0}".format(repository)) - - counter = 0 - for tag in primary_db.iterfind("./main:package", _NAMESPACES): - try: - name = tag.find("./main:name", _NAMESPACES).text - consumer(Package( - repository=repository, - name=tag.find("./main:name", _NAMESPACES).text, - version=self._unparse_version_attrs( - tag.find("./main:version", _NAMESPACES).attrib - ), - filesize=int( - tag.find("./main:size", _NAMESPACES) - .attrib.get("package", -1) - ), - filename=tag.find( - "./main:location", _NAMESPACES - ).attrib["href"], - checksum=self._get_checksum(tag), - mandatory=name in mandatory, - requires=self._get_relations(tag, "requires"), - obsoletes=self._get_relations(tag, "obsoletes"), - provides=self._get_relations(tag, "provides") - )) - except (ValueError, KeyError) as e: - self.logger.error( - "Malformed tag %s - %s: %s", - repository, etree.tostring(tag), six.text_type(e) - ) - raise - counter += 1 - self.logger.info("loaded: %d packages from %s.", counter, repository) - - def rebuild_repository(self, repository, packages): - """Overrides method of superclass.""" - basepath = utils.get_path_from_url(repository.url) - self.logger.info("rebuild repository in %s", basepath) - md_config = createrepo.MetaDataConfig() - try: - md_config.workers = multiprocessing.cpu_count() - md_config.directory = str(basepath) - md_config.update = True - mdgen = createrepo.MetaDataGenerator( - config_obj=md_config, callback=CreaterepoCallBack(self.logger) - ) - mdgen.doPkgMetadata() - mdgen.doRepoMetadata() - mdgen.doFinalMove() - except createrepo.MDError as e: - err_msg = six.text_type(e) - self.logger.exception( - "failed to create yum repository in %s: %s", - basepath, - err_msg - ) - shutil.rmtree( - os.path.join(md_config.outputdir, md_config.tempdir), - ignore_errors=True - ) - raise RuntimeError( - "Failed to create yum repository in {0}." - .format(err_msg)) - - def fork_repository(self, connection, repository, destination, - source=False, locale=False): - # TODO(download gpk) - # TODO(sources and locales) - new_repo = copy.copy(repository) - new_repo.url = utils.localize_repo_url(destination, repository.url) - self.logger.info( - "clone repository %s to %s", repository, new_repo.url - ) - utils.ensure_dir_exist(new_repo.url) - self.rebuild_repository(new_repo, set()) - return new_repo - - def _load_db(self, connection, baseurl, repomd, *aliases): - """Loads database. - - :param connection: the connection object - :param baseurl: the base repository URL - :param repomd: the parsed metadata of repository - :param aliases: the aliases of database name - :return: parsed database file or None if db does not exist - """ - - for dbname in aliases: - self.logger.debug("loading %s database...", dbname) - node = repomd.find( - "./md:data[@type='{0}']".format(dbname), _NAMESPACES - ) - if node is not None: - break - else: - return - - url = urljoin( - baseurl, - node.find("./md:location", _NAMESPACES).attrib["href"] - ) - self.logger.debug("loading %s - %s...", dbname, url) - stream = connection.open_stream(url) - if url.endswith(".gz"): - stream = GzipDecompress(stream) - return etree.parse(stream) - - def _get_mandatory_packages(self, groups_db): - """Get the set of mandatory package names. - - :param groups_db: the parsed groups database - """ - package_names = set() - if groups_db is None: - return package_names - count = 0 - for name in _CORE_GROUPS: - result = groups_db.xpath("./group/id[text()='{0}']".format(name)) - if len(result) == 0: - self.logger.warning("the group '%s' is not found.", name) - continue - group = result[0].getparent() - for t in _MANDATORY_TYPES: - xpath = "./packagelist/packagereq[@type='{0}']".format(t) - for tag in group.iterfind(xpath): - package_names.add(tag.text) - count += 1 - self.logger.info("detected %d mandatory packages.", count) - return package_names - - def _get_relations(self, pkg_tag, name): - """Gets package relations by name from package tag. - - :param pkg_tag: the xml-tag with package description - :param name: the relations name - :return: list of PackageRelation objects - """ - relations = list() - append = relations.append - tags_iter = pkg_tag.iterfind( - "./main:format/rpm:%s/rpm:entry" % name, - _NAMESPACES - ) - for elem in tags_iter: - append(PackageRelation.from_args( - self._unparse_relation_attrs(elem.attrib) - )) - - return relations - - def _get_checksum(self, pkg_tag): - """Gets checksum from package tag.""" - checksum = dict.fromkeys(("md5", "sha1", "sha256"), None) - checksum_tag = pkg_tag.find("./main:checksum", _NAMESPACES) - checksum[checksum_tag.attrib["type"]] = checksum_tag.text - return FileChecksum(**checksum) - - def _unparse_relation_attrs(self, attrs): - """Gets the package relation from attributes. - - :param attrs: the relation tag attributes - :return tuple(name, version_op, version_edge) - """ - if "flags" not in attrs: - return attrs['name'], None - - return ( - attrs['name'], - attrs["flags"].lower(), - self._unparse_version_attrs(attrs) - ) - - @staticmethod - def _unparse_version_attrs(attrs): - """Gets the package version from attributes. - - :param attrs: the relation tag attributes - :return: the PackageVersion object - """ - - return PackageVersion( - int(attrs.get("epoch", 0)), - attrs.get("ver", "0.0").split("."), - attrs.get("rel", "0").split(".") - ) diff --git a/packetary/library/__init__.py b/packetary/library/__init__.py deleted file mode 100644 index 84e9ae6..0000000 --- a/packetary/library/__init__.py +++ /dev/null @@ -1,21 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import eventlet - -eventlet.monkey_patch() diff --git a/packetary/library/checksum.py b/packetary/library/checksum.py deleted file mode 100644 index d8b6b73..0000000 --- a/packetary/library/checksum.py +++ /dev/null @@ -1,84 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import functools -import hashlib - - -class _HashComposite(object): - """Combines several hash methods.""" - - def __init__(self, hash_objects): - self.hash_objects = hash_objects - - def update(self, data): - """Updates the hash objects with the string arg. - - For more details see doc of hashlib.update. - """ - for o in self.hash_objects: - o.update(data) - - def hexdigest(self): - """Returns the list of appropriate hexdigests of hash_objects. - - For more details see doc of hashlib.hexdigest. - """ - return [o.hexdigest() for o in self.hash_objects] - - -def _new_composite(methods): - """Creates new composite method.""" - - def wrapper(): - return _HashComposite([x() for x in methods]) - return wrapper - - -def _checksum(method): - """Makes function to calculate checksum for stream.""" - @functools.wraps(method) - def calculate(stream, chunksize=16 * 1024): - """Calculates checksum for binary stream. - - :param stream: file-like object opened in binary mode. - :return: the checksum of content in terms of method. - """ - - s = method() - while True: - chunk = stream.read(chunksize) - if not chunk: - break - s.update(chunk) - return s.hexdigest() - return calculate - - -md5 = _checksum(hashlib.md5) - -sha1 = _checksum(hashlib.sha1) - -sha256 = _checksum(hashlib.sha256) - - -def composite(*methods): - """Calculate several checksum at one time.""" - return _checksum(_new_composite( - [getattr(hashlib, x) for x in methods] - )) diff --git a/packetary/library/connections.py b/packetary/library/connections.py deleted file mode 100644 index 49b6c9b..0000000 --- a/packetary/library/connections.py +++ /dev/null @@ -1,307 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import errno -import logging -import os -import six -import six.moves.http_client as http_client -import six.moves.urllib.request as urllib -import six.moves.urllib_error as urlerror -import time - -from packetary.library.streams import StreamWrapper -from packetary.library.utils import ensure_dir_exist - - -logger = logging.getLogger(__package__) - - -RETRYABLE_ERRORS = (http_client.HTTPException, IOError) - - -class RangeError(urlerror.URLError): - pass - - -class RetryableRequest(urllib.Request): - MAX_TIMEOUT = 5 - - offset = 0 - retries_left = 1 - retry_interval = 0 - start_time = 0 - - def get_retry_interval(self): - """Calculates progressive retry interval in seconds. - - :return: the time to wait before start retry - """ - # we uses progressive timeout between retries, - # the greatest number of retry will have greatest timeout - # but limited with max_delay - coef = max(self.MAX_TIMEOUT - self.retries_left, 1) - timeout = self.retry_interval * coef - return min(timeout, self.MAX_TIMEOUT) - - -class ResumableResponse(StreamWrapper): - """The http-response wrapper to add resume ability. - - Allows to resume read from same position if connection is lost. - """ - - def __init__(self, request, response, opener): - """Initialises. - - :param request: the original http request - :param response: the original http response - :param opener: the instance of urllib.OpenerDirector - """ - super(ResumableResponse, self).__init__(response) - self.request = request - self.opener = opener - - def read_chunk(self, chunksize): - """Overrides super class method.""" - while 1: - try: - chunk = self.stream.read(chunksize) - self.request.offset += len(chunk) - return chunk - except RETRYABLE_ERRORS as e: - # TODO(check hashsums) - response = self.opener.error( - self.request.get_type(), self.request, - self.stream, 502, six.text_type(e), self.stream.info() - ) - self.stream = response.stream - - -class RetryHandler(urllib.HTTPRedirectHandler): - """urllib Handler to add ability for retrying on server errors.""" - - def redirect_request(self, req, fp, code, msg, headers, newurl): - new_req = urllib.HTTPRedirectHandler.redirect_request( - self, req, fp, code, msg, headers, newurl - ) - if new_req is not None: - # We use class assignment for casting new request to type - # RetryableRequest - new_req.__class__ = RetryableRequest - new_req.retries_left = req.retries_left - new_req.offset = req.offset - new_req.start_time = req.start_time - new_req.retry_interval = req.retry_interval - return new_req - - @staticmethod - def http_request(request): - """Initialises http request. - - :param request: the instance of RetryableRequest - :return: the request - """ - logger.debug("start request: %s", request.get_full_url()) - if request.offset > 0: - request.add_header('Range', 'bytes=%d-' % request.offset) - request.start_time = time.time() - return request - - def http_response(self, request, response): - """Wraps response in a ResumableResponse. - - Checks that partial request completed successfully. - :param request: the instance of RetryableRequest - :param response: the response object - :return: ResumableResponse if success otherwise same response - """ - code, msg = response.getcode(), response.msg - - if 300 <= code < 400: - # the redirect group, pass to next handler as is - return response - - # the server should response partial content if range is specified - if request.offset > 0 and code != 206: - raise RangeError(msg) - - if code >= 400: - logger.error( - "request failed: %s - %d(%s), retries left - %d.", - request.get_full_url(), code, msg, request.retries_left - 1 - ) - if is_retryable_http_error(code) and request.retries_left > 0: - time.sleep(request.get_retry_interval()) - request.retries_left -= 1 - response = self.parent.open(request) - # pass response to next handler as is. - return response - - logger.debug( - "request completed: %s - %d (%s), duration - %d ms.", - request.get_full_url(), response.getcode(), response.msg, - int((time.time() - request.start_time) * 1000) - ) - - return ResumableResponse(request, response, self.parent) - - https_request = http_request - https_response = http_response - - -def is_retryable_http_error(code): - """Checks that http error can be retried. - - :param code: the HTTP_CODE - :return: True if request can be retried otherwise False - """ - return code >= http_client.INTERNAL_SERVER_ERROR - - -class ConnectionsManager(object): - """The connections manager.""" - - def __init__(self, proxy=None, secure_proxy=None, - retries_num=0, retry_interval=0): - """Initialises. - - :param proxy: the url of proxy for http-connections - :param secure_proxy: the url of proxy for https-connections - :param retries_num: the number of allowed retries - :param retry_interval: the time between retries (in seconds) - """ - if proxy: - proxies = { - "http": proxy, - "https": secure_proxy or proxy, - } - else: - proxies = None - - self.retries_num = retries_num - self.retry_interval = retry_interval - self.opener = urllib.build_opener( - RetryHandler(), - urllib.ProxyHandler(proxies) - ) - - def make_request(self, url, offset=0): - """Makes new http request. - - :param url: the remote file`s url - :param offset: the number of bytes from the beginning, - that will be skipped - :return: The new http request - """ - - if url.startswith("/"): - url = "file://" + url - - request = RetryableRequest(url) - request.retries_left = self.retries_num - request.retry_interval = self.retry_interval - request.offset = offset - return request - - def open_stream(self, url, offset=0): - """Opens remote file for streaming. - - :param url: the remote file`s url - :param offset: the number of bytes from the beginning, - that will be skipped - """ - - request = self.make_request(url, offset) - while 1: - try: - return self.opener.open(request) - except (RangeError, urlerror.HTTPError): - raise - except RETRYABLE_ERRORS as e: - if request.retries_left <= 0: - raise - request.retries_left -= 1 - logger.exception( - "Failed to open url - %s: %s. retries left - %d.", - url, six.text_type(e), request.retries_left - ) - time.sleep(request.get_retry_interval()) - - def retrieve(self, url, filename, **attributes): - """Downloads remote file. - - :param url: the remote file`s url - :param filename: the target filename on local filesystem - :param attributes: the file attributes, like size, hashsum, etc. - :return: the count of actually copied bytes - """ - offset = 0 - try: - stats = os.stat(filename) - expected_size = attributes.get('size', -1) - if expected_size == stats.st_size: - # TODO(check hashsum) - return 0 - - if stats.st_size < expected_size: - offset = stats.st_size - except OSError as e: - if e.errno != errno.ENOENT: - raise - ensure_dir_exist(os.path.dirname(filename)) - - logger.info("download: %s from the offset: %d", url, offset) - - fd = os.open(filename, os.O_CREAT | os.O_WRONLY) - try: - return self._copy_stream(fd, url, offset) - except RangeError: - if offset == 0: - raise - logger.warning( - "Failed to resume download, starts from the beginning: %s", - url - ) - return self._copy_stream(fd, url, 0) - finally: - os.fsync(fd) - os.close(fd) - - def _copy_stream(self, fd, url, offset): - """Copies remote file to local. - - :param fd: the file`s descriptor - :param url: the remote file`s url - :param offset: the number of bytes from the beginning, - that will be skipped - :return: the count of actually copied bytes - """ - - source = self.open_stream(url, offset) - os.ftruncate(fd, offset) - os.lseek(fd, offset, os.SEEK_SET) - chunk_size = 16 * 1024 - size = 0 - while 1: - chunk = source.read(chunk_size) - if not chunk: - break - os.write(fd, chunk) - size += len(chunk) - return size diff --git a/packetary/library/executor.py b/packetary/library/executor.py deleted file mode 100644 index 3e5ef47..0000000 --- a/packetary/library/executor.py +++ /dev/null @@ -1,93 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from __future__ import with_statement - -import logging -import sys - -import six - -from eventlet.greenpool import GreenPool - - -logger = logging.getLogger(__package__) - - -class AsynchronousSection(object): - """Allows calling function asynchronously with waiting on exit.""" - - MIN_POOL_SIZE = 1 - - def __init__(self, size=0, ignore_errors_num=0): - """Initialises. - - :param size: the max number of parallel tasks - :param ignore_errors_num: - number of errors which does not stop the execution - """ - - self.executor = GreenPool(max(size, self.MIN_POOL_SIZE)) - self.ignore_errors_num = ignore_errors_num - self.errors = [] - self.tasks = set() - - def __enter__(self): - self.errors[:] = [] - return self - - def __exit__(self, etype, *_): - self.wait(etype is not None) - - def execute(self, func, *args, **kwargs): - """Calls function asynchronously.""" - if 0 <= self.ignore_errors_num < len(self.errors): - raise RuntimeError("Too many errors.") - - gt = self.executor.spawn(func, *args, **kwargs) - self.tasks.add(gt) - gt.link(self.on_complete) - return gt - - def on_complete(self, gt): - """Callback to handle task completion.""" - - try: - gt.wait() - except Exception as e: - logger.error("Task failed: %s", six.text_type(e)) - self.errors.append(sys.exc_info()) - finally: - self.tasks.discard(gt) - - def wait(self, ignore_errors=False): - """Waits until all tasks will be completed. - - Do not use directly, will be called from context manager. - """ - self.executor.waitall() - if len(self.errors) > 0: - for exc_info in self.errors: - logger.exception("error details.", exc_info=exc_info) - - self.errors[:] = [] - if not ignore_errors: - raise RuntimeError( - "Operations completed with errors.\n" - "See log for more details." - ) diff --git a/packetary/library/streams.py b/packetary/library/streams.py deleted file mode 100644 index 5c6f4bf..0000000 --- a/packetary/library/streams.py +++ /dev/null @@ -1,133 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import zlib - - -class StreamWrapper(object): - """Helper class to implement stream wrappers. - - It is base-class for Streamers, - that provides functionality to transform stream on the fly. - The wrapped stream may return data more that required, - the extra read data will be kept in the internal buffer till - next read. - """ - - CHUNK_SIZE = 1024 - - def __init__(self, stream): - """Initializes. - - :param stream: file-like object opened in binary mode. - """ - self.stream = stream - self.unread_tail = b"" - - def __getattr__(self, item): - return getattr(self.stream, item) - - def _read_tail(self): - tmp = self.unread_tail - self.unread_tail = b"" - return tmp - - def _align_chunk(self, chunk, size): - self.unread_tail = chunk[size:] - return chunk[:size] - - def read_chunk(self, chunksize): - """Overrides this method to change default behaviour.""" - return self.stream.read(chunksize) - - def read(self, size=-1): - result = self._read_tail() - if size < 0: - while True: - chunk = self.read_chunk(self.CHUNK_SIZE) - if not chunk: - break - result += chunk - else: - if len(result) > size: - result = self._align_chunk(result, size) - size -= len(result) - while size > 0: - chunk = self.read_chunk(self.CHUNK_SIZE) - if not chunk: - break - if len(chunk) > size: - chunk = self._align_chunk(chunk, size) - size -= len(chunk) - result += chunk - return result - - def readline(self): - pos = self.unread_tail.find(b"\n") - if pos >= 0: - line = self._align_chunk(self.unread_tail, pos + 1) - else: - line = self._read_tail() - while True: - chunk = self.read_chunk(self.CHUNK_SIZE) - if not chunk: - break - pos = chunk.find(b"\n") - if pos >= 0: - line += self._align_chunk(chunk, pos + 1) - break - line += chunk - return line - - def readlines(self): - while True: - line = self.readline() - if not line: - break - yield line - - def __iter__(self): - return self.readlines() - - -class GzipDecompress(StreamWrapper): - """The decompress stream.""" - - def __init__(self, stream): - super(GzipDecompress, self).__init__(stream) - # Magic parameter makes zlib module understand gzip header - # http://stackoverflow.com/questions/1838699/how-can-i-decompress-a-gzip-stream-with-zlib - # This works on cpython and pypy, but not jython. - self.decompress = zlib.decompressobj(16 + zlib.MAX_WBITS) - - def read_chunk(self, chunksize): - if self.decompress.unconsumed_tail: - uncompressed = self.decompress.decompress( - self.decompress.unconsumed_tail, chunksize - ) - if uncompressed: - return uncompressed - - while True: - chunk = self.stream.read(chunksize) - if not chunk: - break - uncompressed = self.decompress.decompress(chunk, chunksize) - if uncompressed: - return uncompressed - return self.decompress.flush() diff --git a/packetary/library/utils.py b/packetary/library/utils.py deleted file mode 100644 index 350892f..0000000 --- a/packetary/library/utils.py +++ /dev/null @@ -1,115 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from __future__ import with_statement - -import errno -import os - -import six - -urlparse = six.moves.urllib.parse.urlparse - - -def append_token_to_string(tokens, token): - """Adds new token to space separated list of tokens. - - :param tokens: the 'sep' separated list - :param token: new item - """ - values = tokens.split() - if token not in values: - values.append(token) - values.sort() - return ' '.join(values) - return tokens - - -def composite_writer(*args): - """Makes helper, that writes into several files simultaneously. - - :param args: the list of file objects - :return: the callable object - writer - """ - def write(text): - """Writes simultaneously to all files with utf-8 encoding control. - - :param text: the text, that needs to write - """ - if isinstance(text, six.text_type): - text = text.encode("utf-8") - for arg in args: - arg.write(text) - return write - - -def get_size_and_checksum_for_files(files, checksum_algo): - """Gets the path, size and checksum for files. - - :param files: the sequence of files - :param checksum_algo: the checksum calculator - :return the sequence of tuples(filename, size, checksum) - """ - - for filename in files: - with open(filename, "rb") as fd: - size = os.fstat(fd.fileno()).st_size - checksum = checksum_algo(fd) - yield filename, size, checksum - - -def get_path_from_url(url, ensure_file=True): - """Get the path from the URL. - - :param url: the URL - :param ensure_file: If True, ensure that scheme is "file" - :return: the path component from URL - :raises ValueError - """ - - comps = urlparse(url, scheme="file") - if ensure_file and comps.scheme != "file": - raise ValueError( - "The absolute path is expected, actual have: {0}.".format(url) - ) - if os.sep != "/": - return comps.path.replace("/", os.sep) - return comps.path - - -def localize_repo_url(localurl, repo_url): - """Gets local repository url. - - :param localurl: the base local URL - :param repo_url: the origin URL of repository - :return: localurl + get_path_from_url(repo_url) - """ - return localurl.rstrip("/") + urlparse(repo_url).path - - -def ensure_dir_exist(path): - """Creates directory if it does not exist. - - :param path: the full path to directory - """ - - try: - os.makedirs(path) - except OSError as e: - if e.errno != errno.EEXIST: - raise diff --git a/packetary/objects/__init__.py b/packetary/objects/__init__.py deleted file mode 100644 index e1b4e5b..0000000 --- a/packetary/objects/__init__.py +++ /dev/null @@ -1,38 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from packetary.objects.index import Index -from packetary.objects.package import FileChecksum -from packetary.objects.package import Package -from packetary.objects.package_relation import PackageRelation -from packetary.objects.package_relation import VersionRange -from packetary.objects.package_version import PackageVersion -from packetary.objects.packages_tree import PackagesTree -from packetary.objects.repository import Repository - - -__all__ = [ - "FileChecksum", - "Index", - "Package", - "PackageRelation", - "PackagesTree", - "PackageVersion", - "Repository", - "VersionRange", -] diff --git a/packetary/objects/base.py b/packetary/objects/base.py deleted file mode 100644 index 2f3cf14..0000000 --- a/packetary/objects/base.py +++ /dev/null @@ -1,62 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import abc -import six - - -@six.add_metaclass(abc.ABCMeta) -class ComparableObject(object): - """Superclass for objects, that should be comparable. - - Note: because python3 does not support __cmp__ slot, use - cmp method to implement all of compare methods. - """ - - @abc.abstractmethod - def cmp(self, other): - """Compares with other object. - - :return: value is negative if if self < other, zero if self == other - strictly positive if x > y - """ - - def __lt__(self, other): - return self.cmp(other) < 0 - - def __le__(self, other): - return self.cmp(other) <= 0 - - def __gt__(self, other): - return self.cmp(other) > 0 - - def __ge__(self, other): - return self.cmp(other) >= 0 - - def __eq__(self, other): - if other is self: - return True - return isinstance(other, type(self)) and self.cmp(other) == 0 - - def __ne__(self, other): - if other is self: - return False - return not isinstance(other, type(self)) or self.cmp(other) != 0 - - def __cmp__(self, other): - return self.cmp(other) diff --git a/packetary/objects/index.py b/packetary/objects/index.py deleted file mode 100644 index fc43fcd..0000000 --- a/packetary/objects/index.py +++ /dev/null @@ -1,208 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from bintrees import FastRBTree -from collections import defaultdict -import functools -import operator -import six - - -def _make_operator(direction, op): - """Makes search operator from low-level operation and search direction.""" - return functools.partial(direction, condition=op) - - -def _start_upperbound(versions, version, condition): - """Gets all versions from [start, version] that meet condition. - - :param versions: the tree of versions. - :param version: the required version - :param condition: condition for search - :return: the list of found versions - """ - - result = list(versions.value_slice(None, version)) - try: - bound = versions.ceiling_item(version) - if condition(bound[0], version): - result.append(bound[1]) - except KeyError: - pass - return result - - -def _lowerbound_end(versions, version, condition): - """Gets all versions from [version, end] that meet condition. - - :param versions: the tree of versions. - :param version: the required version - :param condition: condition for search - :return: the list of found versions - """ - result = [] - items = iter(versions.item_slice(version, None)) - bound = next(items, None) - if bound is None: - return result - if condition(bound[0], version): - result.append(bound[1]) - result.extend(x[1] for x in items) - return result - - -def _equal(tree, version): - """Gets the package with specified version.""" - if version in tree: - return [tree[version]] - return [] - - -def _any(tree, _): - """Gets the package with max version.""" - return list(tree.values()) - - -class Index(object): - """The search index for packages. - - Builds three search-indexes: - - index of packages with versions. - - index of virtual packages (provides). - - index of obsoleted packages (obsoletes). - - Uses to find package by name and range of versions. - """ - - operators = { - None: _any, - "lt": _make_operator(_start_upperbound, operator.lt), - "le": _make_operator(_start_upperbound, operator.le), - "gt": _make_operator(_lowerbound_end, operator.gt), - "ge": _make_operator(_lowerbound_end, operator.ge), - "eq": _equal, - } - - def __init__(self): - self.packages = defaultdict(FastRBTree) - self.obsoletes = defaultdict(FastRBTree) - self.provides = defaultdict(FastRBTree) - - def __iter__(self): - """Iterates over all packages including versions.""" - return self.get_all() - - def __len__(self, _reduce=six.functools.reduce): - """Returns the total number of packages with versions.""" - return _reduce( - lambda x, y: x + len(y), - six.itervalues(self.packages), - 0 - ) - - def get_all(self): - """Gets sequence from all of packages including versions.""" - - for versions in six.itervalues(self.packages): - for version in versions.values(): - yield version - - def find(self, name, version): - """Finds the package by name and range of versions. - - :param name: the package`s name. - :param version: the range of versions. - :return: the package if it is found, otherwise None - """ - candidates = self.find_all(name, version) - if len(candidates) > 0: - return candidates[-1] - return None - - def find_all(self, name, version): - """Finds the packages by name and range of versions. - - :param name: the package`s name. - :param version: the range of versions. - :return: the list of suitable packages - """ - - if name in self.packages: - candidates = self._find_versions( - self.packages[name], version - ) - if len(candidates) > 0: - return candidates - - if name in self.obsoletes: - return self._resolve_relation( - self.obsoletes[name], version - ) - - if name in self.provides: - return self._resolve_relation( - self.provides[name], version - ) - return [] - - def add(self, package): - """Adds new package to indexes. - - :param package: the package object. - """ - self.packages[package.name][package.version] = package - key = package.name, package.version - - for obsolete in package.obsoletes: - self.obsoletes[obsolete.name][key] = obsolete - - for provide in package.provides: - self.provides[provide.name][key] = provide - - def _resolve_relation(self, relations, version): - """Resolve relation according to relations index. - - :param relations: the index of relations - :param version: the range of versions - :return: package if found, otherwise None - """ - for key, candidate in relations.iter_items(reverse=True): - if candidate.version.has_intersection(version): - return [self.packages[key[0]][key[1]]] - return [] - - @staticmethod - def _find_versions(versions, version): - """Searches accurate version. - - Search for the highest version out of intersection - of existing and required range of versions. - - :param versions: the existing versions - :param version: the required range of versions - :return: package if found, otherwise None - """ - - try: - op = Index.operators[version.op] - except KeyError: - raise ValueError( - "Unsupported operation: {0}" - .format(version.op) - ) - return op(versions, version.edge) diff --git a/packetary/objects/package.py b/packetary/objects/package.py deleted file mode 100644 index cb5a7d7..0000000 --- a/packetary/objects/package.py +++ /dev/null @@ -1,80 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from collections import namedtuple - -from packetary.objects.base import ComparableObject - - -FileChecksum = namedtuple("FileChecksum", ("md5", "sha1", "sha256")) - - -class Package(ComparableObject): - """Structure to describe package object.""" - - def __init__(self, repository, name, version, filename, - filesize, checksum, mandatory=False, - requires=None, provides=None, obsoletes=None): - """Initialises. - - :param name: the package`s name - :param version: the package`s version - :param filename: the package`s relative filename - :param filesize: the package`s file size - :param checksum: the package`s checksum - :param requires: the package`s requirements(optional) - :param provides: the package`s provides(optional) - :param obsoletes: the package`s obsoletes(optional) - :param mandatory: indicates that package is mandatory - """ - - self.repository = repository - self.name = name - self.version = version - self.filename = filename - self.checksum = checksum - self.filesize = filesize - self.requires = requires or [] - self.provides = provides or [] - self.obsoletes = obsoletes or [] - self.mandatory = mandatory - - def __copy__(self): - """Creates shallow copy of package.""" - return Package(**self.__dict__) - - def __str__(self): - return "{0} {1}".format(self.name, self.version) - - def __unicode__(self): - return u"{0} {1}".format(self.name, self.version) - - def __hash__(self): - return hash((self.name, self.version)) - - def cmp(self, other): - """Compares with other Package object.""" - if self.name < other.name: - return -1 - if self.name > other.name: - return 1 - if self.version < other.version: - return -1 - if self.version > other.version: - return 1 - return 0 diff --git a/packetary/objects/package_relation.py b/packetary/objects/package_relation.py deleted file mode 100644 index d24fe9b..0000000 --- a/packetary/objects/package_relation.py +++ /dev/null @@ -1,157 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import operator - - -class VersionRange(object): - """Describes the range of versions. - - Range of version is compare operation and edge. - the compare operation can be one of: - equal, greater, less, greater or equal, less or equal. - """ - - __slots__ = ["op", "edge"] - - def __init__(self, op=None, edge=None): - """Initialises. - - :param op: the name of operator to compare. - :param edge: the edge of versions. - """ - self.op = op - self.edge = edge - - def __contains__(self, point): - return getattr(operator, self.op)(point, self.edge) - - def __hash__(self): - return hash((self.op, self.edge)) - - def __eq__(self, other): - if not isinstance(other, VersionRange): - return False - - return self.op == other.op and \ - self.edge == other.edge - - def __str__(self): - if self.edge is not None: - return "{0} {1}".format(self.op, self.edge) - return "any" - - def __unicode__(self): - if self.edge is not None: - return u"{0} {1}".format(self.op, self.edge) - return u"any" - - def has_intersection(self, other): - """Checks that 2 ranges has intersection. - - :param other: the candidate to check - :return: True if intersection exists, otherwise False - :raise TypeError: when other does not instance of VersionRange - """ - - if not isinstance(other, VersionRange): - raise TypeError( - "Unorderable type and {0}" - .format(type(other)) - ) - - if self.op is None or other.op is None: - return True - - if self.op[0] == other.op[0]: - if self.op == 'eq': - return self.edge == other.edge - # the intersection is -inf or +inf - return True - if self.edge == other.edge: - # need to cover case < a and >= a - return self.edge in other and other.edge in self - # all other cases - return self.edge in other or other.edge in self - - -class PackageRelation(object): - """Describes the package`s relation. - - Relation includes the name of required package - and range of versions that satisfies requirement. - """ - - __slots__ = ["name", "version", "alternative"] - - def __init__(self, name, version=None, alternative=None): - """Initialises. - - :param name: the name of required package - :param version: the version range of required package - :param alternative: the alternative relation - """ - self.name = name - self.version = VersionRange() if version is None else version - self.alternative = alternative - - @classmethod - def from_args(cls, *args): - """Construct relation from list of arguments. - - :param args: the list of tuples(name, [version_op, version_edge]) - """ - if len(args) == 0: - return None - - head = args[0] - name = head[0] - version = VersionRange(*head[1:]) - alternative = cls.from_args(*args[1:]) - return cls(name, version, alternative) - - def __iter__(self): - """Iterates over alternatives.""" - r = self - while r is not None: - yield r - r = r.alternative - - def __hash__(self): - return hash((self.name, self.version)) - - def __eq__(self, other): - if not isinstance(other, PackageRelation): - return False - - return self.name == other.name and \ - self.version == other.version - - def __str__(self): - if self.alternative is None: - return "{0} ({1})".format(self.name, self.version) - return "{0} ({1}) | {2}".format( - self.name, self.version, self.alternative - ) - - def __unicode__(self): - if self.alternative is None: - return u"{0} ({1})".format(self.name, self.version) - return u"{0} ({1}) | {2}".format( - self.name, self.version, self.alternative - ) diff --git a/packetary/objects/package_version.py b/packetary/objects/package_version.py deleted file mode 100644 index 50ead20..0000000 --- a/packetary/objects/package_version.py +++ /dev/null @@ -1,127 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from packetary.objects.base import ComparableObject - - -class PackageVersion(ComparableObject): - """The Package version.""" - - __slots__ = ["epoch", "version", "release"] - - def __init__(self, epoch, version, release): - self.epoch = int(epoch) - self.version = tuple(version) - self.release = tuple(release) - - @classmethod - def from_string(cls, text): - """Constructs from string. - - :param text: the version in format '[{epoch-}]-{version}-{release}' - """ - components = text.split("-") - if len(components) > 2: - epoch = components[0] - components = components[1:] - else: - epoch = 0 - return cls(epoch, components[0].split("."), components[1].split(".")) - - def cmp(self, other): - if not isinstance(other, PackageVersion): - other = PackageVersion.from_string(str(other)) - - if not isinstance(other, PackageVersion): - raise TypeError - if self.epoch < other.epoch: - return -1 - if self.epoch > other.epoch: - return 1 - - res = self._cmp_version_part(self.version, other.version) - if res != 0: - return res - return self._cmp_version_part(self.release, other.release) - - def __eq__(self, other): - if other is self: - return True - return self.cmp(other) == 0 - - def __str__(self): - return "{0}-{1}-{2}".format( - self.epoch, - ".".join(str(x) for x in self.version), - ".".join(str(x) for x in self.release) - ) - - @classmethod - def _order(cls, x): - """Return an integer value for character x""" - if x.isdigit(): - return int(x) + 1 - if x.isalpha(): - return ord(x) - return ord(x) + 256 - - @classmethod - def _cmp_version_string(cls, version1, version2): - """Compares two versions as string.""" - la = [cls._order(x) for x in version1] - lb = [cls._order(x) for x in version2] - while la or lb: - a = 0 - b = 0 - if la: - a = la.pop(0) - if lb: - b = lb.pop(0) - if a < b: - return -1 - elif a > b: - return 1 - return 0 - - @classmethod - def _cmp_version_part(cls, version1, version2): - """Compares two versions.""" - ver1_it = iter(version1) - ver2_it = iter(version2) - while True: - v1 = next(ver1_it, None) - v2 = next(ver2_it, None) - - if v1 is None or v2 is None: - if v1 is not None: - return 1 - if v2 is not None: - return -1 - return 0 - - if v1.isdigit() and v2.isdigit(): - a = int(v1) - b = int(v2) - if a < b: - return -1 - if a > b: - return 1 - else: - r = cls._cmp_version_string(v1, v2) - if r != 0: - return r diff --git a/packetary/objects/packages_tree.py b/packetary/objects/packages_tree.py deleted file mode 100644 index a0f7d83..0000000 --- a/packetary/objects/packages_tree.py +++ /dev/null @@ -1,129 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import warnings - -from packetary.objects.index import Index - - -class UnresolvedWarning(UserWarning): - """Warning about unresolved depends.""" - pass - - -class PackagesTree(Index): - """Helper class to deal with dependency graph.""" - - def __init__(self): - super(PackagesTree, self).__init__() - self.mandatory_packages = [] - - def add(self, package): - super(PackagesTree, self).add(package) - # store all mandatory packages in separated list for quick access - if package.mandatory: - self.mandatory_packages.append(package) - - def get_unresolved_dependencies(self, base=None): - """Gets the set of unresolved dependencies. - - :param base: the base index to resolve dependencies - :return: the set of unresolved depends. - """ - external = self.__get_unresolved_dependencies(self) - if base is None: - return external - - unresolved = set() - for relation in external: - for rel in relation: - if base.find(rel.name, rel.version) is not None: - break - else: - unresolved.add(relation) - return unresolved - - def get_minimal_subset(self, main, requirements): - """Gets the minimal work subset. - - :param main: the main index, to complete requirements. - :param requirements: additional requirements. - :return: The set of resolved depends. - """ - - unresolved = set() - resolved = set() - if main is None: - def pkg_filter(*_): - pass - else: - pkg_filter = main.find - self.__get_unresolved_dependencies(main, requirements) - - stack = list() - stack.append(requirements) - - # add all mandatory packages - for pkg in self.mandatory_packages: - resolved.add(pkg) - stack.append(pkg.requires) - - while len(stack) > 0: - required = stack.pop() - for require in required: - for rel in require: - if rel not in unresolved: - if pkg_filter(rel.name, rel.version) is not None: - break - # use all packages that meets depends - candidates = self.find_all(rel.name, rel.version) - for cand in candidates: - if cand not in resolved: - resolved.add(cand) - stack.append(cand.requires) - if len(candidates) > 0: - break - else: - unresolved.add(require) - msg = "Unresolved depends: {0}".format(require) - warnings.warn(UnresolvedWarning(msg)) - - return resolved - - @staticmethod - def __get_unresolved_dependencies(index, unresolved=None): - """Gets the set of unresolved dependencies. - - :param index: the search index. - :param unresolved: the known list of unresolved packages. - :return: the set of unresolved depends. - """ - - if unresolved is None: - unresolved = set() - - for pkg in index: - for require in pkg.requires: - for rel in require: - if rel not in unresolved: - candidate = index.find(rel.name, rel.version) - if candidate is not None and candidate != pkg: - break - else: - unresolved.add(require) - return unresolved diff --git a/packetary/objects/repository.py b/packetary/objects/repository.py deleted file mode 100644 index f2302e7..0000000 --- a/packetary/objects/repository.py +++ /dev/null @@ -1,48 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - -class Repository(object): - """Structure to describe repository object.""" - - def __init__(self, name, url, architecture, origin): - """Initialises. - - :param name: the repository`s name, may be tuple of strings - :param url: the repository`s URL - :param architecture: the repository`s architecture - :param origin: the repository`s origin - """ - self.name = name - self.url = url - self.architecture = architecture - self.origin = origin - - def __str__(self): - if isinstance(self.name, tuple): - return ".".join(self.name) - return self.name or self.url - - def __unicode__(self): - if isinstance(self.name, tuple): - return u".".join(self.name) - return self.name or self.url - - def __copy__(self): - """Creates shallow copy of package.""" - return Repository(**self.__dict__) diff --git a/packetary/objects/statistics.py b/packetary/objects/statistics.py deleted file mode 100644 index 50bf252..0000000 --- a/packetary/objects/statistics.py +++ /dev/null @@ -1,47 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import copy - - -class CopyStatistics(object): - """The statistics of packages copying""" - def __init__(self): - # the number of copied packages - self.copied = 0 - # the number of total packages - self.total = 0 - - def on_package_copied(self, bytes_copied): - """Proceed next copied package.""" - if bytes_copied > 0: - self.copied += 1 - self.total += 1 - - def __iadd__(self, other): - if not isinstance(other, CopyStatistics): - raise TypeError - - self.copied += other.copied - self.total += other.total - return self - - def __add__(self, other): - result = copy.copy(self) - result += other - return result diff --git a/packetary/tests/__init__.py b/packetary/tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/packetary/tests/base.py b/packetary/tests/base.py deleted file mode 100644 index 74f4445..0000000 --- a/packetary/tests/base.py +++ /dev/null @@ -1,27 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -try: - import unittest2 as unittest -except ImportError: - import unittest - - -class TestCase(unittest.TestCase): - - """Test case base class for all unit tests.""" diff --git a/packetary/tests/data/Packages b/packetary/tests/data/Packages deleted file mode 100644 index b1dcf13..0000000 --- a/packetary/tests/data/Packages +++ /dev/null @@ -1,19 +0,0 @@ -Package: test -Source: test.src -Version: 1.1.1-1~u14.04+test -Architecture: all -Maintainer: Test -Installed-Size: 3509 -Homepage: http://localhost/ -Priority: required -Section: web -Filename: pool/main/t/test.deb -Size: 100 -Depends: test2 (>= 0.8.16~exp9)|tes2-old, test3 -Pre-Depends: test-main -Provides: file -Replaces: test-old -SHA256: 14d6e308d8699b7f9ba2fe1ef778c0e38cf295614d308039d687b6b097d50859 -SHA1: 402bd18c145ae3b5344edf07f246be159397fd40 -MD5sum: 1ae09f80109f40dfbfaf3ba423c8625a -Description: test package diff --git a/packetary/tests/data/Release b/packetary/tests/data/Release deleted file mode 100644 index a54fe08..0000000 --- a/packetary/tests/data/Release +++ /dev/null @@ -1,6 +0,0 @@ -Archive: trusty -Version: 14.04 -Component: main -Origin: Ubuntu -Label: Ubuntu -Architecture: amd64 diff --git a/packetary/tests/data/groups.xml b/packetary/tests/data/groups.xml deleted file mode 100644 index 4fadf5e..0000000 --- a/packetary/tests/data/groups.xml +++ /dev/null @@ -1,17 +0,0 @@ - - - - - core - - test1 - test2 - - - - other - - test1 - - - diff --git a/packetary/tests/data/primary.xml b/packetary/tests/data/primary.xml deleted file mode 100644 index 3be4161..0000000 --- a/packetary/tests/data/primary.xml +++ /dev/null @@ -1,62 +0,0 @@ - - - test1 - x86_64 - - e8ed9e0612e813491ed5e7c10502a39e43ec665afd1321541dea211202707a65 - Test package - Test package - Test - http://localhost/ - - - test2 - x86_64 - - e8ed9e0612e813491ed5e7c10502a39e43ec665afd1321541dea211202707a65 - Test package - Test package - Test - http://localhost/ - - diff --git a/packetary/tests/data/repomd.xml b/packetary/tests/data/repomd.xml deleted file mode 100644 index 990ff06..0000000 --- a/packetary/tests/data/repomd.xml +++ /dev/null @@ -1,20 +0,0 @@ - - - 1427842153 - - 32fa7089953ace14f4a6e722bd33c353fcb94d9678d8a062a3b028e54042319c - 32fa7089953ace14f4a6e722bd33c353fcb94d9678d8a062a3b028e54042319c - - 1427842225 - 2528031 - 23175717 - - - 689a2ef671fe1c2245539e9c7b90e9dcd1236f4a0dd376512cfd509531a2b70d - bb7a4b6a6ccc8b4875b569359aedf67f9678cd56da7f372c134200265e276951 - - 1427842225 - 2528031 - 23175717 - - diff --git a/packetary/tests/data/repomd2.xml b/packetary/tests/data/repomd2.xml deleted file mode 100644 index 198ef47..0000000 --- a/packetary/tests/data/repomd2.xml +++ /dev/null @@ -1,20 +0,0 @@ - - - 1427842153 - - 32fa7089953ace14f4a6e722bd33c353fcb94d9678d8a062a3b028e54042319c - 32fa7089953ace14f4a6e722bd33c353fcb94d9678d8a062a3b028e54042319c - - 1427842225 - 2528031 - 23175717 - - - 689a2ef671fe1c2245539e9c7b90e9dcd1236f4a0dd376512cfd509531a2b70d - bb7a4b6a6ccc8b4875b569359aedf67f9678cd56da7f372c134200265e276951 - - 1427842225 - 2528031 - 23175717 - - diff --git a/packetary/tests/stubs/__init__.py b/packetary/tests/stubs/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/packetary/tests/stubs/executor.py b/packetary/tests/stubs/executor.py deleted file mode 100644 index 051454c..0000000 --- a/packetary/tests/stubs/executor.py +++ /dev/null @@ -1,29 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - -class Executor(object): - def __enter__(self): - return self - - def __exit__(self, *_): - return False - - @staticmethod - def execute(f, *args, **kwargs): - return f(*args, **kwargs) diff --git a/packetary/tests/stubs/generator.py b/packetary/tests/stubs/generator.py deleted file mode 100644 index 5c196bd..0000000 --- a/packetary/tests/stubs/generator.py +++ /dev/null @@ -1,52 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from packetary import objects - - -def gen_repository(name="test", url="file:///test", - architecture="x86_64", origin="Test"): - """Helper to create Repository object with default attributes.""" - return objects.Repository(name, url, architecture, origin) - - -def gen_relation(name="test", version=None, alternative=None): - """Helper to create PackageRelation object with default attributes.""" - return objects.PackageRelation( - name=name, - version=objects.VersionRange(*(version or [])), - alternative=alternative - ) - - -def gen_package(idx=1, **kwargs): - """Helper to create Package object with default attributes.""" - repository = gen_repository() - name = kwargs.setdefault("name", "package{0}".format(idx)) - kwargs.setdefault("repository", repository) - kwargs.setdefault("version", 1) - kwargs.setdefault("checksum", objects.FileChecksum("1", "2", "3")) - kwargs.setdefault("filename", "{0}.pkg".format(name)) - kwargs.setdefault("filesize", 1) - for relation in ("requires", "provides", "obsoletes"): - if relation not in kwargs: - kwargs[relation] = [gen_relation( - "{0}{1}".format(relation, idx), ["le", idx + 1] - )] - - return objects.Package(**kwargs) diff --git a/packetary/tests/stubs/helpers.py b/packetary/tests/stubs/helpers.py deleted file mode 100644 index f4ac845..0000000 --- a/packetary/tests/stubs/helpers.py +++ /dev/null @@ -1,66 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from contextlib import closing -import gzip -import mock -import six - - -class CallbacksAdapter(mock.MagicMock): - """Helper to return data through callback.""" - - def __call__(self, *args, **kwargs): - if len(args) > 0: - callback = args[-1] - else: - callback = None - - if not callable(callback): - return super(CallbacksAdapter, self).__call__(*args, **kwargs) - - args = args[:-1] - data = super(CallbacksAdapter, self).__call__(*args, **kwargs) - - if isinstance(data, list): - for d in data: - callback(d) - else: - callback(data) - - -class Buffer(object): - """Helper to hide BytesIO methods.""" - - def __init__(self, io): - self.io = io - self.reset() - - def reset(self): - self.io.seek(0) - - def read(self, s=-1): - return self.io.read(s) - - -def get_compressed(stream): - """Gets compressed stream.""" - compressed = six.BytesIO() - with closing(gzip.GzipFile(fileobj=compressed, mode="wb")) as gz: - gz.write(stream.read()) - return Buffer(compressed) diff --git a/packetary/tests/test_checksums.py b/packetary/tests/test_checksums.py deleted file mode 100644 index 8b0a7c5..0000000 --- a/packetary/tests/test_checksums.py +++ /dev/null @@ -1,52 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import six - -from packetary.library import checksum -from packetary.tests import base - - -class TestChecksum(base.TestCase): - def test_checksum(self): - stream = six.BytesIO(b"line1\nline2\nline3\n") - checksums = { - checksum.md5: "cc3d5ed5fda53dfa81ea6aa951d7e1fe", - checksum.sha1: "8c84f6f36dd2230d3e9c954fa436e5fda90b1957", - checksum.sha256: "66663af9c7aa341431a8ee2ff27b72" - "abd06c9218f517bb6fef948e4803c19e03" - } - for chunksize in (8, 256): - for algo, expected in six.iteritems(checksums): - stream.seek(0) - self.assertEqual( - expected, algo(stream, chunksize) - ) - - def test_composite(self): - stream = six.BytesIO(b"line1\nline2\nline3\n") - result = checksum.composite('md5', 'sha1', 'sha256')(stream) - self.assertEqual( - [ - "cc3d5ed5fda53dfa81ea6aa951d7e1fe", - "8c84f6f36dd2230d3e9c954fa436e5fda90b1957", - "66663af9c7aa341431a8ee2ff27b72" - "abd06c9218f517bb6fef948e4803c19e03" - ], - result - ) diff --git a/packetary/tests/test_cli_commands.py b/packetary/tests/test_cli_commands.py deleted file mode 100644 index 28594b1..0000000 --- a/packetary/tests/test_cli_commands.py +++ /dev/null @@ -1,154 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock -import subprocess - -# The cmd2 does not work with python3.5 -# because it tries to get access to the property mswindows, -# that was removed in 3.5 -subprocess.mswindows = False - -from packetary.cli.commands import clone -from packetary.cli.commands import packages -from packetary.cli.commands import unresolved -from packetary.tests import base -from packetary.tests.stubs.generator import gen_package -from packetary.tests.stubs.generator import gen_relation -from packetary.tests.stubs.generator import gen_repository -from packetary.tests.stubs.helpers import CallbacksAdapter - - -@mock.patch.multiple( - "packetary.api", - RepositoryController=mock.DEFAULT, - ConnectionsManager=mock.DEFAULT, - AsynchronousSection=mock.MagicMock() -) -@mock.patch( - "packetary.cli.commands.base.BaseRepoCommand.stdout" -) -class TestCliCommands(base.TestCase): - common_argv = [ - "--ignore-errors-num=3", - "--threads-num=8", - "--retries-num=10", - "--retry-interval=1", - "--http-proxy=http://proxy", - "--https-proxy=https://proxy" - ] - - clone_argv = [ - "-o", "http://localhost/origin", - "-d", ".", - "-r", "http://localhost/requires", - "-b", "test-package", - "-t", "deb", - "-a", "x86_64", - "--clean", - ] - - packages_argv = [ - "-o", "http://localhost/origin", - "-t", "deb", - "-a", "x86_64" - ] - - unresolved_argv = [ - "-o", "http://localhost/origin", - "-t", "deb", - "-a", "x86_64" - ] - - def start_cmd(self, cmd, argv): - cmd.debug(argv + self.common_argv) - - def check_context(self, context, ConnectionsManager): - self.assertEqual(3, context._ignore_errors_num) - self.assertEqual(8, context._threads_num) - self.assertIs(context._connection, ConnectionsManager.return_value) - ConnectionsManager.assert_called_once_with( - proxy="http://proxy", - secure_proxy="https://proxy", - retries_num=10, - retry_interval=1 - ) - - def test_clone_cmd(self, stdout, RepositoryController, **kwargs): - ctrl = RepositoryController.load() - ctrl.copy_packages = CallbacksAdapter() - ctrl.load_repositories = CallbacksAdapter() - ctrl.load_packages = CallbacksAdapter() - ctrl.copy_packages.return_value = [1, 0] - repo = gen_repository() - ctrl.load_repositories.side_effect = [repo, gen_repository()] - ctrl.load_packages.side_effect = [ - gen_package(repository=repo), - gen_package() - ] - self.start_cmd(clone, self.clone_argv) - RepositoryController.load.assert_called_with( - mock.ANY, "deb", "x86_64" - ) - self.check_context( - RepositoryController.load.call_args[0][0], **kwargs - ) - stdout.write.assert_called_once_with( - "Packages copied: 1/2.\n" - ) - - def test_get_packages_cmd(self, stdout, RepositoryController, **kwargs): - ctrl = RepositoryController.load() - ctrl.load_packages = CallbacksAdapter() - ctrl.load_packages.return_value = gen_package( - name="test1", - filesize=1, - requires=None, - obsoletes=None, - provides=None - ) - self.start_cmd(packages, self.packages_argv) - RepositoryController.load.assert_called_with( - mock.ANY, "deb", "x86_64" - ) - self.check_context( - RepositoryController.load.call_args[0][0], **kwargs - ) - self.assertIn( - "test1; test; 1; test1.pkg; 1;", - stdout.write.call_args_list[3][0][0] - ) - - def test_get_unresolved_cmd(self, stdout, RepositoryController, **kwargs): - ctrl = RepositoryController.load() - ctrl.load_packages = CallbacksAdapter() - ctrl.load_packages.return_value = gen_package( - name="test1", - requires=[gen_relation("test2")] - ) - self.start_cmd(unresolved, self.unresolved_argv) - RepositoryController.load.assert_called_with( - mock.ANY, "deb", "x86_64" - ) - self.check_context( - RepositoryController.load.call_args[0][0], **kwargs - ) - self.assertIn( - "test2; any; -", - stdout.write.call_args_list[3][0][0] - ) diff --git a/packetary/tests/test_command_utils.py b/packetary/tests/test_command_utils.py deleted file mode 100644 index 0022208..0000000 --- a/packetary/tests/test_command_utils.py +++ /dev/null @@ -1,75 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock - -from packetary.cli.commands import utils -from packetary.tests import base - - -class Dummy(object): - pass - - -class TestCommandUtils(base.TestCase): - @mock.patch("packetary.cli.commands.utils.open") - def test_read_lines_from_file(self, open_mock): - open_mock().__enter__.return_value = [ - "line1\n", - " # comment\n", - "line2 \n" - ] - - self.assertEqual( - ["line1", "line2"], - utils.read_lines_from_file("test.txt") - ) - - def test_get_object_attrs(self): - obj = Dummy() - obj.attr_int = 0 - obj.attr_str = "text" - obj.attr_none = None - self.assertEqual( - [0, "text", None], - utils.get_object_attrs(obj, ["attr_int", "attr_str", "attr_none"]) - ) - - def test_get_display_value(self): - self.assertEqual(u"", utils.get_display_value("")) - self.assertEqual(u"-", utils.get_display_value(None)) - self.assertEqual(u"0", utils.get_display_value(0)) - self.assertEqual(u"", utils.get_display_value([])) - self.assertEqual( - u"1, a, None", - utils.get_display_value([1, "a", None]) - ) - self.assertEqual(u"1", utils.get_display_value(1)) - - def test_make_display_attr_getter(self): - obj = Dummy() - obj.attr_int = 0 - obj.attr_str = "text" - obj.attr_none = None - formatter = utils.make_display_attr_getter( - ["attr_int", "attr_str", "attr_none"] - ) - self.assertEqual( - [u"0", u"text", u"-"], - formatter(obj) - ) diff --git a/packetary/tests/test_connections.py b/packetary/tests/test_connections.py deleted file mode 100644 index c80b03d..0000000 --- a/packetary/tests/test_connections.py +++ /dev/null @@ -1,323 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock -import six -import time - -from packetary.library import connections -from packetary.tests import base - - -@mock.patch("packetary.library.connections.logger") -class TestConnectionManager(base.TestCase): - def _check_proxies(self, manager, http_proxy, https_proxy): - for h in manager.opener.handlers: - if isinstance(h, connections.urllib.ProxyHandler): - self.assertEqual( - (http_proxy, https_proxy), - (h.proxies["http"], h.proxies["https"]) - ) - break - else: - self.fail("ProxyHandler should be in list of handlers.") - - def test_set_proxy(self, _): - manager = connections.ConnectionsManager(proxy="http://localhost") - self._check_proxies( - manager, "http://localhost", "http://localhost" - ) - manager = connections.ConnectionsManager( - proxy="http://localhost", secure_proxy="https://localhost") - self._check_proxies( - manager, "http://localhost", "https://localhost" - ) - manager = connections.ConnectionsManager(retries_num=2) - self.assertEqual(2, manager.retries_num) - for h in manager.opener.handlers: - if isinstance(h, connections.RetryHandler): - break - else: - self.fail("RetryHandler should be in list of handlers.") - - @mock.patch("packetary.library.connections.urllib.build_opener") - def test_make_request(self, *_): - manager = connections.ConnectionsManager(retries_num=2) - request = manager.make_request("/test/file", 0) - self.assertIsInstance(request, connections.RetryableRequest) - self.assertEqual("file:///test/file", request.get_full_url()) - self.assertEqual(0, request.offset) - self.assertEqual(2, request.retries_left) - request2 = manager.make_request("http://server/path", 100) - self.assertEqual("http://server/path", request2.get_full_url()) - self.assertEqual(100, request2.offset) - - @mock.patch("packetary.library.connections.urllib.build_opener") - def test_open_stream(self, *_): - manager = connections.ConnectionsManager(retries_num=2) - manager.open_stream("/test/file") - self.assertEqual(1, manager.opener.open.call_count) - args = manager.opener.open.call_args[0] - self.assertIsInstance(args[0], connections.RetryableRequest) - self.assertEqual(2, args[0].retries_left) - - @mock.patch("packetary.library.connections.urllib.build_opener") - def test_retries_on_io_error(self, _, logger): - manager = connections.ConnectionsManager(retries_num=2) - manager.opener.open.side_effect = [ - IOError("I/O error"), - mock.MagicMock() - ] - manager.open_stream("/test/file") - self.assertEqual(2, manager.opener.open.call_count) - logger.exception.assert_called_with( - "Failed to open url - %s: %s. retries left - %d.", - "/test/file", "I/O error", 1 - ) - - manager.opener.open.side_effect = IOError("I/O error") - with self.assertRaises(IOError): - manager.open_stream("/test/file") - logger.exception.assert_called_with( - "Failed to open url - %s: %s. retries left - %d.", - "/test/file", "I/O error", 0 - ) - - @mock.patch.multiple( - "packetary.library.connections.urllib.HTTPHandler", - http_request=mock.DEFAULT, - http_open=mock.DEFAULT - ) - def test_retries_on_50x(self, logger, http_open, http_request): - request = connections.RetryableRequest("http:///localhost/file1.txt") - request.retries_left = 1 - http_request.return_value = request - response_mock = mock.MagicMock(code=501, msg="not found") - response_mock.getcode.return_value = response_mock.code - http_open.return_value = response_mock - manager = connections.ConnectionsManager(retries_num=2) - with self.assertRaises(connections.urlerror.HTTPError) as trapper: - manager.open_stream("http:///localhost/file1.txt") - self.assertEqual(501, trapper.exception.code) - self.assertEqual(2, http_request.call_count) - for retry_num in six.moves.range(1): - logger.error.assert_any_call( - "request failed: %s - %d(%s), retries left - %d.", - mock.ANY, 501, mock.ANY, retry_num - ) - - @mock.patch("packetary.library.connections.urllib.build_opener") - def test_raise_other_errors(self, *_): - manager = connections.ConnectionsManager() - manager.opener.open.side_effect = \ - connections.urlerror.HTTPError("", 500, "", {}, None) - - with self.assertRaises(connections.urlerror.URLError): - manager.open_stream("/test/file") - - self.assertEqual(1, manager.opener.open.call_count) - - @mock.patch("packetary.library.connections.urllib.build_opener") - @mock.patch("packetary.library.connections.time") - def test_progressive_delay_between_request(self, time_mock, *_): - manager = connections.ConnectionsManager( - retries_num=6, retry_interval=1 - ) - manager.opener.open.side_effect = IOError("I/O Error") - - with self.assertRaises(IOError): - manager.open_stream("/test/file") - - self.assertEqual(7, manager.opener.open.call_count) - self.assertEqual( - [1, 1, 2, 3, 4, 5], - [x[0][0] for x in time_mock.sleep.call_args_list] - ) - - @mock.patch("packetary.library.connections.urllib.build_opener") - @mock.patch("packetary.library.connections.ensure_dir_exist") - @mock.patch("packetary.library.connections.os") - def test_retrieve_from_offset(self, os, *_): - manager = connections.ConnectionsManager() - os.stat.return_value = mock.MagicMock(st_size=10) - os.open.return_value = 1 - response = mock.MagicMock() - manager.opener.open.return_value = response - response.read.side_effect = [b"test", b""] - manager.retrieve("/file/src", "/file/dst", size=20) - os.lseek.assert_called_once_with(1, 10, os.SEEK_SET) - os.ftruncate.assert_called_once_with(1, 10) - self.assertEqual(1, os.write.call_count) - os.fsync.assert_called_once_with(1) - os.close.assert_called_once_with(1) - - @mock.patch("packetary.library.connections.urllib.build_opener") - @mock.patch("packetary.library.connections.ensure_dir_exist") - @mock.patch("packetary.library.connections.os") - def test_retrieve_non_existence(self, os, *_): - manager = connections.ConnectionsManager() - os.stat.side_effect = OSError(2, "") - os.open.return_value = 1 - response = mock.MagicMock() - manager.opener.open.return_value = response - response.read.side_effect = [b"test", b""] - manager.retrieve("/file/src", "/file/dst", size=20) - os.lseek.assert_called_once_with(1, 0, os.SEEK_SET) - os.ftruncate.assert_called_once_with(1, 0) - self.assertEqual(1, os.write.call_count) - os.fsync.assert_called_once_with(1) - os.close.assert_called_once_with(1) - - @mock.patch("packetary.library.connections.urllib.build_opener", - new=mock.MagicMock()) - @mock.patch("packetary.library.connections.ensure_dir_exist", - new=mock.MagicMock()) - @mock.patch("packetary.library.connections.os") - def test_retrieve_from_offset_fail(self, os, logger): - manager = connections.ConnectionsManager(retries_num=2) - os.stat.return_value = mock.MagicMock(st_size=10) - os.open.return_value = 1 - response = mock.MagicMock() - manager.opener.open.side_effect = [ - connections.RangeError("error"), response - ] - response.read.side_effect = [b"test", b""] - manager.retrieve("/file/src", "/file/dst", size=20) - logger.warning.assert_called_once_with( - "Failed to resume download, starts from the beginning: %s", - "/file/src" - ) - os.lseek.assert_called_once_with(1, 0, os.SEEK_SET) - os.ftruncate.assert_called_once_with(1, 0) - self.assertEqual(1, os.write.call_count) - os.fsync.assert_called_once_with(1) - os.close.assert_called_once_with(1) - - -@mock.patch("packetary.library.connections.logger") -class TestRetryHandler(base.TestCase): - def setUp(self): - super(TestRetryHandler, self).setUp() - self.handler = connections.RetryHandler() - self.handler.add_parent(mock.MagicMock()) - - def test_start_request(self, logger): - request = mock.MagicMock() - request.offset = 0 - request.get_full_url.return_value = "/file/test" - request = self.handler.http_request(request) - request.start_time <= time.time() - logger.debug.assert_called_with("start request: %s", "/file/test") - request.offset = 1 - request = self.handler.http_request(request) - request.add_header.assert_called_once_with('Range', 'bytes=1-') - - def test_handle_response(self, logger): - request = mock.MagicMock() - request.offset = 0 - request.start_time.__rsub__.return_value = 0.01 - request.get_full_url.return_value = "/file/test" - response = mock.MagicMock() - response.getcode.return_value = 200 - response.msg = "test" - r = self.handler.http_response(request, response) - self.assertIsInstance(r, connections.ResumableResponse) - logger.debug.assert_called_with( - "request completed: %s - %d (%s), duration - %d ms.", - "/file/test", 200, "test", 10 - ) - - def test_handle_partial_response(self, _): - request = mock.MagicMock() - request.offset = 1 - request.get_full_url.return_value = "/file/test" - response = mock.MagicMock() - response.getcode.return_value = 200 - response.msg = "test" - with self.assertRaises(connections.RangeError): - self.handler.http_response(request, response) - response.getcode.return_value = 206 - self.handler.http_response(request, response) - - def test_handle_error(self, logger): - request = mock.MagicMock(retries_left=1, retry_interval=0, offset=0) - request.get_retry_interval.return_value = 0 - request.get_full_url.return_value = "/test" - response_mock = mock.MagicMock(code=500, msg="error") - response_mock.getcode.return_value = response_mock.code - self.handler.http_response(request, response_mock) - logger.error.assert_called_with( - "request failed: %s - %d(%s), retries left - %d.", - "/test", 500, "error", 0 - ) - self.handler.http_response(request, response_mock) - self.handler.parent.open.assert_called_once_with(request) - - @mock.patch( - 'packetary.library.connections.urllib.' - 'HTTPRedirectHandler.redirect_request' - ) - def test_redirect_request(self, redirect_mock, _): - redirect_mock.return_value = connections.urllib.Request( - 'http://localhost/' - ) - req = mock.MagicMock(retries_left=10, retry_interval=5, offset=100) - new_req = self.handler.redirect_request(req, -1, 301, "", {}, "") - self.assertIsInstance(new_req, connections.RetryableRequest) - self.assertEqual(req.retries_left, new_req.retries_left) - self.assertEqual(req.retry_interval, new_req.retry_interval) - self.assertEqual(req.offset, new_req.offset) - redirect_mock.return_value = None - self.assertIsNone( - self.handler.redirect_request(req, -1, 301, "", {}, "") - ) - - -class TestResumeableResponse(base.TestCase): - def setUp(self): - super(TestResumeableResponse, self).setUp() - self.request = mock.MagicMock() - self.opener = mock.MagicMock() - self.stream = mock.MagicMock() - - def test_resume_read(self): - self.request.offset = 0 - response = connections.ResumableResponse( - self.request, - self.stream, - self.opener - ) - self.stream.read.side_effect = [ - b"chunk1", IOError(), b"chunk2", b"" - ] - self.opener.error.return_value = response - data = response.read() - self.assertEqual(b"chunk1chunk2", data) - self.assertEqual(12, self.request.offset) - self.assertEqual(1, self.opener.error.call_count) - - def test_read(self): - self.request.offset = 0 - response = connections.ResumableResponse( - self.request, - six.BytesIO(b"line1\nline2\nline3\n"), - self.opener - ) - self.assertEqual( - b"line1\nline2\nline3\n", response.read() - ) diff --git a/packetary/tests/test_deb_driver.py b/packetary/tests/test_deb_driver.py deleted file mode 100644 index a8ea21b..0000000 --- a/packetary/tests/test_deb_driver.py +++ /dev/null @@ -1,270 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock -import os.path as path -import six - - -from packetary.drivers import deb_driver -from packetary.library.utils import localize_repo_url -from packetary.tests import base -from packetary.tests.stubs.generator import gen_package -from packetary.tests.stubs.generator import gen_repository -from packetary.tests.stubs.helpers import get_compressed - - -PACKAGES = path.join(path.dirname(__file__), "data", "Packages") -RELEASE = path.join(path.dirname(__file__), "data", "Release") - - -class TestDebDriver(base.TestCase): - @classmethod - def setUpClass(cls): - super(TestDebDriver, cls).setUpClass() - cls.driver = deb_driver.DebRepositoryDriver() - - def setUp(self): - self.connection = mock.MagicMock() - - def test_parse_urls(self): - self.assertItemsEqual( - [ - ("http://host", "trusty", "main"), - ("http://host", "trusty", "restricted"), - ], - self.driver.parse_urls( - ["http://host/dists/ trusty main restricted"] - ) - ) - self.assertItemsEqual( - [("http://host", "trusty", "main")], - self.driver.parse_urls( - ["http://host/dists trusty main"] - ) - ) - self.assertItemsEqual( - [("http://host", "trusty", "main")], - self.driver.parse_urls( - ["http://host/ trusty main"] - ) - ) - self.assertItemsEqual( - [ - ("http://host", "trusty", "main"), - ("http://host2", "trusty", "main"), - ], - self.driver.parse_urls( - [ - "http://host/ trusty main", - "http://host2/dists/ trusty main", - ] - ) - ) - - def test_get_repository(self): - repos = [] - with open(RELEASE, "rb") as stream: - self.connection.open_stream.return_value = stream - self.driver.get_repository( - self.connection, - ("http://host", "trusty", "main"), - "x86_64", - repos.append - ) - self.connection.open_stream.assert_called_once_with( - "http://host/dists/trusty/main/binary-amd64/Release" - ) - self.assertEqual(1, len(repos)) - repo = repos[0] - self.assertEqual(("trusty", "main"), repo.name) - self.assertEqual("Ubuntu", repo.origin) - self.assertEqual("x86_64", repo.architecture) - self.assertEqual("http://host/", repo.url) - - def test_get_packages(self): - packages = [] - repo = gen_repository(name=("trusty", "main"), url="http://host/") - with open(PACKAGES, "rb") as s: - self.connection.open_stream.return_value = get_compressed(s) - self.driver.get_packages( - self.connection, - repo, - packages.append - ) - - self.connection.open_stream.assert_called_once_with( - "http://host/dists/trusty/main/binary-amd64/Packages.gz", - ) - self.assertEqual(1, len(packages)) - package = packages[0] - self.assertEqual("test", package.name) - self.assertEqual("1.1.1-1~u14.04+test", package.version) - self.assertEqual(100, package.filesize) - self.assertEqual( - deb_driver.FileChecksum( - '1ae09f80109f40dfbfaf3ba423c8625a', - '402bd18c145ae3b5344edf07f246be159397fd40', - '14d6e308d8699b7f9ba2fe1ef778c0e3' - '8cf295614d308039d687b6b097d50859'), - package.checksum - ) - self.assertEqual( - "pool/main/t/test.deb", package.filename - ) - self.assertTrue(package.mandatory) - self.assertItemsEqual( - [ - 'test-main (any)', - 'test2 (ge 0.8.16~exp9) | tes2-old (any)', - 'test3 (any)' - ], - (str(x) for x in package.requires) - ) - self.assertItemsEqual( - ["file (any)"], - (str(x) for x in package.provides) - ) - self.assertEqual([], package.obsoletes) - - @mock.patch.multiple( - "packetary.drivers.deb_driver", - deb822=mock.DEFAULT, - debfile=mock.DEFAULT, - fcntl=mock.DEFAULT, - gzip=mock.DEFAULT, - utils=mock.DEFAULT, - os=mock.DEFAULT, - open=mock.DEFAULT - ) - def test_rebuild_repository(self, os, debfile, deb822, fcntl, - gzip, utils, open): - repo = gen_repository(name=("trusty", "main"), url="file:///repo") - package = gen_package(name="test", repository=repo) - os.path.join = lambda *x: "/".join(x) - utils.get_path_from_url = lambda x: x[7:] - - files = [ - mock.MagicMock(), # Packages, w - mock.MagicMock(), # Release, a+b - mock.MagicMock(), # Packages, rb - mock.MagicMock(), # Release, rb - mock.MagicMock() # Packages.gz, rb - ] - open.side_effect = files - self.driver.rebuild_repository(repo, [package]) - open.assert_any_call( - "/repo/dists/trusty/main/binary-amd64/Packages", "wb" - ) - gzip.open.assert_called_once_with( - "/repo/dists/trusty/main/binary-amd64/Packages.gz", "wb" - ) - debfile.DebFile.assert_called_once_with("/repo/test.pkg") - - @mock.patch.multiple( - "packetary.drivers.deb_driver", - deb822=mock.DEFAULT, - gzip=mock.DEFAULT, - open=mock.DEFAULT, - os=mock.DEFAULT, - utils=mock.DEFAULT - ) - def test_fork_repository(self, deb822, gzip, open, os, utils): - os.path.sep = "/" - os.path.join = lambda *x: "/".join(x) - utils.get_path_from_url = lambda x: x - utils.localize_repo_url = localize_repo_url - repo = gen_repository( - name=("trusty", "main"), url="http://localhost/test/" - ) - files = [ - mock.MagicMock(), - mock.MagicMock() - ] - open.side_effect = files - new_repo = self.driver.fork_repository(self.connection, repo, "/root") - self.assertEqual(repo.name, new_repo.name) - self.assertEqual(repo.architecture, new_repo.architecture) - self.assertEqual(repo.origin, new_repo.origin) - self.assertEqual("/root/test/", new_repo.url) - utils.ensure_dir_exist.assert_called_once_with(os.path.dirname()) - open.assert_any_call( - "/root/test/dists/trusty/main/binary-amd64/Release", "wb" - ) - open.assert_any_call( - "/root/test/dists/trusty/main/binary-amd64/Packages", "ab" - ) - gzip.open.assert_called_once_with( - "/root/test/dists/trusty/main/binary-amd64/Packages.gz", "ab" - ) - - @mock.patch.multiple( - "packetary.drivers.deb_driver", - fcntl=mock.DEFAULT, - gzip=mock.DEFAULT, - open=mock.DEFAULT, - os=mock.DEFAULT, - utils=mock.DEFAULT - ) - def test_update_suite_index( - self, os, fcntl, gzip, open, utils): - repo = gen_repository(name=("trusty", "main"), url="/repo") - files = [ - mock.MagicMock(), # Release, a+b - mock.MagicMock(), # Packages, rb - mock.MagicMock(), # Release, rb - mock.MagicMock() # Packages.gz, rb - ] - files[0].items.return_value = [ - ("SHA1", "invalid 1 main/binary-amd64/Packages\n"), - ("Architectures", "i386"), - ("Components", "restricted"), - ] - os.path.join = lambda *x: "/".join(x) - open().__enter__.side_effect = files - utils.get_path_from_url.return_value = "/root" - utils.append_token_to_string.side_effect = [ - "amd64 i386", "main restricted" - ] - - utils.get_size_and_checksum_for_files.return_value = ( - ( - "/root/dists/trusty/main/binary-amd64/{0}".format(name), - 10, - (k + "_value" for k in deb_driver._CHECKSUM_METHODS) - ) - for name in deb_driver._REPOSITORY_FILES - ) - self.driver._update_suite_index(repo) - open.assert_any_call("/root/dists/trusty/Release", "a+b") - files[0].seek.assert_called_once_with(0) - files[0].truncate.assert_called_once_with(0) - files[0].write.assert_any_call(six.b("Architectures: amd64 i386\n")) - files[0].write.assert_any_call(six.b("Components: main restricted\n")) - for k in deb_driver._CHECKSUM_METHODS: - files[0].write.assert_any_call(six.b( - '{0}:\n' - ' {1} 10 main/binary-amd64/Packages\n' - ' {1} 10 main/binary-amd64/Release\n' - ' {1} 10 main/binary-amd64/Packages.gz\n' - .format(k, k + "_value") - )) - open.assert_any_call("/root/dists/trusty/Release", "a+b") - print([x.fileno() for x in files]) - fcntl.flock.assert_any_call(files[0].fileno(), fcntl.LOCK_EX) - fcntl.flock.assert_any_call(files[0].fileno(), fcntl.LOCK_UN) diff --git a/packetary/tests/test_executor.py b/packetary/tests/test_executor.py deleted file mode 100644 index 3fff595..0000000 --- a/packetary/tests/test_executor.py +++ /dev/null @@ -1,65 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock -import threading -import time - -from packetary.library import executor -from packetary.tests import base - - -def _raise_value_error(*_): - raise ValueError("error") - - -@mock.patch("packetary.library.executor.logger") -class TestAsynchronousSection(base.TestCase): - def setUp(self): - super(TestAsynchronousSection, self).setUp() - self.results = [] - - def test_isolation(self, _): - section1 = executor.AsynchronousSection() - section2 = executor.AsynchronousSection() - event = threading.Event() - section1.execute(event.wait) - section2.execute(time.sleep, 0) - section2.wait() - event.set() - section1.wait() - - def test_ignore_errors(self, logger): - section = executor.AsynchronousSection(ignore_errors_num=1) - section.execute(_raise_value_error) - section.execute(time.sleep, 0) - section.wait(ignore_errors=True) - logger.exception.assert_called_with( - "error details.", exc_info=mock.ANY - ) - - def test_fail_if_too_many_errors(self, _): - section = executor.AsynchronousSection(size=1, ignore_errors_num=0) - section.execute(_raise_value_error) - time.sleep(0) # switch context - with self.assertRaisesRegexp(RuntimeError, "Too many errors"): - section.execute(time.sleep, 0) - - with self.assertRaisesRegexp( - RuntimeError, "Operations completed with errors"): - section.wait(ignore_errors=False) diff --git a/packetary/tests/test_index.py b/packetary/tests/test_index.py deleted file mode 100644 index 4348bc3..0000000 --- a/packetary/tests/test_index.py +++ /dev/null @@ -1,191 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import six - -from packetary.objects.index import Index - -from packetary import objects -from packetary.tests import base -from packetary.tests.stubs.generator import gen_package -from packetary.tests.stubs.generator import gen_relation - - -class TestIndex(base.TestCase): - def test_add(self): - index = Index() - index.add(gen_package(version=1)) - self.assertIn("package1", index.packages) - self.assertIn(1, index.packages["package1"]) - self.assertIn("obsoletes1", index.obsoletes) - self.assertIn("provides1", index.provides) - - index.add(gen_package(version=2)) - self.assertEqual(1, len(index.packages)) - self.assertIn(1, index.packages["package1"]) - self.assertIn(2, index.packages["package1"]) - self.assertEqual(1, len(index.obsoletes)) - self.assertEqual(1, len(index.provides)) - - def test_find(self): - index = Index() - p1 = gen_package(version=1) - p2 = gen_package(version=2) - index.add(p1) - index.add(p2) - - self.assertIs( - p1, - index.find("package1", objects.VersionRange("eq", 1)) - ) - self.assertIs( - p2, - index.find("package1", objects.VersionRange()) - ) - self.assertIsNone( - index.find("package1", objects.VersionRange("gt", 2)) - ) - - def test_find_all(self): - index = Index() - p11 = gen_package(idx=1, version=1) - p12 = gen_package(idx=1, version=2) - p21 = gen_package(idx=2, version=1) - p22 = gen_package(idx=2, version=2) - index.add(p11) - index.add(p12) - index.add(p21) - index.add(p22) - - self.assertItemsEqual( - [p11, p12], - index.find_all("package1", objects.VersionRange()) - ) - self.assertItemsEqual( - [p21, p22], - index.find_all("package2", objects.VersionRange("le", 2)) - ) - - def test_find_newest_package(self): - index = Index() - p1 = gen_package(idx=1, version=2) - p2 = gen_package(idx=2, version=2) - p2.obsoletes.append( - gen_relation(p1.name, ["lt", p1.version]) - ) - index.add(p1) - index.add(p2) - - self.assertIs( - p1, index.find(p1.name, objects.VersionRange("eq", p1.version)) - ) - self.assertIs( - p2, index.find(p1.name, objects.VersionRange("eq", 1)) - ) - - def test_find_top_down(self): - index = Index() - p1 = gen_package(version=1) - p2 = gen_package(version=2) - index.add(p1) - index.add(p2) - self.assertIs( - p2, - index.find("package1", objects.VersionRange("le", 2)) - ) - self.assertIs( - p1, - index.find("package1", objects.VersionRange("lt", 2)) - ) - self.assertIsNone( - index.find("package1", objects.VersionRange("lt", 1)) - ) - - def test_find_down_up(self): - index = Index() - p1 = gen_package(version=1) - p2 = gen_package(version=2) - index.add(p1) - index.add(p2) - self.assertIs( - p2, - index.find("package1", objects.VersionRange("ge", 2)) - ) - self.assertIs( - p2, - index.find("package1", objects.VersionRange("gt", 1)) - ) - self.assertIsNone( - index.find("package1", objects.VersionRange("gt", 2)) - ) - - def test_find_accurate(self): - index = Index() - p1 = gen_package(version=1) - p2 = gen_package(version=2) - index.add(p1) - index.add(p2) - self.assertIs( - p1, - index.find("package1", objects.VersionRange("eq", 1)) - ) - self.assertIsNone( - index.find("package1", objects.VersionRange("eq", 3)) - ) - - def test_find_obsolete(self): - index = Index() - p1 = gen_package(version=1) - index.add(p1) - - self.assertIs( - p1, index.find("obsoletes1", objects.VersionRange("le", 2)) - ) - self.assertIsNone( - index.find("obsoletes1", objects.VersionRange("gt", 2)) - ) - - def test_find_provides(self): - index = Index() - p1 = gen_package(version=1) - p2 = gen_package(version=2) - index.add(p1) - index.add(p2) - - self.assertIs( - p2, index.find("provides1", objects.VersionRange("ge", 2)) - ) - self.assertIsNone( - index.find("provides1", objects.VersionRange("gt", 2)) - ) - - def test_len(self): - index = Index() - for i in six.moves.range(3): - index.add(gen_package(idx=i + 1)) - self.assertEqual(3, len(index)) - - for i in six.moves.range(3): - index.add(gen_package(idx=i + 1, version=2)) - self.assertEqual(6, len(index)) - self.assertEqual(3, len(index.packages)) - - for i in six.moves.range(3): - index.add(gen_package(idx=i + 1, version=2)) - self.assertEqual(6, len(index)) - self.assertEqual(3, len(index.packages)) diff --git a/packetary/tests/test_library_utils.py b/packetary/tests/test_library_utils.py deleted file mode 100644 index b3a8d0b..0000000 --- a/packetary/tests/test_library_utils.py +++ /dev/null @@ -1,112 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock - -from packetary.library import utils -from packetary.tests import base - - -class TestLibraryUtils(base.TestCase): - def test_append_token_to_string(self): - self.assertEqual( - "v1 v2 v3", - utils.append_token_to_string("v2 v3", "v1") - ) - self.assertEqual( - "v1", - utils.append_token_to_string("", "v1") - ) - self.assertEqual( - "v1 v2 v3 v4", - utils.append_token_to_string('v1\tv2 v3', "v4") - ) - self.assertEqual( - "v1 v2 v3", - utils.append_token_to_string('v1 v2 v3', "v1") - ) - - def test_composite_writer(self): - fds = [ - mock.MagicMock(), - mock.MagicMock() - ] - writer = utils.composite_writer(*fds) - writer(u"text1") - writer(b"text2") - for fd in fds: - fd.write.assert_any_call(b"text1") - fd.write.assert_any_call(b"text2") - - @mock.patch.multiple( - "packetary.library.utils", - os=mock.DEFAULT, - open=mock.DEFAULT - ) - def test_get_size_and_checksum_for_files(self, os, open): - files = [ - "/file1.txt", - "/file2.txt" - ] - os.fstat.side_effect = [ - mock.MagicMock(st_size=1), - mock.MagicMock(st_size=2) - ] - r = list(utils.get_size_and_checksum_for_files( - files, mock.MagicMock(side_effect=["1", "2"]) - )) - self.assertEqual( - [("/file1.txt", 1, "1"), ("/file2.txt", 2, "2")], - r - ) - - def test_get_path_from_url(self): - self.assertEqual( - "/a/f.txt", - utils.get_path_from_url("/a/f.txt") - ) - - self.assertEqual( - "/a/f.txt", - utils.get_path_from_url("file:///a/f.txt?size=1") - ) - - with self.assertRaises(ValueError): - utils.get_path_from_url("http:///a/f.txt") - - self.assertEqual( - "/f.txt", - utils.get_path_from_url("http://host/f.txt", False) - ) - - @mock.patch("packetary.library.utils.os") - def test_ensure_dir_exist(self, os): - os.makedirs.side_effect = [ - True, - OSError(utils.errno.EEXIST, ""), - OSError(utils.errno.EACCES, ""), - ValueError() - ] - utils.ensure_dir_exist("/nonexisted") - os.makedirs.assert_called_with("/nonexisted") - utils.ensure_dir_exist("/existed") - os.makedirs.assert_called_with("/existed") - with self.assertRaises(OSError): - utils.ensure_dir_exist("/private") - with self.assertRaises(ValueError): - utils.ensure_dir_exist(1) diff --git a/packetary/tests/test_objects.py b/packetary/tests/test_objects.py deleted file mode 100644 index 39dbc6e..0000000 --- a/packetary/tests/test_objects.py +++ /dev/null @@ -1,234 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import copy -import six - -from packetary.objects import PackageRelation -from packetary.objects import PackageVersion -from packetary.objects import VersionRange - - -from packetary.tests import base -from packetary.tests.stubs import generator - - -class TestObjectBase(base.TestCase): - def check_copy(self, origin): - clone = copy.copy(origin) - self.assertIsNot(origin, clone) - self.assertEqual(origin, clone) - origin_name = origin.name - origin.name += "1" - self.assertEqual( - origin_name, - clone.name - ) - - def check_ordering(self, *args): - for i in six.moves.range(len(args) - 1, 1, -1): - self.assertLess(args[i - 1], args[i]) - self.assertGreater(args[i], args[i - 1]) - - def check_equal(self, o1, o11, o2): - self.assertEqual(o1, o11) - self.assertEqual(o11, o1) - self.assertNotEqual(o1, o2) - self.assertNotEqual(o2, o1) - self.assertNotEqual(o1, None) - - def check_hashable(self, o1, o2): - d = dict() - d[o1] = o2 - d[o2] = o1 - - self.assertIs(o2, d[o1]) - self.assertIs(o1, d[o2]) - - -class TestPackageObject(TestObjectBase): - def test_copy(self): - self.check_copy(generator.gen_package(name="test1")) - - def test_ordering(self): - self.check_ordering([ - generator.gen_package(name="test1", version=1), - generator.gen_package(name="test1", version=2), - generator.gen_package(name="test2", version=1), - generator.gen_package(name="test2", version=2) - ]) - - def test_equal(self): - self.check_equal( - generator.gen_package(name="test1", version=1), - generator.gen_package(name="test1", version=1), - generator.gen_package(name="test2", version=1) - ) - - def test_hashable(self): - self.check_hashable( - generator.gen_package(name="test1", version=1), - generator.gen_package(name="test2", version=1), - ) - self.check_hashable( - generator.gen_package(name="test1", version=1), - generator.gen_package(name="test1", version=2), - ) - - -class TestRepositoryObject(base.TestCase): - def test_copy(self): - origin = generator.gen_repository() - clone = copy.copy(origin) - self.assertEqual(clone.name, origin.name) - self.assertEqual(clone.architecture, origin.architecture) - - def test_str(self): - self.assertEqual( - "a.b", - str(generator.gen_repository(name=("a", "b"))) - ) - self.assertEqual( - "/a/b/", - str(generator.gen_repository(name="", url="/a/b/")) - ) - self.assertEqual( - "a", - str(generator.gen_repository(name="a", url="/a/b/")) - ) - - -class TestRelationObject(TestObjectBase): - def test_equal(self): - self.check_equal( - generator.gen_relation(name="test1"), - generator.gen_relation(name="test1"), - generator.gen_relation(name="test2") - ) - - def test_hashable(self): - self.check_hashable( - generator.gen_relation(name="test1"), - generator.gen_relation(name="test1", version=["le", 1]) - ) - - def test_from_args(self): - r = PackageRelation.from_args( - ("test", "le", 2), ("test2",), ("test3",) - ) - self.assertEqual("test", r.name) - self.assertEqual("le", r.version.op) - self.assertEqual(2, r.version.edge) - self.assertEqual("test2", r.alternative.name) - self.assertEqual(VersionRange(), r.alternative.version) - self.assertEqual("test3", r.alternative.alternative.name) - self.assertEqual(VersionRange(), r.alternative.alternative.version) - self.assertIsNone(r.alternative.alternative.alternative) - - def test_iter(self): - it = iter(PackageRelation.from_args( - ("test", "le", 2), ("test2", "ge", 3)) - ) - self.assertEqual("test", next(it).name) - self.assertEqual("test2", next(it).name) - with self.assertRaises(StopIteration): - next(it) - - -class TestVersionRange(TestObjectBase): - def test_equal(self): - self.check_equal( - VersionRange("eq", 1), - VersionRange("eq", 1), - VersionRange("le", 1) - ) - - def test_hashable(self): - self.check_hashable( - VersionRange(op="le"), - VersionRange(op="le", edge=3) - ) - - def __check_intersection(self, assertion, cases): - for data in cases: - v1 = VersionRange(*data[0]) - v2 = VersionRange(*data[1]) - assertion( - v1.has_intersection(v2), msg="%s and %s" % (v1, v2) - ) - assertion( - v2.has_intersection(v1), msg="%s and %s" % (v2, v1) - ) - - def test_have_intersection(self): - cases = [ - (("eq", 2), ("eq", 2)), - (("eq", 2), ("lt", 3)), - (("eq", 2), ("gt", 1)), - (("lt", 2), ("gt", 1)), - (("lt", 2), ("lt", 3)), - (("lt", 2), ("lt", 2)), - (("lt", 2), ("le", 2)), - (("gt", 2), ("gt", 1)), - (("gt", 2), ("lt", 3)), - (("gt", 2), ("ge", 2)), - (("gt", 2), ("gt", 2)), - (("ge", 2), ("le", 2)), - ((None, None), ("eq", 2)), - ] - self.__check_intersection(self.assertTrue, cases) - - def test_does_not_have_intersection(self): - cases = [ - (("eq", 2), ("eq", 1)), - (("eq", 2), ("lt", 2)), - (("eq", 2), ("gt", 2)), - (("eq", 2), ("gt", 3)), - (("eq", 2), ("lt", 1)), - (("lt", 2), ("ge", 2)), - (("lt", 2), ("gt", 3)), - (("gt", 2), ("le", 2)), - (("gt", 2), ("lt", 1)), - ] - self.__check_intersection(self.assertFalse, cases) - - def test_intersection_is_typesafe(self): - with self.assertRaises(TypeError): - VersionRange("eq", 1).has_intersection(("eq", 1)) - - -class TestPackageVersion(base.TestCase): - def test_get_from_string(self): - ver = PackageVersion.from_string("1.0-22") - self.assertEqual(0, ver.epoch) - self.assertEqual(('1', '0'), ver.version) - self.assertEqual(('22',), ver.release) - - ver2 = PackageVersion.from_string("1-11.0-2") - self.assertEqual(1, ver2.epoch) - self.assertEqual(('11', '0'), ver2.version) - self.assertEqual(('2',), ver2.release) - - def test_compare(self): - ver1 = PackageVersion.from_string("6.3-31.5") - ver2 = PackageVersion.from_string("13.9-16.12") - self.assertLess(ver1, ver2) - self.assertGreater(ver2, ver1) - self.assertEqual(ver1, ver1) - self.assertLess(ver1, "6.3-40") - self.assertGreater(ver1, "6.3-31.4a") diff --git a/packetary/tests/test_packages_tree.py b/packetary/tests/test_packages_tree.py deleted file mode 100644 index f7e936e..0000000 --- a/packetary/tests/test_packages_tree.py +++ /dev/null @@ -1,135 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import warnings - -from packetary.objects import Index -from packetary.objects import PackagesTree -from packetary.tests import base -from packetary.tests.stubs import generator - - -class TestPackagesTree(base.TestCase): - def setUp(self): - super(TestPackagesTree, self).setUp() - - def test_get_unresolved_dependencies(self): - ptree = PackagesTree() - ptree.add(generator.gen_package( - 1, requires=[generator.gen_relation("unresolved")])) - ptree.add(generator.gen_package(2, requires=None)) - ptree.add(generator.gen_package( - 3, requires=[generator.gen_relation("package1")] - )) - ptree.add(generator.gen_package( - 4, - requires=[generator.gen_relation("loop")], - obsoletes=[generator.gen_relation("loop", ["le", 1])] - )) - - unresolved = ptree.get_unresolved_dependencies() - self.assertItemsEqual( - ["loop", "unresolved"], - (x.name for x in unresolved) - ) - - def test_get_unresolved_dependencies_with_main(self): - ptree = PackagesTree() - ptree.add(generator.gen_package( - 1, requires=[generator.gen_relation("unresolved")])) - ptree.add(generator.gen_package(2, requires=None)) - ptree.add(generator.gen_package( - 3, requires=[generator.gen_relation("package1")] - )) - ptree.add(generator.gen_package( - 4, - requires=[generator.gen_relation("package5")] - )) - main = Index() - main.add(generator.gen_package(5, requires=[ - generator.gen_relation("package6") - ])) - - unresolved = ptree.get_unresolved_dependencies(main) - self.assertItemsEqual( - ["unresolved"], - (x.name for x in unresolved) - ) - - def test_get_minimal_subset_with_master(self): - ptree = PackagesTree() - ptree.add(generator.gen_package(1, requires=None)) - ptree.add(generator.gen_package(2, requires=None)) - ptree.add(generator.gen_package(3, requires=None)) - ptree.add(generator.gen_package( - 4, requires=[generator.gen_relation("package1")] - )) - - master = Index() - master.add(generator.gen_package(1, requires=None)) - master.add(generator.gen_package( - 5, - requires=[generator.gen_relation( - "package10", - alternative=generator.gen_relation("package4") - )] - )) - - unresolved = set([generator.gen_relation("package3")]) - resolved = ptree.get_minimal_subset(master, unresolved) - self.assertItemsEqual( - ["package3", "package4"], - (x.name for x in resolved) - ) - - def test_get_minimal_subset_without_master(self): - ptree = PackagesTree() - ptree.add(generator.gen_package(1, requires=None)) - ptree.add(generator.gen_package(2, requires=None)) - ptree.add(generator.gen_package( - 3, requires=[generator.gen_relation("package1")] - )) - unresolved = set([generator.gen_relation("package3")]) - resolved = ptree.get_minimal_subset(None, unresolved) - self.assertItemsEqual( - ["package3", "package1"], - (x.name for x in resolved) - ) - - def test_mandatory_packages_always_included(self): - ptree = PackagesTree() - ptree.add(generator.gen_package(1, requires=None, mandatory=True)) - ptree.add(generator.gen_package(2, requires=None)) - ptree.add(generator.gen_package(3, requires=None)) - unresolved = set([generator.gen_relation("package3")]) - resolved = ptree.get_minimal_subset(None, unresolved) - self.assertItemsEqual( - ["package3", "package1"], - (x.name for x in resolved) - ) - - def test_warning_if_unresolved(self): - ptree = PackagesTree() - ptree.add(generator.gen_package( - 1, requires=None)) - - with warnings.catch_warnings(record=True) as log: - ptree.get_minimal_subset( - None, [generator.gen_relation("package2")] - ) - self.assertIn("package2", str(log[0])) diff --git a/packetary/tests/test_repository_api.py b/packetary/tests/test_repository_api.py deleted file mode 100644 index 7ea5599..0000000 --- a/packetary/tests/test_repository_api.py +++ /dev/null @@ -1,257 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock - -from packetary.api import Configuration -from packetary.api import Context -from packetary.api import RepositoryApi -from packetary.tests import base -from packetary.tests.stubs import generator -from packetary.tests.stubs.helpers import CallbacksAdapter - - -class TestRepositoryApi(base.TestCase): - def test_get_packages_as_is(self): - controller = CallbacksAdapter() - pkg = generator.gen_package(name="test") - controller.load_packages.side_effect = [ - pkg - ] - api = RepositoryApi(controller) - packages = api.get_packages("file:///repo1") - self.assertEqual(1, len(packages)) - package = packages.pop() - self.assertIs(pkg, package) - - def test_get_packages_with_depends_resolving(self): - controller = CallbacksAdapter() - controller.load_packages.side_effect = [ - [ - generator.gen_package(idx=1, requires=None), - generator.gen_package( - idx=2, requires=[generator.gen_relation("package1")] - ), - generator.gen_package( - idx=3, requires=[generator.gen_relation("package1")] - ), - generator.gen_package(idx=4, requires=None), - generator.gen_package(idx=5, requires=None), - ], - generator.gen_package( - idx=6, requires=[generator.gen_relation("package2")] - ), - ] - - api = RepositoryApi(controller) - packages = api.get_packages([ - "file:///repo1", "file:///repo2" - ], - "file:///repo3", ["package4"] - ) - - self.assertEqual(3, len(packages)) - self.assertItemsEqual( - ["package1", "package4", "package2"], - (x.name for x in packages) - ) - controller.load_repositories.assert_any_call( - ["file:///repo1", "file:///repo2"] - ) - controller.load_repositories.assert_any_call( - "file:///repo3" - ) - - def test_clone_repositories_as_is(self): - controller = CallbacksAdapter() - repo = generator.gen_repository(name="repo1") - packages = [ - generator.gen_package(name="test1", repository=repo), - generator.gen_package(name="test2", repository=repo) - ] - mirror = generator.gen_repository(name="mirror") - controller.load_repositories.return_value = repo - controller.load_packages.return_value = packages - controller.clone_repositories.return_value = {repo: mirror} - controller.copy_packages.return_value = [0, 1] - api = RepositoryApi(controller) - stats = api.clone_repositories( - ["file:///repo1"], "/mirror", keep_existing=True - ) - self.assertEqual(2, stats.total) - self.assertEqual(1, stats.copied) - controller.copy_packages.assert_called_once_with( - mirror, set(packages), True - ) - - def test_copy_minimal_subset_of_repository(self): - controller = CallbacksAdapter() - repo1 = generator.gen_repository(name="repo1") - repo2 = generator.gen_repository(name="repo2") - repo3 = generator.gen_repository(name="repo3") - mirror1 = generator.gen_repository(name="mirror1") - mirror2 = generator.gen_repository(name="mirror2") - pkg_group1 = [ - generator.gen_package( - idx=1, requires=None, repository=repo1 - ), - generator.gen_package( - idx=1, version=2, requires=None, repository=repo1 - ), - generator.gen_package( - idx=2, requires=None, repository=repo1 - ) - ] - pkg_group2 = [ - generator.gen_package( - idx=4, - requires=[generator.gen_relation("package1")], - repository=repo2, - mandatory=True, - ) - ] - pkg_group3 = [ - generator.gen_package( - idx=3, requires=None, repository=repo1 - ) - ] - controller.load_repositories.side_effect = [[repo1, repo2], repo3] - controller.load_packages.side_effect = [ - pkg_group1 + pkg_group2 + pkg_group3, - generator.gen_package( - idx=6, - repository=repo3, - requires=[generator.gen_relation("package2")] - ) - ] - controller.clone_repositories.return_value = { - repo1: mirror1, repo2: mirror2 - } - controller.copy_packages.return_value = 1 - api = RepositoryApi(controller) - api.clone_repositories( - ["file:///repo1", "file:///repo2"], "/mirror", - ["file:///repo3"], - keep_existing=True - ) - controller.copy_packages.assert_any_call( - mirror1, set(pkg_group1), True - ) - controller.copy_packages.assert_any_call( - mirror2, set(pkg_group2), True - ) - self.assertEqual(2, controller.copy_packages.call_count) - - def test_get_unresolved(self): - controller = CallbacksAdapter() - pkg = generator.gen_package( - name="test", requires=[generator.gen_relation("test2")] - ) - controller.load_packages.side_effect = [ - pkg - ] - api = RepositoryApi(controller) - r = api.get_unresolved_dependencies("file:///repo1") - controller.load_repositories.assert_called_once_with("file:///repo1") - self.assertItemsEqual( - ["test2"], - (x.name for x in r) - ) - - def test_get_unresolved_with_main(self): - controller = CallbacksAdapter() - pkg1 = generator.gen_package( - name="test1", requires=[ - generator.gen_relation("test2"), - generator.gen_relation("test3") - ] - ) - pkg2 = generator.gen_package( - name="test2", requires=[generator.gen_relation("test4")] - ) - controller.load_packages.side_effect = [ - pkg1, pkg2 - ] - api = RepositoryApi(controller) - r = api.get_unresolved_dependencies("file:///repo1", "file:///repo2") - controller.load_repositories.assert_any_call("file:///repo1") - controller.load_repositories.assert_any_call("file:///repo2") - self.assertItemsEqual( - ["test3"], - (x.name for x in r) - ) - - def test_parse_requirements(self): - requirements = RepositoryApi._parse_requirements( - ["p1 le 2 | p2 | p3 ge 2"] - ) - - expected = generator.gen_relation( - "p1", - ["le", '2'], - generator.gen_relation( - "p2", - None, - generator.gen_relation( - "p3", - ["ge", '2'] - ) - ) - ) - self.assertEqual(1, len(requirements)) - self.assertEqual( - list(expected), - list(requirements.pop()) - ) - - -class TestContext(base.TestCase): - @classmethod - def setUpClass(cls): - cls.config = Configuration( - threads_num=2, - ignore_errors_num=3, - retries_num=5, - retry_interval=5, - http_proxy="http://localhost", - https_proxy="https://localhost" - ) - - @mock.patch("packetary.api.ConnectionsManager") - def test_initialise_connection_manager(self, conn_manager): - context = Context(self.config) - conn_manager.assert_called_once_with( - proxy="http://localhost", - secure_proxy="https://localhost", - retries_num=5, - retry_interval=5 - ) - - self.assertIs( - conn_manager(), - context.connection - ) - - @mock.patch("packetary.api.AsynchronousSection") - def test_asynchronous_section(self, async_section): - context = Context(self.config) - s = context.async_section() - async_section.assert_called_with(2, 3) - self.assertIs(s, async_section()) - context.async_section(0) - async_section.assert_called_with(2, 0) diff --git a/packetary/tests/test_repository_contoller.py b/packetary/tests/test_repository_contoller.py deleted file mode 100644 index 9eb6cba..0000000 --- a/packetary/tests/test_repository_contoller.py +++ /dev/null @@ -1,145 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import copy -import mock -import six - -from packetary.controllers import RepositoryController -from packetary.tests import base -from packetary.tests.stubs.executor import Executor -from packetary.tests.stubs.generator import gen_package -from packetary.tests.stubs.generator import gen_repository -from packetary.tests.stubs.helpers import CallbacksAdapter - - -class TestRepositoryController(base.TestCase): - def setUp(self): - self.driver = mock.MagicMock() - self.context = mock.MagicMock() - self.context.async_section.return_value = Executor() - self.ctrl = RepositoryController(self.context, self.driver, "x86_64") - - def test_load_fail_if_unknown_driver(self): - with self.assertRaisesRegexp(NotImplementedError, "unknown_driver"): - RepositoryController.load( - self.context, - "unknown_driver", - "x86_64" - ) - - @mock.patch("packetary.controllers.repository.stevedore") - def test_load_driver(self, stevedore): - stevedore.ExtensionManager.return_value = { - "test": mock.MagicMock(obj=self.driver) - } - RepositoryController._drivers = None - controller = RepositoryController.load(self.context, "test", "x86_64") - self.assertIs(self.driver, controller.driver) - - def test_load_repositories(self): - self.driver.parse_urls.return_value = ["test1"] - consumer = mock.MagicMock() - self.ctrl.load_repositories("file:///test1", consumer) - self.driver.parse_urls.assert_called_once_with(["file:///test1"]) - self.driver.get_repository.assert_called_once_with( - self.context.connection, "test1", "x86_64", consumer - ) - for url in [six.u("file:///test1"), ["file:///test1"]]: - self.driver.reset_mock() - self.ctrl.load_repositories(url, consumer) - if not isinstance(url, list): - url = [url] - self.driver.parse_urls.assert_called_once_with(url) - - def test_load_packages(self): - repo = mock.MagicMock() - consumer = mock.MagicMock() - self.ctrl.load_packages([repo], consumer) - self.driver.get_packages.assert_called_once_with( - self.context.connection, repo, consumer - ) - - @mock.patch("packetary.controllers.repository.os") - def test_assign_packages(self, os): - repo = gen_repository(url="/test/repo") - packages = [ - gen_package(name="test1", repository=repo), - gen_package(name="test2", repository=repo) - ] - existed_packages = [ - gen_package(name="test3", repository=repo), - gen_package(name="test2", repository=repo) - ] - - os.path.join = lambda *x: "/".join(x) - self.driver.get_packages = CallbacksAdapter() - self.driver.get_packages.return_value = existed_packages - self.ctrl.assign_packages(repo, packages, True) - os.remove.assert_not_called() - all_packages = set(packages + existed_packages) - self.driver.rebuild_repository.assert_called_once_with( - repo, all_packages - ) - self.driver.rebuild_repository.reset_mock() - self.ctrl.assign_packages(repo, packages, False) - self.driver.rebuild_repository.assert_called_once_with( - repo, set(packages) - ) - os.remove.assert_called_once_with("/test/repo/test3.pkg") - - def test_copy_packages(self): - repo = gen_repository(url="file:///repo/") - packages = [ - gen_package(name="test1", repository=repo, filesize=10), - gen_package(name="test2", repository=repo, filesize=-1) - ] - target = gen_repository(url="/test/repo") - self.context.connection.retrieve.side_effect = [0, 10] - observer = mock.MagicMock() - self.ctrl.copy_packages(target, packages, True, observer) - observer.assert_has_calls([mock.call(0), mock.call(10)]) - self.context.connection.retrieve.assert_any_call( - "file:///repo/test1.pkg", - "/test/repo/test1.pkg", - size=10 - ) - self.context.connection.retrieve.assert_any_call( - "file:///repo/test2.pkg", - "/test/repo/test2.pkg", - size=-1 - ) - self.driver.rebuild_repository.assert_called_once_with( - target, set(packages) - ) - - @mock.patch("packetary.controllers.repository.os") - def test_clone_repository(self, os): - os.path.abspath.return_value = "/root/repo" - repos = [ - gen_repository(name="test1"), - gen_repository(name="test2") - ] - clones = [copy.copy(x) for x in repos] - self.driver.fork_repository.side_effect = clones - mirrors = self.ctrl.clone_repositories(repos, "./repo") - for r in repos: - self.driver.fork_repository.assert_any_call( - self.context.connection, r, "/root/repo", False, False - ) - self.assertEqual(mirrors, dict(zip(repos, clones))) diff --git a/packetary/tests/test_rpm_driver.py b/packetary/tests/test_rpm_driver.py deleted file mode 100644 index 47dc6fb..0000000 --- a/packetary/tests/test_rpm_driver.py +++ /dev/null @@ -1,211 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import mock -import os.path as path -import sys - -import six - -from packetary.library.utils import localize_repo_url -from packetary.objects import FileChecksum -from packetary.tests import base -from packetary.tests.stubs.generator import gen_repository -from packetary.tests.stubs.helpers import get_compressed - - -REPOMD = path.join(path.dirname(__file__), "data", "repomd.xml") - -REPOMD2 = path.join(path.dirname(__file__), "data", "repomd2.xml") - -PRIMARY_DB = path.join(path.dirname(__file__), "data", "primary.xml") - -GROUPS_DB = path.join(path.dirname(__file__), "data", "groups.xml") - - -class TestRpmDriver(base.TestCase): - @classmethod - def setUpClass(cls): - cls.createrepo = sys.modules["createrepo"] = mock.MagicMock() - # import driver class after patching sys.modules - from packetary.drivers import rpm_driver - - super(TestRpmDriver, cls).setUpClass() - cls.driver = rpm_driver.RpmRepositoryDriver() - - def setUp(self): - self.createrepo.reset_mock() - self.connection = mock.MagicMock() - - def test_parse_urls(self): - self.assertItemsEqual( - [ - "http://host/centos/os", - "http://host/centos/updates" - ], - self.driver.parse_urls([ - "http://host/centos/os", - "http://host/centos/updates/", - ]) - ) - - def test_get_repository(self): - repos = [] - - self.driver.get_repository( - self.connection, - "http://host/centos/os/x86_64", - "x86_64", - repos.append - ) - - self.assertEqual(1, len(repos)) - repo = repos[0] - self.assertEqual("/centos/os/x86_64", repo.name) - self.assertEqual("", repo.origin) - self.assertEqual("x86_64", repo.architecture) - self.assertEqual("http://host/centos/os/x86_64/", repo.url) - - def test_get_packages(self): - streams = [] - for conv, fname in zip( - (lambda x: six.BytesIO(x.read()), - get_compressed, get_compressed), - (REPOMD, GROUPS_DB, PRIMARY_DB) - ): - with open(fname, "rb") as s: - streams.append(conv(s)) - - packages = [] - self.connection.open_stream.side_effect = streams - self.driver.get_packages( - self.connection, - gen_repository("test", url="http://host/centos/os/x86_64/"), - packages.append - ) - self.connection.open_stream.assert_any_call( - "http://host/centos/os/x86_64/repodata/repomd.xml" - ) - self.connection.open_stream.assert_any_call( - "http://host/centos/os/x86_64/repodata/groups.xml.gz" - ) - self.connection.open_stream.assert_any_call( - "http://host/centos/os/x86_64/repodata/primary.xml.gz" - ) - self.assertEqual(2, len(packages)) - package = packages[0] - self.assertEqual("test1", package.name) - self.assertEqual("1.1.1.1-1.el7", package.version) - self.assertEqual(100, package.filesize) - self.assertEqual( - FileChecksum( - None, - None, - 'e8ed9e0612e813491ed5e7c10502a39e' - '43ec665afd1321541dea211202707a65'), - package.checksum - ) - self.assertEqual( - "Packages/test1.rpm", package.filename - ) - self.assertItemsEqual( - ['test2 (eq 0-1.1.1.1-1.el7)'], - (str(x) for x in package.requires) - ) - self.assertItemsEqual( - ["file (any)"], - (str(x) for x in package.provides) - ) - self.assertItemsEqual( - ["test-old (any)"], - (str(x) for x in package.obsoletes) - ) - self.assertTrue(package.mandatory) - self.assertFalse(packages[1].mandatory) - - def test_get_packages_if_group_not_gzipped(self): - streams = [] - for conv, fname in zip( - (lambda x: six.BytesIO(x.read()), - lambda x: six.BytesIO(x.read()), - get_compressed), - (REPOMD2, GROUPS_DB, PRIMARY_DB) - ): - with open(fname, "rb") as s: - streams.append(conv(s)) - - packages = [] - self.connection.open_stream.side_effect = streams - self.driver.get_packages( - self.connection, - gen_repository("test", url="http://host/centos/os/x86_64/"), - packages.append - ) - self.connection.open_stream.assert_any_call( - "http://host/centos/os/x86_64/repodata/groups.xml" - ) - self.assertEqual(2, len(packages)) - package = packages[0] - self.assertTrue(package.mandatory) - - @mock.patch("packetary.drivers.rpm_driver.shutil") - def test_rebuild_repository(self, shutil): - self.createrepo.MDError = ValueError - self.createrepo.MetaDataGenerator().doFinalMove.side_effect = [ - None, self.createrepo.MDError() - ] - repo = gen_repository("test", url="file:///repo/os/x86_64") - self.createrepo.MetaDataConfig().outputdir = "/repo/os/x86_64" - self.createrepo.MetaDataConfig().tempdir = "tmp" - - self.driver.rebuild_repository(repo, set()) - - self.assertEqual( - "/repo/os/x86_64", - self.createrepo.MetaDataConfig().directory - ) - self.assertTrue(self.createrepo.MetaDataConfig().update) - self.createrepo.MetaDataGenerator()\ - .doPkgMetadata.assert_called_once_with() - self.createrepo.MetaDataGenerator()\ - .doRepoMetadata.assert_called_once_with() - self.createrepo.MetaDataGenerator()\ - .doFinalMove.assert_called_once_with() - - with self.assertRaises(RuntimeError): - self.driver.rebuild_repository(repo, set()) - shutil.rmtree.assert_called_once_with( - "/repo/os/x86_64/tmp", ignore_errors=True - ) - - @mock.patch("packetary.drivers.rpm_driver.utils") - def test_fork_repository(self, utils): - repo = gen_repository("os", url="http://localhost/os/x86_64/") - utils.localize_repo_url = localize_repo_url - new_repo = self.driver.fork_repository( - self.connection, - repo, - "/repo" - ) - - utils.ensure_dir_exist.assert_called_once_with("/repo/os/x86_64/") - self.assertEqual(repo.name, new_repo.name) - self.assertEqual(repo.architecture, new_repo.architecture) - self.assertEqual("/repo/os/x86_64/", new_repo.url) - self.createrepo.MetaDataGenerator()\ - .doFinalMove.assert_called_once_with() diff --git a/packetary/tests/test_streams.py b/packetary/tests/test_streams.py deleted file mode 100644 index 68b4a8f..0000000 --- a/packetary/tests/test_streams.py +++ /dev/null @@ -1,99 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import six - -from packetary.library import streams -from packetary.tests import base -from packetary.tests.stubs.helpers import get_compressed - - -class TestBufferedStream(base.TestCase): - def setUp(self): - super(TestBufferedStream, self).setUp() - self.stream = streams.StreamWrapper( - six.BytesIO(b"line1\nline2\nline3\n") - ) - - def test_read(self): - self.stream.CHUNK_SIZE = 10 - chunk = self.stream.read(5) - self.assertEqual(b"line1", chunk) - self.assertEqual(b"\nline", self.stream.unread_tail) - chunk = self.stream.read(1024) - self.assertEqual(b"\nline2\nline3\n", chunk) - self.assertEqual(b"", self.stream.unread_tail) - - def test_readline(self): - self.stream.CHUNK_SIZE = 12 - chunk = self.stream.readline() - self.assertEqual(b"line1\n", chunk) - self.assertEqual(b"line2\n", self.stream.unread_tail) - lines = list(self.stream.readlines()) - self.assertEqual([b"line2\n", b"line3\n"], lines) - self.assertEqual(b"", self.stream.unread_tail) - - def test_readlines(self): - self.stream.CHUNK_SIZE = 12 - lines = list(self.stream.readlines()) - self.assertEqual( - [b"line1\n", b"line2\n", b"line3\n"], - lines) - - -class TestGzipDecompress(base.TestCase): - @classmethod - def setUpClass(cls): - cls.compressed = get_compressed(six.BytesIO(b"line1\nline2\nline3\n")) - - def setUp(self): - super(TestGzipDecompress, self).setUp() - self.compressed.reset() - self.stream = streams.GzipDecompress(self.compressed) - - def test_read(self): - chunk = self.stream.read(5) - self.assertEqual(b"line1", chunk) - self.assertEqual(b"\nline2\nline3\n", self.stream.unread_tail) - chunk = self.stream.read(1024) - self.assertEqual(b"\nline2\nline3\n", chunk) - self.assertEqual(b"", self.stream.unread_tail) - - def test_readline(self): - self.stream.CHUNK_SIZE = 12 - chunk = self.stream.readline() - self.assertEqual(b"line1\n", chunk) - self.assertEqual(b"line2\nl", self.stream.unread_tail) - lines = list(self.stream.readlines()) - self.assertEqual([b"line2\n", b"line3\n"], lines) - self.assertEqual(b"", self.stream.unread_tail) - - def test_readlines(self): - self.stream.CHUNK_SIZE = 12 - lines = list(self.stream.readlines()) - self.assertEqual( - [b"line1\n", b"line2\n", b"line3\n"], - lines) - - def test_handle_case_if_not_enough_data_to_decompress(self): - self.stream.CHUNK_SIZE = 1 - chunk = self.stream.read() - self.assertEqual( - b"line1\nline2\nline3\n", - chunk - ) diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index a369282..0000000 --- a/requirements.txt +++ /dev/null @@ -1,14 +0,0 @@ -# The order of packages is significant, because pip processes them in the order -# of appearance. Changing the order has an impact on the overall integration -# process, which may cause wedges in the gate later. - -pbr>=0.8 -Babel>=1.3 -cliff>=1.7.0 -eventlet>=0.15 -bintrees>=2.0.2 -chardet>=2.0.1 -stevedore>=1.1.0 -six>=1.5.2 -python-debian>=0.1.21 -lxml>=3.2 diff --git a/setup.cfg b/setup.cfg deleted file mode 100644 index 8c0a808..0000000 --- a/setup.cfg +++ /dev/null @@ -1,67 +0,0 @@ -[metadata] -name = packetary -version = 10.0.0 -summary = The chain of tools to manage package`s lifecycle. -description-file = - README.rst -author = Mirantis Inc. -author_email = product@mirantis.com -url = http://mirantis.com -home-page = http://mirantis.com -classifier = - Development Status :: 4 - Beta - Environment :: OpenStack - Intended Audience :: Information Technology - Intended Audience :: System Administrators - License :: OSI Approved :: GNU General Public License v2 (GPLv2) - Operating System :: POSIX :: Linux - Programming Language :: Python - Programming Language :: Python :: 2 - Programming Language :: Python :: 2.7 - Programming Language :: Python :: 3 - Programming Language :: Python :: 3.4 - Topic :: Utilities - -[files] -packages = - packetary - -[entry_points] -console_scripts = - packetary=packetary.cli.app:main - -packetary.drivers = - deb=packetary.drivers.deb_driver:DebRepositoryDriver - rpm=packetary.drivers.rpm_driver:RpmRepositoryDriver - -packetary = - clone=packetary.cli.commands.clone:CloneCommand - packages=packetary.cli.commands.packages:ListOfPackages - unresolved=packetary.cli.commands.unresolved:ListOfUnresolved - -[build_sphinx] -source-dir = doc/source -build-dir = doc/build -all_files = 1 - -[upload_sphinx] -upload-dir = doc/build/html - -[compile_catalog] -directory = packetary/locale -domain = packetary - -[update_catalog] -domain = packetary -output_dir = packetary/locale -input_file = packetary/locale/packetary.pot - -[extract_messages] -keywords = _ gettext ngettext l_ lazy_gettext -mapping_file = babel.cfg -output_file = packetary/locale/packetary.pot - -[global] -setup-hooks = - pbr.hooks.setup_hook - setup_hooks.setup_hook diff --git a/setup.py b/setup.py deleted file mode 100644 index ec4d54a..0000000 --- a/setup.py +++ /dev/null @@ -1,33 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT -import setuptools - -# In python < 2.7.4, a lazy loading of package `pbr` will break -# setuptools if some other modules registered functions in `atexit`. -# solution from: http://bugs.python.org/issue15881#msg170215 -try: - import multiprocessing # noqa -except ImportError: - pass - - -setuptools.setup( - setup_requires=['pbr'], - pbr=True) diff --git a/setup_hooks.py b/setup_hooks.py deleted file mode 100644 index 94cc99e..0000000 --- a/setup_hooks.py +++ /dev/null @@ -1,25 +0,0 @@ -# -*- coding: utf-8 -*- - -# Copyright 2015 Mirantis, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - -def setup_hook(config): - import pbr - import pbr.packaging - - # this monkey patch is to avoid appending git version to version - pbr.packaging._get_version_from_git = lambda pre_version: pre_version diff --git a/specs/fuel-mirror.spec b/specs/fuel-mirror.spec deleted file mode 100644 index ef646cc..0000000 --- a/specs/fuel-mirror.spec +++ /dev/null @@ -1,95 +0,0 @@ -%define name fuel-mirror -%{!?version: %define version 10.0.0} -%{!?release: %define release 1} - -Name: %{name} -Version: %{version} -Release: %{release} -Source0: %{name}-%{version}.tar.gz -Summary: Utility to create RPM and DEB mirror -URL: http://mirantis.com -License: GPLv2 -Group: Utilities -BuildRoot: %{_tmppath}/%{name}-%{version}-buildroot -Prefix: %{_prefix} -BuildRequires: git -BuildRequires: python-setuptools -BuildRequires: python-pbr -BuildArch: noarch - -Requires: python -Requires: python-babel >= 1.3 -Requires: python-cliff >= 1.7.0 -Requires: python-fuelclient >= 7.0.0 -Requires: python-packetary == %{version} -Requires: python-pbr >= 0.8 -Requires: python-six >= 1.5.2 -Requires: PyYAML >= 3.10 -# Workaroud for babel bug -Requires: pytz - - -%description -Provides two commands fuel-mirror and fuel-createmirror. -Second one is for backward compatibility with the previous -generation of the utility. These commands could be used -to create local copies of MOS and upstream deb and rpm -repositories. - -%package -n python-packetary -Summary: Library that allows to build and clone deb and rpm repositories -Group: Development/Libraries - -Requires: createrepo -Requires: python -Requires: python-babel >= 1.3 -Requires: python-bintrees >= 2.0.2 -Requires: python-chardet >= 2.0.1 -Requires: python-cliff >= 1.7.0 -Requires: python-debian >= 0.1.21 -Requires: python-eventlet >= 0.15 -Requires: python-lxml >= 1.1.23 -Requires: python-pbr >= 0.8 -Requires: python-six >= 1.5.2 -Requires: python-stevedore >= 1.1.0 -# Workaroud for babel bug -Requires: pytz - -%description -n python-packetary -Provides object model and API for dealing with deb -and rpm repositories. One can use this framework to -implement operations like building repository -from a set of packages, clone repository, find package -dependencies, mix repositories, pull out a subset of -packages into a separate repository, etc. - - -%prep -%setup -cq -n %{name}-%{version} - -%build - -cd %{_builddir}/%{name}-%{version} && python setup.py build -cd %{_builddir}/%{name}-%{version}/contrib/fuel_mirror && python setup.py build - -%install -cd %{_builddir}/%{name}-%{version} && python setup.py install --single-version-externally-managed -O1 --root=$RPM_BUILD_ROOT --record=%{_builddir}/%{name}-%{version}/INSTALLED_FILES -cd %{_builddir}/%{name}-%{version}/contrib/fuel_mirror && python setup.py install --single-version-externally-managed -O1 --root=$RPM_BUILD_ROOT --record=%{_builddir}/%{name}-%{version}/contrib/fuel_mirror/INSTALLED_FILES - -mkdir -p %{buildroot}/etc/%{name} -mkdir -p %{buildroot}/usr/bin -mkdir -p %{buildroot}/usr/share/%{name} -install -m 755 %{_builddir}/%{name}-%{version}/contrib/fuel_mirror/scripts/fuel-createmirror %{buildroot}/usr/bin/fuel-createmirror -install -m 755 %{_builddir}/%{name}-%{version}/contrib/fuel_mirror/etc/config.yaml %{buildroot}/etc/%{name}/config.yaml - -%clean -rm -rf $RPM_BUILD_ROOT - -%files -f %{_builddir}/%{name}-%{version}/contrib/fuel_mirror/INSTALLED_FILES -%defattr(0755,root,root) -/usr/bin/fuel-createmirror -%attr(0644,root,root) /etc/%{name}/config.yaml - - -%files -n python-packetary -f %{_builddir}/%{name}-%{version}/INSTALLED_FILES -%defattr(-,root,root) diff --git a/test-requirements.txt b/test-requirements.txt deleted file mode 100644 index 0c885d3..0000000 --- a/test-requirements.txt +++ /dev/null @@ -1,15 +0,0 @@ -# The order of packages is significant, because pip processes them in the order -# of appearance. Changing the order has an impact on the overall integration -# process, which may cause wedges in the gate later. - -hacking<0.11,>=0.10.0 - -coverage>=3.6 -discover -python-subunit>=0.0.18 -sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2 -oslosphinx>=2.5.0 # Apache-2.0 -oslotest>=1.10.0 # Apache-2.0 -testrepository>=0.0.18 -testscenarios>=0.4 -testtools>=1.4.0 diff --git a/tox.ini b/tox.ini index 3e02f93..2717aed 100644 --- a/tox.ini +++ b/tox.ini @@ -1,36 +1,12 @@ [tox] minversion = 1.6 -envlist = py34,py27,pep8 +envlist = pep8 skipsdist = True [testenv] -usedevelop = True -install_command = pip install -U {opts} {packages} -setenv = - VIRTUAL_ENV={envdir} -passenv = http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY -deps = -r{toxinidir}/test-requirements.txt -commands = python setup.py test --slowest --testr-args='{posargs:packetary}' +usedevelop = False +whitelist_externals = bash +commands = [testenv:pep8] -commands = flake8 {posargs:packetary} - -[testenv:venv] -commands = {posargs:packetary} - -[testenv:cover] -commands = python setup.py test --coverage --testr-args='{posargs:packetary}' - -[testenv:docs] -commands = python setup.py build_sphinx - -[testenv:debug] -commands = oslo_debug_helper {posargs:packetary} - -[flake8] -# E123, E125 skipped as they are invalid PEP-8. - -show-source = True -ignore = E123,E125 -builtins = _ -exclude=*egg,*lib/python*,*openstack/common*,.git,.idea,.tox,.venv,build,dist,doc +commands = bash -c "exit 0"