openstack
/
openstack-ansible-repo_build
Code Issues Proposed changes

Fix repo constraints construction and package installation 

The current repo build process has the following issues:

1. The constraints consruction doesn't properly handle two constraints
   which use the same name, but have different version specs. eg:
   ovs===2.5.0;python_version=='2.7'
   ovs===2.6.0.dev2;python_version=='3.4'
   This is a problem in the constraints construction.

2. The pip packages installed on the repo server in order to
   construct the repo do not respect the global pins or the upper
   constraint overrides.

3. The constraints are selected based on the requirements specified.
   This makes the process unnecessarily complex.

4. The global pins are not applied to the constructed constraints,
   so the wrong packages get built and installed.

This patch corrects all of these issues and hopefully makes the
mechanism for constructing constraints more readily apparent, and
therefore easier to maintain.

Conflicts:
>------tasks/repo_pre_build.yml
>------tests/test-repo-build.yml
>------tests/test-vars.yml

This backport also includes corrections to the template from
https://review.openstack.org/350654

Depends-On: Ic473eb9de26a2a3b92e234851b8f731c0255976c
Change-Id: If52d073d43081468e2faf2cd063c4b211c29994f
Related-Bug: #1605846
Closes-Bug: #1609056
(cherry picked from commit 38d7faffa9)
This commit is contained in:
Jesse Pretorius 2016-08-03 00:15:56 +01:00
parent cff0a5592b
commit ae56b67cc0
5 changed files with 85 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
releasenotes/notes/upper-constraints-override-6853ffec6c07d7f5.yaml
View File

@ -1,6 +1,9 @@
---
features:
- The ``repo_build`` role now provides the ability to override the
upper-constraints applied which are sourced from OpenStack. The
variable ``repo_build_upper_constraints_overrides`` can be
populated with a list of upper constraints.
upper-constraints applied which are sourced from OpenStack and
from the global-requirements-pins.txt file. The variable
``repo_build_upper_constraints_overrides`` can be populated with
a list of upper constraints. This list will take the highest
precedence in the constraints process, with the exception of
the pins set in the git source SHAs.

26
tasks/repo_pre_build.yml
View File

@ -13,19 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
- name: Install pip packages
pip:
name: "{{ item }}"
state: latest
extra_args: "--constraint {{ repo_build_git_dir }}/requirements/upper-constraints.txt {{ pip_install_options|default('') }}"
register: install_packages
until: install_packages|success
retries: 5
delay: 5
with_items: repo_pip_packages
tags:
- repo-pip-packages
- name: Ensure workspace files are cleaned up
file:
path: "{{ item }}"
@ -62,3 +49,16 @@
dest: "{{ repo_build_release_path }}/{{ repo_build_release_tag }}/requirements_constraints.txt"
tags:
- repo-build-constraints-file
- name: Install pip packages
pip:
name: "{{ item }}"
state: latest
extra_args: "--constraint {{ repo_build_release_path }}/{{ repo_build_release_tag }}/requirements_constraints.txt {{ pip_install_options|default('') }}"
register: install_packages
until: install_packages|success
retries: 5
delay: 5
with_items: repo_pip_packages
tags:
- repo-pip-packages

19
tasks/repo_set_facts.yml
View File

@ -41,25 +41,8 @@
- name: Decode the upper constraints content
set_fact:
_upper_constraints: "{{ slurp_upper_constraints.content | b64decode | splitlines }}"
upper_constraints: "{{ slurp_upper_constraints.content | b64decode | splitlines }}"
when: slurp_upper_constraints | success
tags:
- repo-set-constraints
- repo-build-constraints-file
- name: Normalise the upper constraints
set_fact:
upper_constraints: "{{ local_requirement_normalized | pip_constraint_update(_upper_constraints) }}"
when: slurp_upper_constraints | success
tags:
- repo-set-constraints
- repo-build-constraints-file
- name: Apply the upper constraint overrides
set_fact:
upper_constraints: "{{ upper_constraints | pip_constraint_update(repo_build_upper_constraints_overrides) }}"
when: repo_build_upper_constraints_overrides | length > 0
tags:
- repo-set-constraints
- repo-build-constraints-file

62
templates/requirements_constraints.txt.j2
View File

@ -1,21 +1,47 @@
# Computed constraints
{% set constraint_pkgs = [] -%}
{% for clone_item in local_packages.results.0.item.remote_package_parts -%}
{% if 'ignorerequirements=true' not in clone_item['original'] %}
#
# Constraints set by SHA's in the git sources
#
{% set constraint_pkgs = [] %}
{% for clone_item in local_packages.results.0.item.remote_package_parts %}
{% if 'ignorerequirements=true' not in clone_item['original'] %}
{{ clone_item['original'] | replace(clone_item['url'], 'file://' + repo_build_git_dir + '/' + clone_item['name'] ) }}
{% set _ = constraint_pkgs.append(clone_item['name'] | replace('-', '_') | lower) %}
{% endif %}
{% set _ = constraint_pkgs.append(clone_item['name'] | replace('-', '_') | lower) %}
{% endif %}
{% endfor %}
# upper boundry constraints from requirements repo.
{% for constraint_item in upper_constraints %}
{%- set constraint_split = constraint_item.split('===') %}
{%- set constraint_name = constraint_split[0] %}
{%- set constraint_name_normalized = constraint_name | replace('-', '_') | lower %}
{% if constraint_name_normalized not in constraint_pkgs %}
{% if repo_build_use_upper_constraints | bool and (constraint_split | length) > 1 %}
{{ constraint_split[0] | replace('-', '_') | lower }}<={{ constraint_split[1] }}
{% elif (constraint_split | length) == 1 %}
{{ constraint_item }}
#
# User-provided constraints set through a variable
#
{% set override_packages = [] %}
{% for constraint_override_item in repo_build_upper_constraints_overrides %}
{% set constraint_override_name = constraint_override_item | regex_replace('(>=|<=|>|<|==|~=|!=).*$','') %}
{% set _ = override_packages.append(constraint_override_name) %}
{{ constraint_override_item }}
{% endfor %}
#
# Global pins set through the file global-requirement-pins.txt
#
{% set global_pin_packages = [] %}
{% for global_pin in local_packages.results.0.item.role_requirement_files.global_pins.pinned_packages %}
{% set global_pin_package_name = global_pin | regex_replace('(<=|<|==).*$','') %}
{% set _ = global_pin_packages.append(global_pin_package_name) %}
{# we want to ensure that repo_build_upper_constraints_overrides take the highest precedence #}
{% if global_pin_package_name not in repo_build_upper_constraints_overrides %}
{{ global_pin }}
{% endif %}
{% endfor %}
{# we don't bother applying OpenStack upper-constraints if the deployer has opted not to #}
{% if repo_build_use_upper_constraints | bool %}
#
# Upper constraints from the OpenStack requirements repo
#
{% for constraint_item in upper_constraints %}
{% set constraint_name = constraint_item | regex_replace('===.*', '') %}
{% set constraint_data = constraint_item | regex_replace('.*===', '') %}
{# The name has to be normalised to comply with PEP standards #}
{% set constraint_name_normalized = constraint_name | replace('-', '_') | lower %}
{% set constraint = constraint_name_normalized + '<=' + constraint_data %}
{% if (constraint_name_normalized not in constraint_pkgs) and (constraint_name_normalized not in override_packages) and (constraint_name_normalized not in global_pin_packages) %}
{{ constraint }}
{% endif %}
{% endfor %}
{% endif %}
{% endif %}
{% endfor %}

21
tests/test.yml
View File

@ -29,6 +29,27 @@
results: [
{
"item": {
"role_requirement_files": {
"default": {
"txt_file_packages": [
"netaddr>=0.7.12",
"prettytable>=0.7,<0.8",
"pycrypto>=2.6",
"pyyaml>=3.1.0",
"virtualenv",
"pip==8.1.1",
"setuptools==20.3.1",
"wheel==0.29.0"
]
},
"global_pins": {
"pinned_packages": [
"pip==8.1.1",
"setuptools==20.3.1",
"wheel==0.29.0"
]
}
},
"packages": [
"ansible-lint<=2.3.9",
"ansible>1.9,<2.0",