Some roles may have requirements.txt files. These are
not files we want to take into account when compiling
requirements for the repo build, so we remove them.
This has specifically become an issue due to ceph-ansible
carrying conflicting requirements to OpenStack's.
Change-Id: I0fd29dc4dcb008689f36d947bc971643dab8feec
In https://review.openstack.org/567646 this line (which was
used for debugging) was included by mistake. Leaving it there
results in malformed yaml when executing the py_pkgs plugin
from a console, so it must be removed.
Change-Id: I58a76109c272b0be384c4113069f48415e8ab010
There is no need to build the docs python packages when doing
a repo build. The packages aren't required for a successful
deployment of OpenStack. The output of their data just adds
noise to the output of the lookup, and delays the build with
unnecessary extra wheel builds.
Change-Id: Ia60e1a49d9befdbafdaf89e5cf21e17985a706e7
The repo build does not need to build the requirements necessary
for role tests. This wastes time unnecessarily and increases the
size of the lookup output, making it more confusing than it
already is.
The REQUIREMENTS_FILE_TYPES it looks for is now reduced to only
'requirements.txt' and 'global-requirement-pins.txt'.
Change-Id: I11acc957ce4fe6024a40d5bb048a448deaf41778
The indexing of distro packages by the py_pkgs lookup was
intended for a purpose which never came to fruition. As
such it is not used. The resulting output is somewhat
confusing and very verbose. As such, I think it's best
to remove it.
Change-Id: I0d133804a91feae03e44047eede589c75806b2da
When the user gives a folder to the lookup, it's expected to work,
whatever the content of the folder is.
In the Zuul v3 case, we are symlinking the role that are checked
out as part of the Zuul dependency resolution, for example when
Depends-On is used.
If we don't follow symlinks, those checkout roles aren't part
of the repo build process, and won't be build.
Change-Id: Ic09f6a9ee28e3f7919226f22b803ee1a1b8ad53d
Ansible v1.9 was last used by OSA in Mitaka. The extra code to keep the
config_template and py_pkgs plugins compatible should no longer be
needed.
Change-Id: I14b32af1dc9bfe8b09c52f0205afaaad9139b58c
in py3 basestring is no longer valid so we need to support basestring
for as long as we support py2 but we also need to have a way to carry
forward seemslessly. A try and except block has been added to allow us
to support both.
Py3 also will not allow us to "sort" a dict, while this was allowed in
py2 it didn't do anything however in py3 it causes a stacktrace. To fix
this issue a try/except block has been added to catch the type error and
simply set the value without sorting it.
Change-Id: I81c135c5bf0867f7faea248457494547cbcd87cd
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Python 3 raises errors when attempting to compare types that don't have
a meaningful natural ordering.
If a processed file does not belong to a role it's not neccessary to
create a mapping of a project_group for a `None` role, to prevent
'None' from being compared to other string role names within the
'role_project_groups' dict.
For variables containing lists of dictionaries, like
'remote_package_parts', sort using the 'name' property of each
dictionary.
Change-Id: I9ec4b0b3b7236450f19c1d6186eb310fb7f1e083
Implements: blueprint goal-python35
basestring no longer exists in Python 3, str can be used in its place.
Change-Id: Icc386307a88c2a9ed2a3e7957b5f12a54c8ce8f4
Implements: blueprint goal-python35
This is a lookup in case you need to parse a ubuntu repo Packages file
for a list of packages. It's useful in osa if you want to have a mirror
of an ubuntu repo, and output a list of its packages, with their
location and checksum.
You can use the output dict to feed another tool (for example
downloading with aria).
Change-Id: I10ba72f0b64267ae6afe57317e135d782445e934
Some packages simply do not play well with others. One such case is
"tempest" where time and time again it's unbound requirements break
everything. This change allows the package indexer to examine remote
repos and exclude packages from the global package building list by
using the metadata value ``ignorerequirements=True``. When a package is
ignored it can still be built within in a venv or used by referencing
the git repos however it will no longer be part of the global wheel
building package list.
Change-Id: Iacf15eea14a2cd9c98bbb59d99e2a1ea94bd0a5b
Closes-Bug: #1631992
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The distro packages index was not collecting packages and variables
as it should've been. This change ensures that it now is.
Change-Id: I9ba19d273658cc93456c43475694fc640b4e58b8
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The proprietary packages index should never be present in the package
index. This change makes sure that anything using the proprietary key
is fully ignored.
Closes-Bug: 1622275
Change-Id: I5f084487c1085868eac84bcccadc85c43ec63caa
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This py_pkg lookup plugin is now able to index distro packages that use
the "distro_package" name within them.
Change-Id: I0e08311f217a52c3d312a4e1726961f0e51a8721
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change implements indexing of py packags from individual
keys so that we can further tune the build process to be even
more selective by default.
Change-Id: Iad00e4f1268187a29e90d7b8e6afb5abfd1e92a2
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change creates new group membership items so that we can
do selective git clones and wheel building based on defined
groups.
It also itemises the global-requirements-pins to ensure that they can
be used to update upper-constraints.
Needed-By: Idda16b4f382eee57c7469af898859d6d81d4eb30
Needed-By: I6e5a554d6f87058ef5cb63adf7995b066cb6fd78
Needed-By: I6a0508e8aaa4993bc68e72e3d2988d6db6470f6d
Change-Id: I6a0508e8aaa4993bc68e72e3d2988d6db6470f6d
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change fixes a regression created when we Updated lookup plugin
to allow it to also index requirement files. The offending change is
here: "I5e981c818a1da50aa0ae15630655a8dfac4f3db7". This simply ensures
that only non-optional packages are part of the specific role
requirements.
Change-Id: I1b65a595d8691fdd38ffd568db71a1bcc06207c9
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change indexes all of the requirement files found within the
plugins search path. A new section has been added for
"role_requirement_files" which contains keys using the requirement
file name. In the event that a requirement file is found and it is
not within a known role it the vaules of the requirement file will
be added to the "default" section.
Change-Id: I5e981c818a1da50aa0ae15630655a8dfac4f3db7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The plugin needs to omit the "other" requirements files when
indexing python packages.
Change-Id: I52600165c24031d499c604044c30fb403ec51741
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change ensures that the dependency file ordering
for requirements file is kept intact for all fot the discovered
file used to build the repo environment.
Change-Id: I36d2044965c8a0809a417083c3f24c406b5a36c9
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change simply adds a case statement to the known
packages to build where it will NOT build a proprietary
package. This change is being made in support of OSA being
able to deploy closed source software.
To support proprietary packages the a new method named
"_package_build_index" was added and a loop was added the
original "_process_files" method. These changes allow
the plugin to index all packages and in a given variable file
and then find the any package that is proprietary. If the
proprietary package has a coresponding known git repo
assosiated with it the package will be added to the build list
otherwise it will simply be skipped and it will be up to the
deployer to make the closed source software available to the
deployment.
Change-Id: I75a29f710aa65c13c3067c06c02b951e44a38161
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change update the lookup plugin such that it will run
under ansible 1.9, 2.0, and 2.1 (head of devel branch). This
change ensures compatibility across all known versions of
ansible at this time.
Change-Id: I11586927a363107f9cd3e8264845b9eda88a9c40
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Several GitLab users have said that they have to include the .git in
in the URL of their _git_repo variable overrides and then subsequently
override the package name using _git_package_name. Failing to do this
means the repo is cloned with .git in the directory name.
This change updates lookup/py_pkgs.py to strip a trailing newline,
a trailing .git, or a trailing .git/ from _git_repo variables.
Change-Id: I174ab1398be8e180c346ea29fa71238cc3698c45
Processed yaml configuration files are expected to have iterable
key-value pairs so loaded_configs should be validated to make sure
they're dictionary objects.
Ansible Vault encrypted user_secrets will be loaded by yaml as a string
and then ignored from further processing.
Change-Id: I08c5bff89eae2d49ab595317f249a420196c9cb1
The standard Ansible names for the plugins folders [1] are singular.
This patch renames our plural names to singular names so that when
the plugins are downloaded into a default location there is no further
configuration required.
[1] https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py
Change-Id: Ib965750f267d11287a6196e2544c3a014a3e2029
Jesse Pretorius