diff --git a/.gitignore b/.gitignore index 8a1c59f3cc..78303126aa 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ *.egg* *.mo +*.pot *.pyc *.sw? *.sqlite3 diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst index fc6380097d..05c6255053 100644 --- a/doc/source/contributing.rst +++ b/doc/source/contributing.rst @@ -89,9 +89,8 @@ After You Write Your Patch Once you've made your changes, there are a few things to do: -* Make sure the unit tests pass: ``./run_tests.sh`` for Python, and ``npm run test`` for JS. -* Make sure the linting tasks pass: ``./run_tests.sh --pep8`` for Python, and ``npm run lint`` for JS. -* Make sure your code is ready for translation: ``./run_tests.sh --pseudo de`` See :ref:`pseudo_translation` for more information. +* Make sure the unit tests and linting tasks pass by running ``tox`` +* Make sure your code is ready for translation: See :ref:`pseudo_translation`. * Make sure your code is up-to-date with the latest master: ``git pull --rebase`` * Finally, run ``git review`` to upload your changes to Gerrit for review. @@ -132,7 +131,7 @@ Python ------ We follow PEP8_ for all our Python code, and use ``pep8.py`` (available -via the shortcut ``./run_tests.sh --pep8``) to validate that our code +via the shortcut ``tox -e pep8``) to validate that our code meets proper Python style guidelines. .. _PEP8: http://www.python.org/dev/peps/pep-0008/ diff --git a/doc/source/quickstart.rst b/doc/source/quickstart.rst index d5924ba54c..0a5f765b18 100644 --- a/doc/source/quickstart.rst +++ b/doc/source/quickstart.rst @@ -22,20 +22,10 @@ On RPM-based distributions (e.g., Fedora/RHEL/CentOS/Scientific Linux):: Setup ===== -To setup a Horizon development environment simply clone the Horizon git -repository from http://github.com/openstack/horizon and execute the -``run_tests.sh`` script from the root folder (see :doc:`ref/run_tests`):: +To begin setting up a Horizon development environment simply clone the Horizon +git repository from https://git.openstack.org/cgit/openstack/horizon.:: - > git clone https://github.com/openstack/horizon.git - > cd horizon - > ./run_tests.sh - -.. note:: - - Running ``run_tests.sh`` will build a virtualenv, ``.venv``, where all the - python dependencies for Horizon are installed and referenced. After the - dependencies are installed, the unit test suites in the Horizon repo will be - executed. There should be no errors from the tests. + > git clone https://git.openstack.org/openstack/horizon Next you will need to setup your Django application config by copying ``openstack_dashboard/local/local_settings.py.example`` to ``openstack_dashboard/local/local_settings.py``. To do this quickly you can use the following command:: @@ -92,21 +82,21 @@ order to prevent Conflicts for future migrations:: > mv openstack_dashboard/local/local_settings.diff openstack_dashboard/local/local_settings.diff.old > python manage.py migrate_settings --gendiff -To start the Horizon development server use ``run_tests.sh``:: +To start the Horizon development server use ``tox``:: - > ./run_tests.sh --runserver localhost:9000 + > tox -e runserver .. note:: The default port for runserver is 8000 which is already consumed by - heat-api-cfn in DevStack. If not running in DevStack - `./run_tests.sh --runserver` will start the test server at - `http://localhost:8000`. + heat-api-cfn in DevStack. If running in DevStack + `tox -e runserver -- localhost:9000` will start the test server at + `http://localhost:9000`. .. note:: - The ``run_tests.sh`` script provides wrappers around ``manage.py``. + The ``tox`` environments provide wrappers around ``manage.py``. For more information on manage.py which is a django, see `https://docs.djangoproject.com/en/dev/ref/django-admin/` diff --git a/doc/source/ref/run_tests.rst b/doc/source/ref/run_tests.rst index 951c82e812..7f88e0d8e8 100644 --- a/doc/source/ref/run_tests.rst +++ b/doc/source/ref/run_tests.rst @@ -2,6 +2,12 @@ The ``run_tests.sh`` Script =========================== +.. warning:: + + This script is deprecated as of Newton (11.0), and will be removed in + Queens (13.0), in favor of tox. The tox docs can be found at + https://tox.readthedocs.io/en/latest/ + .. contents:: Contents: :local: diff --git a/doc/source/testing.rst b/doc/source/testing.rst index 78ddf10c0a..d56a3c1a7f 100644 --- a/doc/source/testing.rst +++ b/doc/source/testing.rst @@ -10,34 +10,33 @@ Because Horizon is composed of both the ``horizon`` app and the tests. While they can be run individually without problem, there is an easier way: -Included at the root of the repository is the ``run_tests.sh`` script +Included at the root of the repository is the ``tox.ini`` config which invokes both sets of tests, and optionally generates analyses on both -components in the process. This script is what Jenkins uses to verify the +components in the process. ``tox`` is what Jenkins uses to verify the stability of the project, so you should make sure you run it and it passes before you submit any pull requests/patches. -To run the tests:: +To run all tests:: - $ ./run_tests.sh - -It's also possible to :doc:`run a subset of unit tests`. - -.. seealso:: - - :doc:`ref/run_tests` - Full reference for the ``run_tests.sh`` script. + $ tox +It's also possible to run a subset of the tests. Open ``tox.ini`` in the +Horizon root directory to see a list of test environments. You can read more +about tox in general at https://tox.readthedocs.io/en/latest/. By default running the Selenium tests will open your Firefox browser (you have to install it first, else an error is raised), and you will be able to see the -tests actions. +tests actions:: + + $ tox -e selenium-headless + If you want to run the suite headless, without being able to see them (as they are ran on Jenkins), you can run the tests:: - $ ./run_tests.sh --with-selenium --selenium-headless + $ tox -e selenium-headless Selenium will use a virtual display in this case, instead of your own. In order -to run the tests this way you have to install the dependency `xvfb`, like +to run the tests this way you have to install the dependency `xvfb`, like this:: $ sudo apt-get install xvfb @@ -49,12 +48,90 @@ for a Debian OS flavour, or for Fedora/Red Hat flavours:: If you can't run a virtual display, or would prefer not to, you can use the PhantomJS web driver instead:: - $ ./run_tests.sh --with-selenium --selenium-phantomjs + $ tox -e selenium-phantomjs If you need to install PhantomJS, you may do so with `npm` like this:: $ npm -g install phantomjs +Alternatively, many distributions have system packages for phantomjs, or +it can be downloaded from http://phantomjs.org/download.html. + +tox Test Environments +===================== + +This is a list of test environments available to be executed by +``tox -e ``. + +pep8 +---- + +Runs pep8, which is a tool that checks Python code style. You can read more +about pep8 at https://www.python.org/dev/peps/pep-0008/ + +py27dj18, py27dj19, py27dj110 +----------------------------- + +Runs the Python unit tests against Django 1.8, Django 1.9 and Django 1.10 +respectively + +All other dependencies are as defined by the upper-constraints file at +https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt + +You can run a subset of the tests by passing the test path as an argument to +tox:: + + $ tox -e py27dj18 -- openstack_dashboard.dashboards.identity.users.tests + +You can also pass other arguments. For example, to drop into a live debugger +when a test fails you can use:: + + $ tox -e py27dj18 -- --pdb + +py34 +---- + +Runs the Python unit tests with a Python 3.4 environment. + +py35 +---- + +Runs the Python unit tests with a Python 3.5 environment. + +releasenotes +------------ + +Outputs Horizons release notes as HTML to ``releasenotes/build/html``. + +Also takes an alternative builder as an optional argument, such as +``tox -e docs -- ``, which will output to +``releasenotes/build/``. Available builders are listed at +http://www.sphinx-doc.org/en/latest/builders.html + +npm +--- + +Installs the npm dependencies listed in ``package.json`` and runs the +JavaScript tests. Can also take optional arguments, which will be executed +as an npm script following the dependency install, instead of ``test``. + +Example:: + + $ tox -e npm -- lintq + +docs +---- + +Outputs Horizons documentation as HTML to ``doc/build/html``. + +Also takes an alternative builder as an optional argument, such as +``tox -e docs -- ``, which will output to ``doc/build/``. +Available builders are listed at +http://www.sphinx-doc.org/en/latest/builders.html + +Example:: + + $ tox -e docs -- latexpdf Writing tests ============= diff --git a/doc/source/topics/angularjs.rst b/doc/source/topics/angularjs.rst index cadc8e02c1..939370449c 100644 --- a/doc/source/topics/angularjs.rst +++ b/doc/source/topics/angularjs.rst @@ -38,7 +38,7 @@ ESLint ESLint is a tool for identifying and reporting on patterns in your JS code, and is part of the automated tests run by Jenkins. You can run ESLint from the -horizon root directory with ``npm run lint``, or alternatively on a specific +horizon root directory with ``tox -e npm -- lint``, or alternatively on a specific directory or file with ``eslint file.js``. Horizon includes a `.eslintrc` in its root directory, that is used by the @@ -217,10 +217,13 @@ Testing ======= 1. Open /jasmine in a browser. The development server can be run - with``./run_tests.sh --runserver`` from the horizon root directory. -2. ``npm run test`` from the horizon root directory. + with ``tox -e runserver`` from the horizon root directory; by default, this will + run the development server at ``http://localhost:8000``. +2. ``tox -e npm`` from the horizon root directory. -The code linting job can be run with ``npm run lint``. +The code linting job can be run with ``tox -e npm -- lint``. If there are many +warnings, you can also use ``tox -e npm -- lintq`` to see only errors and +ignore warnings. For more detailed information, see :doc:`javascript_testing`. diff --git a/doc/source/topics/customizing.rst b/doc/source/topics/customizing.rst index 4c5d0ef618..9115b0e7ed 100644 --- a/doc/source/topics/customizing.rst +++ b/doc/source/topics/customizing.rst @@ -68,7 +68,7 @@ theme's ``_variables.scss``:: @import "/themes/default/variables"; Once you have made your changes you must re-generate the static files with - ``./run_tests.sh -m collectstatic``. + ``tox -e manage -- collectstatic``. By default, all of the themes configured by ``AVAILABLE_THEMES`` setting are collected by horizon during the `collectstatic` process. By default, the themes diff --git a/doc/source/topics/install.rst b/doc/source/topics/install.rst index 129791943b..91104afe65 100644 --- a/doc/source/topics/install.rst +++ b/doc/source/topics/install.rst @@ -39,7 +39,7 @@ Installation message catalogs:: $ sudo apt-get install gettext - $ ./run_tests.sh --compilemessages + $ tox -e manage -- compilemessages This command compiles translation message catalogs within Python virtualenv named ``.venv``. After this step, you can remove diff --git a/doc/source/topics/javascript_testing.rst b/doc/source/topics/javascript_testing.rst index d0d3d0855e..1fd702d34a 100644 --- a/doc/source/topics/javascript_testing.rst +++ b/doc/source/topics/javascript_testing.rst @@ -31,12 +31,13 @@ Running Tests Tests can be run in two ways: 1. Open /jasmine in a browser. The development server can be - run with ``./run_tests.sh --runserver`` from the horizon root directory. - 2. ``npm run test`` from the horizon root directory. This runs Karma, + run with ``tox -e runserver`` from the horizon root directory. + 2. ``tox -e npm`` from the horizon root directory. This runs Karma, so it will run all the tests against PhantomJS and generate coverage reports. -The code linting job can be run with ``npm run lint``. +The code linting job can be run with ``tox -e npm -- lint``, or +``tox -e npm -- lintq`` to show errors, but not warnings. Coverage Reports ---------------- @@ -45,7 +46,7 @@ Our Karma setup includes a plugin to generate test coverage reports. When developing, be sure to check the coverage reports on the master branch and compare your development branch; this will help identify missing tests. -To generate coverage reports, run ``npm run test``. The coverage reports can be +To generate coverage reports, run ``tox -e npm``. The coverage reports can be found at ``horizon/coverage-karma/`` (framework tests) and ``openstack_dashboard/coverage-karma/`` (dashboard tests). Load ``/index.html`` in a browser to view the reports. diff --git a/doc/source/topics/translation.rst b/doc/source/topics/translation.rst index 82da561ee3..a3207d0946 100644 --- a/doc/source/topics/translation.rst +++ b/doc/source/topics/translation.rst @@ -46,13 +46,19 @@ translated. Lets break this up into steps we can follow: to locate them. Refer to the guide below on how to use translation and what these markers look like. -2. Once marked, we can then run ``./run_tests.sh --makemessages``, which +2. Once marked, we can then run ``tox -e manage -- extract_messages``, which searches the codebase for these markers and extracts them into a Portable Object Template (POT) file. In horizon, we extract from both the ``horizon`` folder and the ``openstack_dashboard`` folder. We use the AngularJS extractor for JavaScript and HTML files and the Django extractor for Python and Django templates; both extractors are Babel plugins. +3. To update the .po files, you can run ``tox -e manage -- update_catalog`` to + update the .po file for every language, or you can specify a specific + language to update like this: ``tox -e manage -- update_catalog de``. This + is useful if you want to add a few extra translatabale strings for a + downstream customisation. + .. Note :: When pushing code upstream, the only requirement is to mark the strings @@ -242,12 +248,12 @@ translations to validate that your code is ready for translation. Running the pseudo translation tool ----------------------------------- -#. Make sure your English po file is up to date: - ``./run_tests.sh --makemessages`` +#. Make sure your .pot files are up to date: + ``tox -e manage -- extract_messages`` #. Run the pseudo tool to create pseudo translations. For example, to replace the German translation with a pseudo translation: - ``./run_tests.sh --pseudo de`` -#. Compile the catalog: ``./run_tests.sh --compilemessages`` + ``tox -e manage -- update_catalog de --pseudo`` +#. Compile the catalog: ``tox -e manage -- compilemessages`` #. Run your development server. #. Log in and change to the language you pseudo translated. diff --git a/doc/source/tutorials/dashboard.rst b/doc/source/tutorials/dashboard.rst index 96071ae5f4..412f85de76 100644 --- a/doc/source/tutorials/dashboard.rst +++ b/doc/source/tutorials/dashboard.rst @@ -30,20 +30,19 @@ The quick version ----------------- Horizon provides a custom management command to create a typical base -dashboard structure for you. Run the following commands at the same location -where the ``run_tests.sh`` file resides. It generates most of the boilerplate -code you need:: +dashboard structure for you. Run the following commands in your Horizon root +directory. It generates most of the boilerplate code you need:: - mkdir openstack_dashboard/dashboards/mydashboard + $ mkdir openstack_dashboard/dashboards/mydashboard - ./run_tests.sh -m startdash mydashboard \ - --target openstack_dashboard/dashboards/mydashboard + $ tox -e manage -- startdash mydashboard \ + --target openstack_dashboard/dashboards/mydashboard - mkdir openstack_dashboard/dashboards/mydashboard/mypanel + $ mkdir openstack_dashboard/dashboards/mydashboard/mypanel - ./run_tests.sh -m startpanel mypanel \ - --dashboard=openstack_dashboard.dashboards.mydashboard \ - --target=openstack_dashboard/dashboards/mydashboard/mypanel + $ tox -e manage -- startpanel mypanel \ + --dashboard=openstack_dashboard.dashboards.mydashboard \ + --target=openstack_dashboard/dashboards/mydashboard/mypanel You will notice that the directory ``mydashboard`` gets automatically @@ -562,10 +561,9 @@ Run and check the dashboard Everything is in place, now run ``Horizon`` on the different port:: - ./run_tests.sh --runserver 0.0.0.0:8877 + $ tox -e runserver -- 0:9000 - -Go to ``http://:8877`` using a browser. After login as an admin +Go to ``http://:9000`` using a browser. After login as an admin you should be able see ``My Dashboard`` shows up at the left side on horizon. Click it, ``My Group`` will expand with ``My Panel``. Click on ``My Panel``, the right side panel will display an ``Instances Tab`` which has an diff --git a/doc/source/tutorials/table_actions.rst b/doc/source/tutorials/table_actions.rst index 9cbb69a64d..b02d74bd57 100644 --- a/doc/source/tutorials/table_actions.rst +++ b/doc/source/tutorials/table_actions.rst @@ -274,10 +274,10 @@ Run and check the dashboard We must once again run horizon to verify our dashboard is working:: - ./run_tests.sh --runserver 0.0.0.0:8877 + $ tox -e runserver -- 0:9000 -Go to ``http://:8877`` using a browser. After login as an admin, +Go to ``http://:9000`` using a browser. After login as an admin, display ``My Panel`` to see the ``Instances`` table. For every ``ACTIVE`` instance in the table, there will be a ``Create Snapshot`` action on the row. Click on ``Create Snapshot``, enter a snapshot name in the form that is shown, diff --git a/horizon/karma.conf.js b/horizon/karma.conf.js index b8eeed9bc0..e3cc7d3baa 100644 --- a/horizon/karma.conf.js +++ b/horizon/karma.conf.js @@ -20,23 +20,14 @@ var fs = require('fs'); var path = require('path'); module.exports = function (config) { - var xstaticPath; - var basePaths = [ - './.venv', - './.tox/py27' - ]; + var xstaticPath = path.resolve('./.tox/npm'); - for (var i = 0; i < basePaths.length; i++) { - var basePath = path.resolve(basePaths[i]); - - if (fs.existsSync(basePath)) { - xstaticPath = basePath + '/lib/python2.7/site-packages/xstatic/pkg/'; - break; - } + if (fs.existsSync(xstaticPath)) { + xstaticPath += '/lib/python2.7/site-packages/xstatic/pkg/'; } if (!xstaticPath) { - console.error('xStatic libraries not found, please set up venv'); + console.error('xStatic libraries not found, please run `tox -e npm`'); process.exit(1); } diff --git a/openstack_dashboard/karma.conf.js b/openstack_dashboard/karma.conf.js index 0b91931f0a..e0acf2c52e 100644 --- a/openstack_dashboard/karma.conf.js +++ b/openstack_dashboard/karma.conf.js @@ -20,23 +20,14 @@ var fs = require('fs'); var path = require('path'); module.exports = function (config) { - var xstaticPath; - var basePaths = [ - './.venv', - './.tox/py27' - ]; + var xstaticPath = path.resolve('./.tox/npm'); - for (var i = 0; i < basePaths.length; i++) { - var basePath = path.resolve(basePaths[i]); - - if (fs.existsSync(basePath)) { - xstaticPath = basePath + '/lib/python2.7/site-packages/xstatic/pkg/'; - break; - } + if (fs.existsSync(xstaticPath)) { + xstaticPath += '/lib/python2.7/site-packages/xstatic/pkg/'; } if (!xstaticPath) { - console.error('xStatic libraries not found, please set up venv'); + console.error('xStatic libraries not found, please run `tox -e npm`'); process.exit(1); } diff --git a/openstack_dashboard/management/commands/extract_messages.py b/openstack_dashboard/management/commands/extract_messages.py new file mode 100644 index 0000000000..f015f26b38 --- /dev/null +++ b/openstack_dashboard/management/commands/extract_messages.py @@ -0,0 +1,57 @@ +# Copyright 2016 Cisco Systems, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from distutils.dist import Distribution +import os +from subprocess import call + +from django.core.management.base import BaseCommand + + +class Command(BaseCommand): + help = ('Extract strings that have been marked for translation into .POT ' + 'files.') + + def add_arguments(self, parser): + parser.add_argument('-m', '--module', type=str, nargs='+', + default=['openstack_dashboard', 'horizon'], + help=("The target python module(s) to extract " + "strings from")) + parser.add_argument('-d', '--domain', choices=['django', 'djangojs'], + nargs='+', default=['django', 'djangojs'], + help="Domain(s) of the .pot file") + parser.add_argument('--check-only', action='store_true', + help=("Checks that extraction works correctly, " + "then deletes the .pot file to avoid " + "polluting the source code")) + + def handle(self, *args, **options): + cmd = ('python setup.py extract_messages -F babel-{domain}.cfg ' + '-o {module}/locale/{domain}.pot') + distribution = Distribution() + distribution.parse_config_files(distribution.find_config_files()) + + if options['check_only']: + cmd += " ; rm {module}/locale/{domain}.pot" + + for module in options['module']: + for domain in options['domain']: + potfile = '{module}/locale/{domain}.pot'.format(module=module, + domain=domain) + if not os.path.exists(potfile): + with open(potfile, 'wb') as f: + f.write(b'') + + call(cmd.format(module=module, domain=domain, potfile=potfile), + shell=True) diff --git a/openstack_dashboard/management/commands/update_catalog.py b/openstack_dashboard/management/commands/update_catalog.py new file mode 100644 index 0000000000..522dd7f28d --- /dev/null +++ b/openstack_dashboard/management/commands/update_catalog.py @@ -0,0 +1,122 @@ +# coding: utf-8 + +# Copyright 2016 Cisco Systems, Inc. +# Copyright 2015 IBM Corp. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import babel.messages.catalog as catalog +import os +from subprocess import call + +from django.conf import settings +from django.core.management.base import BaseCommand +from django.utils import translation + +LANGUAGE_CODES = [language[0] for language in settings.LANGUAGES + if language[0] != 'en'] +POTFILE = "{module}/locale/{domain}.pot" +POFILE = "{module}/locale/{locale}/LC_MESSAGES/{domain}.po" + + +def translate(segment): + prefix = u"" + # When the id starts with a newline the mo compiler enforces that + # the translated message must also start with a newline. Make + # sure that doesn't get broken when prepending the bracket. + if segment.startswith('\n'): + prefix = u"\n" + orig_size = len(segment) + # Add extra expansion space based on recommendation from + # http://www-01.ibm.com/software/globalization/guidelines/a3.html + if orig_size < 20: + multiplier = 1 + elif orig_size < 30: + multiplier = 0.8 + elif orig_size < 50: + multiplier = 0.6 + elif orig_size < 70: + multiplier = 0.4 + else: + multiplier = 0.3 + extra_length = int(max(0, (orig_size * multiplier) - 10)) + extra_chars = "~" * extra_length + return u"{0}[~{1}~您好яшçあ{2}]".format(prefix, segment, extra_chars) + + +class Command(BaseCommand): + help = 'Update a translation catalog for a specified language' + + def add_arguments(self, parser): + parser.add_argument('-l', '--language', choices=LANGUAGE_CODES, + default=LANGUAGE_CODES, nargs='+', + help=("The language code(s) to pseudo translate")) + parser.add_argument('-m', '--module', type=str, nargs='+', + default=['openstack_dashboard', 'horizon'], + help=("The target python module(s) to extract " + "strings from")) + parser.add_argument('-d', '--domain', choices=['django', 'djangojs'], + nargs='+', default=['django', 'djangojs'], + help="Domain(s) of the .POT file") + parser.add_argument('--pseudo', action='store_true', + help=("Creates a pseudo translation for the " + "specified locale, to check for " + "translatable string coverage")) + + def handle(self, *args, **options): + for module in options['module']: + for domain in options['domain']: + potfile = POTFILE.format(module=module, domain=domain) + + for language in options['language']: + # Get the locale code for the language code given and + # work around broken django conversion function + locales = {'ko': 'ko_KR', 'pl': 'pl_PL', 'tr': 'tr_TR'} + locale = locales.get(language, + translation.to_locale(language)) + pofile = POFILE.format(module=module, locale=locale, + domain=domain) + + # If this isn't a pseudo translation, execute pybabel + if not options['pseudo']: + if not os.path.exists(pofile): + with open(pofile, 'wb') as fobj: + fobj.write(b'') + + cmd = ('pybabel update -l {locale} -i {potfile} ' + '-o {pofile}').format(locale=locale, + potfile=potfile, + pofile=pofile) + call(cmd, shell=True) + continue + + # Pseudo translation logic + with open(potfile, 'r') as f: + pot_cat = pofile.read_po(f, ignore_obsolete=True) + + new_cat = catalog.Catalog(locale=locale, + last_translator="pseudo.py", + charset="utf-8") + num_plurals = new_cat.num_plurals + + for msg in pot_cat: + if msg.pluralizable: + msg.string = [ + translate(u"{}:{}".format(i, msg.id[0])) + for i in range(num_plurals)] + else: + msg.string = translate(msg.id) + new_cat[msg.id] = msg + + with open(pofile, 'w') as f: + pofile.write_po(f, new_cat, ignore_obsolete=True) diff --git a/package.json b/package.json index 495f8766c5..135295ba86 100644 --- a/package.json +++ b/package.json @@ -19,7 +19,7 @@ "karma-threshold-reporter": "0.1.15" }, "scripts": { - "postinstall": "if [ ! -d .venv ]; then tox -epy27 --notest; fi", + "postinstall": "if [ ! -d .tox/npm ]; then tox -e npm --notest; fi", "test": "karma start horizon/karma.conf.js --single-run && karma start openstack_dashboard/karma.conf.js --single-run", "lint": "eslint --no-color openstack_dashboard/static horizon/static openstack_dashboard/dashboards/*/static", "lintq": "eslint --quiet openstack_dashboard/static horizon/static openstack_dashboard/dashboards/*/static" diff --git a/releasenotes/notes/bp/enhance-tox-26f73a048b88df2f.yaml b/releasenotes/notes/bp/enhance-tox-26f73a048b88df2f.yaml new file mode 100644 index 0000000000..9d9f0eff0b --- /dev/null +++ b/releasenotes/notes/bp/enhance-tox-26f73a048b88df2f.yaml @@ -0,0 +1,11 @@ +--- +features: + - The hard-coded run_tests commands for extracting translatable strings and + updating message catalogs have been ported to django management commands + as extract_messages and update_catalog. These accept several parameters + to make them easier to use with downstream customisations and string + modifications, but the default behaviour is the same as before. +deprecations: + - The run_tests.sh script is now deprecated and all functionality has + been provided by either tox or manage.py. run_tests will be removed + in Queens (13.0). diff --git a/tools/pseudo.py b/tools/pseudo.py index fbb14cb76c..4a5ef1a0df 100755 --- a/tools/pseudo.py +++ b/tools/pseudo.py @@ -19,6 +19,10 @@ import argparse import babel.messages.catalog as catalog import babel.messages.pofile as pofile +# NOTE: This implementation has been superseded by the pseudo_translate +# management command, and will be removed in Queens (13.0) when run_tests.sh +# is also removed. + def translate(segment): prefix = u"" diff --git a/tools/unit_tests.sh b/tools/unit_tests.sh new file mode 100755 index 0000000000..edff8acfcb --- /dev/null +++ b/tools/unit_tests.sh @@ -0,0 +1,30 @@ +# Uses envpython and toxinidir from tox run to construct a test command +testcommand="${1} ${2}/manage.py test" +posargs="${@:3}" + +# Attempt to identify if any of the arguments passed from tox is a test subset +if [ -n "$posargs" ]; then + for arg in "$posargs" + do + if [ ${arg:0:1} != "-" ]; then + subset=$arg + fi + done +fi + +# If we are running a test subset, supply the correct settings file. +# If not, simply run the entire test suite. +if [ -n "$subset" ]; then + project="${subset%%.*}" + + if [ $project == "horizon" ]; then + $testcommand --settings=horizon.test.settings $posargs + elif [ $project == "openstack_dashboard" ]; then + $testcommand --settings=openstack_dashboard.test.settings \ + --exclude-dir=openstack_dashboard/test/integration_tests $posargs + fi +else + $testcommand horizon --settings=horizon.test.settings $posargs + $testcommand openstack_dashboard --settings=openstack_dashboard.test.settings \ + --exclude-dir=openstack_dashboard/test/integration_tests $posargs +fi diff --git a/tox.ini b/tox.ini index 72405b1106..3b30f3cd5f 100644 --- a/tox.ini +++ b/tox.ini @@ -1,105 +1,64 @@ [tox] -envlist = pep8,py27dj18,py27,py34,py35,releasenotes -minversion = 1.6 +envlist = pep8,py27dj{18,19,110},py34,py35,releasenotes,npm +minversion = 2.3.2 skipsdist = True [testenv] -basepython=python2.7 install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages} setenv = VIRTUAL_ENV={envdir} INTEGRATION_TESTS=0 - SELENIUM_HEADLESS=0 - SELENIUM_PHANTOMJS=0 NOSE_WITH_OPENSTACK=1 NOSE_OPENSTACK_SHOW_ELAPSED=1 +whitelist_externals = + bash deps = -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -whitelist_externals = - bash commands = - # Try to detect whether a limited test suite is being specified and if so - # direct the testing to that suite's project; otherwise run the full suite - # in both horizon and openstack_dashboard "projects". - bash -c 'project=`echo {posargs} | cut -d. -f1`; \ - if [ -z "$project" ]; then \ - EXIT_STATUS=0; \ - {envpython} {toxinidir}/manage.py test horizon --settings=horizon.test.settings {posargs} || EXIT_STATUS=$?; \ - {envpython} {toxinidir}/manage.py test openstack_dashboard --settings=openstack_dashboard.test.settings --exclude-dir=openstack_dashboard/test/integration_tests {posargs} || EXIT_STATUS=$?; \ - exit $EXIT_STATUS; \ - else \ - {envpython} {toxinidir}/manage.py test {posargs} --settings=$project.test.settings --exclude-dir=openstack_dashboard/test/integration_tests; \ - fi' - -# Django-1.8 is LTS -[testenv:py27dj18] -commands = - pip install django>=1.8,<1.9 - {envpython} {toxinidir}/manage.py test horizon --settings=horizon.test.settings {posargs} - {envpython} {toxinidir}/manage.py test openstack_dashboard --settings=openstack_dashboard.test.settings --exclude-dir=openstack_dashboard/test/integration_tests {posargs} + docs: sphinx-build -W -b html doc/source doc/build/html + horizon: {envpython} {toxinidir}/manage.py test --settings=horizon.test.settings {posargs} + manage: {envpython} {toxinidir}/manage.py {posargs} + py27: {[unit_tests]commands} + py27dj18: {[unit_tests]commands} + py34: {[unit_tests]commands} + py35: {[unit_tests]commands} + openstack_dashboard: {envpython} {toxinidir}/manage.py test --settings=openstack_dashboard.test.settings {posargs} + releasenotes: sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html + runserver: {envpython} {toxinidir}/manage.py runserver {posargs} + venv: {posargs} [testenv:py34] -basepython = python3.4 setenv = + PYTHONUNBUFFERED = 1 {[testenv]setenv} - PYTHONUNBUFFERED=1 +commands = {[unit_tests]commands} [testenv:py35] -basepython = python3.5 setenv = + PYTHONUNBUFFERED = 1 {[testenv]setenv} - PYTHONUNBUFFERED=1 +commands = {[unit_tests]commands} + +[testenv:py27dj19] +commands = + pip install -U django>=1.9,<1.10 + {[unit_tests]commands} + +[testenv:py27dj110] +commands = + pip install -U django>=1.10,<1.11 + {[unit_tests]commands} + +[unit_tests] +commands = bash {toxinidir}/tools/unit_tests.sh {envpython} {toxinidir} {posargs} [testenv:pep8] usedevelop = True -whitelist_externals = - git - rm -setenv = - {[testenv]setenv} - DJANGO_SETTINGS_MODULE=openstack_dashboard.test.settings commands = - {[testenv:extractmessages_check]commands} - {[testenv:docs]commands} + {envpython} {toxinidir}/manage.py extract_messages --check-only flake8 -[testenv:extractmessages] -usedevelop = True -setenv = - {[testenv]setenv} -commands = - {envpython} {toxinidir}/setup.py extract_messages -F babel-django.cfg -o horizon/locale/django.pot --input-dirs horizon/ - {envpython} {toxinidir}/setup.py extract_messages -F babel-djangojs.cfg -o horizon/locale/djangojs.pot --input-dirs horizon/ - {envpython} {toxinidir}/setup.py extract_messages -F babel-django.cfg -o openstack_dashboard/locale/django.pot --input-dirs openstack_dashboard/ - {envpython} {toxinidir}/setup.py extract_messages -F babel-djangojs.cfg -o openstack_dashboard/locale/djangojs.pot --input-dirs openstack_dashboard/ - -[testenv:extractmessages_check] -# Only checks to see if translation files can be extracted and cleans afterwards -usedevelop = True -whitelist_externals = - rm -setenv = - {[testenv]setenv} -commands = - {[testenv:extractmessages]commands} - rm horizon/locale/django.pot - rm horizon/locale/djangojs.pot - rm openstack_dashboard/locale/django.pot - rm openstack_dashboard/locale/djangojs.pot - -[testenv:compilemessages] -usedevelop = False -commands = - /bin/bash {toxinidir}/run_tests.sh --compilemessages -N - -[testenv:venv] -commands = {posargs} - -[testenv:manage] -# Env to launch manage.py commands -commands = {envpython} {toxinidir}/manage.py {posargs} - [testenv:cover] commands = coverage erase @@ -108,13 +67,28 @@ commands = coverage xml coverage html -[testenv:py27dj19] -commands = pip install django>=1.9,<1.10 - /bin/bash run_tests.sh -N --no-pep8 {posargs} +[testenv:selenium] +setenv = + {[testenv]setenv} + WITH_SELENIUM=1 + SKIP_UNITTESTS=1 +commands = {[unit_tests]commands} -[testenv:py27dj110] -commands = pip install django>=1.10,<1.11 - /bin/bash run_tests.sh -N --no-pep8 {posargs} +[testenv:selenium-headless] +setenv = + {[testenv]setenv} + SELENIUM_HEADLESS=1 + WITH_SELENIUM=1 + SKIP_UNITTESTS=1 +commands = {[unit_tests]commands} + +[testenv:selenium-phantomjs] +setenv = + {[testenv]setenv} + SELENIUM_PHANTOMJS=1 + WITH_SELENIUM=1 + SKIP_UNITTESTS=1 +commands = {[unit_tests]commands} [testenv:py27integration] # Run integration tests only @@ -134,16 +108,6 @@ commands = npm install npm run {posargs:test} -[testenv:docs] -setenv = DJANGO_SETTINGS_MODULE=openstack_dashboard.test.settings -commands = sphinx-build -W -b html doc/source doc/build/html - -[testenv:releasenotes] -commands = sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html - -[testenv:runserver] -commands = {envpython} {toxinidir}/manage.py runserver {posargs} - [testenv:tests_system_packages] # Provide an environment for system packagers that dont want anything from pip # Any extra deps needed for this env can be passed by setting TOX_EXTRA_DEPS @@ -153,8 +117,7 @@ passenv = TOX_EXTRA_DEPS deps = commands = pip install -U {env:TOX_EXTRA_DEPS:} - {envpython} {toxinidir}/manage.py test horizon --settings=horizon.test.settings {posargs} - {envpython} {toxinidir}/manage.py test openstack_dashboard --settings=openstack_dashboard.test.settings {posargs} + {[unit_tests]commands} [flake8] exclude = .venv,.git,.tox,dist,*lib/python*,*egg,build,panel_template,dash_template,local_settings.py,*/local/*,*/test/test_plugins/*,.ropeproject,node_modules