Improved performance of parse_file_contents() method

Before change Ic6236665f2d55b24a56a99120ac57fc2b18e32eb
policy file had been parsed by jsonutils.loads()

For some reasons jsonutils.loads() is much faster than
yaml.safe_load(). For example this script takes 30 seconds:

  s = '{"a": 3, "b": "c"}'
  for i in xrange(100000):
       yaml.safe_load(s)

And this script takes only 1 second!

  s = '{"a": 3, "b": "c"}'
  for i in xrange(100000):
      jsonutils.loads(s)

Moreover policy rules are presented in JSON format
by default (at least in Nova).

This patch proposes to parse file by jsonutils.loads().
If it was parsed - use parsed data. If it's failed - try to
parse with yaml.safe_load().

It will have no/minimum performance effect on real env because,
as I understand, we parse policy file only one time during
service start. But it will have great performance improvement
in our jenkins jobs.

Statistics for nova py27, py34 and functional jobs:

 * with this change I0c18e9746b742a2fff60795da8a5daddda7cf469
 * without this change Ie4af661f4ca16b63444692f3736835f9ce42adc1

Here was taken pure time of running "tox -e py27/py34/functional" commands

                 without (sec)         with (sec)
py27                  421                 240
py34                  330                 170
functional            662                 495

As you see performance of unit tests was increased almost in two times.

These changes covered by tests in test_policy.py file. There we
test method Rules.load() with invalid/valid JSON/YAML. As you
can see method Rules.load() directly uses parse_file_contents().

Change-Id: I43782d245d7652ba69613b26fe598ac79ec19929
This commit is contained in:
Sergey Nikitin 2016-11-29 20:33:11 +03:00
parent a66ee73e8c
commit f56b767ddd
1 changed files with 13 additions and 5 deletions

View File

@ -325,11 +325,19 @@ def parse_file_contents(data):
'policy_name2': 'policy2,...}
"""
try:
parsed = yaml.safe_load(data)
except yaml.YAMLError as e:
# For backwards-compatibility, convert yaml error to ValueError,
# which is what JSON loader raised.
raise ValueError(six.text_type(e))
# NOTE(snikitin): jsonutils.loads() is much faster than
# yaml.safe_load(). However jsonutils.loads() parses only JSON while
# yaml.safe_load() parses JSON and YAML. So here we try to parse data
# by jsonutils.loads() first. In case of failure yaml.safe_load()
# will be used instead.
parsed = jsonutils.loads(data)
except ValueError:
try:
parsed = yaml.safe_load(data)
except yaml.YAMLError as e:
# For backwards-compatibility, convert yaml error to ValueError,
# which is what JSON loader raised.
raise ValueError(six.text_type(e))
return parsed