Commit Graph

46 Commits

Author SHA1 Message Date
Dr. Jens Harbott a014b0bda9 Update sitemap generator with current series
This hasn't been updated in four years, we need to get current content
into the sitemap and drop unmaintained releases.

Change-Id: If2f9a3459f7570b92328002ca8c13ebcca67f54c
2024-03-28 16:26:31 +01:00
Hervé Beraud fbe3f2fe7c Adding pre-commit
Introduced changes:
- pre-commit config and rules
- Add pre-commit to pep8 gate, Flake8 is covered in the pre-commit hooks.
- Applying fixes for pre-commit compliance in all code.

Also commit hash will be used instead of version tags in pre-commit to
prevend arbitrary code from running in developer's machines.

pre-commit will be used to:
- trailing whitespace;
- Replaces or checks mixed line ending (mixed-line-ending);
- Forbid files which have a UTF-8 byte-order marker (check-byte-order-marker);
- Checks that non-binary executables have a proper
  shebang (check-executables-have-shebangs);
- Check for files that contain merge conflict strings (check-merge-conflict);
- Check for debugger imports and py37+ breakpoint()
  calls in python source (debug-statements);
- Attempts to load all yaml files to verify syntax (check-yaml);
- Run flake8 checks (flake8) (local)

For further details about tests please refer to:
https://github.com/pre-commit/pre-commit-hooks

Change-Id: I7ac1599e903577e28fb64bb07a6b984e1ff8a023
Signed-off-by: Moisés Guimarães de Medeiros <moguimar@redhat.com>
2020-09-15 15:05:13 +02:00
Andreas Jaeger 8cca4479b6 Update sitemap
* Update releases: Newton is unmainted; train, ussuri are relased
* We have redirects to some sites, ignore them
* Remove python2 code
* Use https as starting point

Switch repo also to victoria PTI jobs.

Change-Id: I744e725b32a797b0e208ac7e81afe4b8d3578751
2020-06-21 21:48:53 +02:00
Andreas Jaeger b9d280acce Update sitemap for Python3
Fix sitemap code to work under Python3.

Change-Id: I7aad21f612047ce264b93460f7604b63fa951cab
2020-03-24 12:04:33 +01:00
Andreas Jaeger 06d7ac27b7 Update docs building
Use sphinx-build and cleanup unused config.
Switch to openstackdocstheme 1.20.0 and remove obsolete settings from
conf.py files.

Update some RST files, they had wrong indentation.

Change-Id: Iaad2841db809f8a343fb8b1031cf8d0587d70442
2019-08-11 09:48:06 +02:00
zhouxinyong e1cd64caef Advancing the protocal of the website to HTTPS in README.rst.
Change-Id: I42c644e1e76c161ef0bd7473bf3996cb485f03e4
2018-11-14 01:11:46 +08:00
Andreas Jaeger a3fdc40659 Update sitemap file
Add new releases, add already stein to not forget about it.

Update regex for releases, we do not publish only to /ocata/
but also to /PROJECT/ocata - tread all these the same.

Change-Id: I8ef454f779904d91e5e79c83c488dcba42ba2150
2018-09-09 12:00:39 +02:00
Nguyen Hung Phuong 8625096bce Clean imports in code
In some part in the code we import objects. In the Openstack style guidelines
they recommend to import only modules. [1]: "Do not import objects, only modules".

[1] https://docs.openstack.org/hacking/0.10.3/

Change-Id: I49452e41f9674e6ebfcc1577d59d21750e957677
2018-02-21 13:38:16 +07:00
melissaml c67afa0246 Fix to use "." to source script files
Adhering to coding conventions. Refer to ``Code conventions`` at
https://docs.openstack.org/contributor-guide/ for details.

Change-Id: I01a66baf06c1f4358b2b09eab7782c05ed0ab4cb
2017-08-29 02:18:04 +08:00
Jenkins ef5a4cd893 Merge "Do not add EOL releases to sitemap" 2017-07-31 14:09:03 +00:00
Jenkins 7cf3852d9e Merge "Change priorities for sitemap" 2017-07-31 14:08:57 +00:00
Jenkins f1aeebc81f Merge "Document installation of scrapy via pip" 2017-07-31 14:05:13 +00:00
Andreas Jaeger fb838cf897 Do not add EOL releases to sitemap
Block out EOL releases from the sitemap.

also, lower value for lowest prio.

Change-Id: I1500ae1d602b9471cbacb62854f9c53f8378cb79
2017-07-28 20:30:33 +02:00
Andreas Jaeger 6898821aeb Change priorities for sitemap
Downgrade old EOL releases, give current maintained releases and
development versions a higher priority.

this needs changes to the test framework, update mocking. Also, rename
URL to be docs.openstack.org and not .com.

Change-Id: I2c2c0408e203a65a9541baaf55ffe60694463975
2017-07-28 20:30:06 +02:00
Andreas Jaeger d4c18f5683 sitemap: Add ocata
Handle ocata as older release.

Change-Id: I7437faf717ca277e2ba72ef38e65a2979af11346
2017-07-25 20:02:40 +02:00
Christian Berendt 9c10de3687 Document installation of scrapy via pip
requirements.txt files was dropped with I22c018149b2eefde6ca5c38c22ac06886fe9a7a8.

Change-Id: I8a3219e2a2ac5fb34172225d3322ea13e21da80d
2017-07-03 17:27:00 +02:00
Brian Moss d3bc42483a Update sitemap tests
- add py3 to tox.ini (gate already tests py3)
- move all tests to $GITROOT/test so they can all run
  through testr
- add scrapy to test-requirements.txt to support sitemap tests
- move tests from test_items.py to test_sitemap_file.py
- fix broken sitemap tests
- add newton to list of old releases in sitemap_file.py
- ignore flake8 H101 as it returns false positives for Sphinx conf.py
- Use openstackdocstheme for docs
- Update sitemap README
- Restructure repo docs
- fix minor style issues

Change-Id: I22c018149b2eefde6ca5c38c22ac06886fe9a7a8
2017-04-29 10:36:27 +10:00
Andreas Jaeger c99bb681b2 Mitaka is an old release now
Update settings for mitaka

Change-Id: I122aafd4a3136da209ccd26d13d9060a134ce727
2016-10-06 14:33:34 +02:00
Jenkins 381cae363e Merge "Move SitemapItem class into generator.spiders.sitemap_file" 2016-10-06 12:31:37 +00:00
Christian Berendt 8a8411e3ff Improve the README file of the sitemap generator
Change-Id: I4143b4dc7846b4dcdb873921a0ee75a62cc8f8ef
2016-10-06 13:57:49 +02:00
Christian Berendt 92a4fb6527 Move SitemapItem class into generator.spiders.sitemap_file
This will solve the following issue:

ImportError: No module named sitemap.generator

Change-Id: I449f0fb0c87613c36447cca94f4f5857f3d31afa
2016-10-06 13:49:43 +02:00
gecong1973 52d9c52723 Change assertTrue(isinstance()) by optimal assert
Some of tests use different method of
assertTrue(isinstance(A, B)) or assertEqual(type(A), B).
The correct way is to use assertIsInstance(A, B) provided
by testtools

Change-Id: Ib8dec9d44ce66b1b7ff1486a234fae73d29f23ff
2016-09-30 16:30:29 +08:00
Percila c050836a8f doc-tools unit tests
Renamed sitemap file to avoid module name conflict
when importing at the sitemap unittest

Added py.test tox environment

Change-Id: I94480e374b29802414b62591a51c04ecd804905e
Closes-Bug: #1387716
2016-08-03 07:05:51 +00:00
Rebecca Finn ad80212a90 Remove Icehouse and Juno content from sitemap
Icehouse and Juno docs should not be accesed, so this updates the sitemap
generator tool to skip them. The sitemap can then be regenerated.

Partial-Bug: #1602647
Change-Id: I3117b114881c3a12b6a1a9565a0a720c9c8678b1
2016-07-14 16:49:28 +00:00
Christian Berendt f6715a3ba5 [sitemap] mark liberty as an old release
Change-Id: I8a16a4dd1051a351750b448e59e518e8223629b9
2016-04-05 12:59:58 +02:00
Christian Berendt 44243238ac [sitemap] do not add empty URLs to the list of start URLs
This will solve the 'ValueError: Missing scheme in request url'
exception.

Change-Id: I295b2a519668b36413ec2ba4ef3ba9dfaac6e859
2016-01-22 14:58:44 +01:00
Christian Berendt 8ebb376bb8 [sitemap] set higher priority for files of the current release
Change-Id: I9dbaa787354582f2f766fcce58aff95766d242c7
2016-01-13 11:40:52 +01:00
Jenkins c47df24295 Merge "[sitemap] respect the robots.txt policies" 2015-10-20 12:15:50 +00:00
Jenkins 9c5b3e7171 Merge "[sitemap] introduce attribute to define start URLs" 2015-10-20 12:15:38 +00:00
Jenkins 47c1209f61 Merge "[sitemap] if last-modified not set in header use current date" 2015-10-20 01:12:12 +00:00
Christian Berendt 815fcb5bb6 [sitemap] introduce attribute to define start URLs
Change-Id: I09018a824310ef48f6b62f04f09d6fd704808119
Closes-bug: #1506601
2015-10-16 11:22:05 +02:00
Christian Berendt 8f05ac1b0a [sitemap] respect the robots.txt policies
Change-Id: I219eff6241479c3c883c55079b48a88591a2a99d
2015-10-16 07:59:33 +02:00
Christian Berendt b7ed3589d3 [sitemap] remove deprecated URL from settings
Change-Id: I2c3b1a7757b0a01b5822b827b3445cd46bdbc57c
2015-10-16 07:32:49 +02:00
Christian Berendt 6b5b7d530a [sitemap] if last-modified not set in header use current date
This solves the following issue when last-modified is not set in the
response header:

---snip---
KeyError: 'Last-Modified'
---snap---

Change-Id: I372f9638f6b5006ff250b9dbe0e75cc1556d18c2
2015-10-16 07:28:10 +02:00
Christian Berendt 02d0060144 [sitemap] increase the DOWNLOAD_WARNSIZE
The default value for the DOWNLOAD_WARNSIZE setting is 33554432 (32MB).
This patch will double the value. This solves the following warning:

---snip---
WARNING: Received (37889579) bytes larger than download warn size
(33554432).
---snap---

Change-Id: I5be4bcefa6fe2eb474901bb958669978dd6e1fef
2015-10-02 10:38:45 +02:00
Christian Berendt 94d69f1868 [sitemap] resolve remaining ScrapyDeprecationWarning exceptions
* SgmlLinkExtractor is deprecated
* Module `scrapy.contrib.exporter` is deprecated

Change-Id: Ie3207a537abfd2b75602000fb4fef8f4c1e8ba56
2015-10-02 10:34:13 +02:00
Christian Berendt 013fcd3665 [sitemap] add a requirements file
Change-Id: I872944d35b5b78a4e966c4c9f39174aaa67aa91e
2015-10-02 08:28:37 +02:00
Jenkins 89624681c1 Merge "[sitemap] transform README from Markdown to reStructuredText" 2015-08-22 13:23:53 +00:00
Christian Berendt 5b21d5f8e2 [sitemap] transform README from Markdown to reStructuredText
Change-Id: Icdd8269323075cb3b9a7f81d9118cf57b36a7267
2015-08-22 10:10:07 +02:00
Christian Berendt 8a2df7b6ef [sitemap] resolve several ScrapyDeprecationWarning exceptions
Change-Id: Ie87e6c316eecd45006332088912b31809b4c9679
2015-08-22 10:06:59 +02:00
Andreas Jaeger a21cf5f375 Remove /draft from sitemap
Do not include /draft in sitemap.

Paritally implements bp draft-publishing

Change-Id: Iddfef2933c7a397da6f084da1285716b0078ea06
2015-04-18 09:43:10 +02:00
Anne Gentle b4f43bdcaa Update sitemap generator
- Removes grizzly, adds juno
- Removes /api/ just like /trunk/ now that <service>-api repos have moved
to openstack-attic

Change-Id: I4b5dea2379d58b3ed506f70c4a9539a64f892f8e
2015-02-13 23:46:09 -06:00
Andreas Jaeger a6fbc259b6 Improve sitemap generation
Filter out any trunk links

Change-Id: Icf35370237e0209c836ed8883040bf6fe4c9a7c0
2014-06-27 14:25:14 +02:00
Christian Berendt d449a6bcd2 Bump hacking to 0.9.x series
Require at least 0.9.1 because 0.9.0. had a minor bug.

This change also fixes all found issues.

Rewrote some docstrings to fit H405.

Change-Id: Ifeef11b783dbe70b2652d2b9ea29d5b20f69f2ce
2014-06-12 09:17:22 +02:00
Christian Berendt 31c25891c1 script to generate the sitemap.xml for docs.openstack.org
This script crawls all available sites on http://docs.openstack.org and extracts
all URLs. Based on the URLs the script generates a sitemap for search engines
according to the protocol described at http://www.sitemaps.org/protocol.html.

Change-Id: Id7839d2048989da503d31e436455aea9bb4cdc1d
2014-05-29 01:29:18 +02:00
annegentle 40ccf58a78 Updates sitemap.xml for havana release links
Adds an XSLT that transforms a generated sitemap, removing folsom
and trunk links

backport: stable/havana

Change-Id: I4484d50fa60eb53e3155fe4e119c30e54b03832e
Partial-bug: 1191447
2013-10-21 18:46:55 -05:00