Unicode content from disk must become bytes
While trying to figure out some tests for the fix in #203 it became clear that the handling of encoded-on-disk-but-textual content when using <@ to load content was not sufficiently robust in both python 2 and 3. We want to load content to become unicode so that it is used internally correctly, for example in the replace_template handling used for substitutions, but when an HTTP request is made the content must become whatever the versioned equivalent of bytes is. Therefore the code has been changed to load non_binary content and decode it to six.text_type, do replace_template on it, and then turn the body, if it is not already bytes, into bytes. This is verified by adjusting data.json to have some utf8 in it and adding and loading a utf8.txt example file.
This commit is contained in:
parent
d8fe9f801b
commit
c697b533e0
|
@ -436,6 +436,11 @@ class HTTPTestCase(testtools.TestCase):
|
|||
else:
|
||||
body = ''
|
||||
|
||||
# ensure body is bytes, encoding as UTF-8 because that's
|
||||
# what we do here
|
||||
if isinstance(body, six.text_type):
|
||||
body = body.encode('UTF-8')
|
||||
|
||||
if test['poll']:
|
||||
count = test['poll'].get('count', 1)
|
||||
delay = test['poll'].get('delay', 1)
|
||||
|
@ -473,11 +478,7 @@ class HTTPTestCase(testtools.TestCase):
|
|||
if data.startswith('<@'):
|
||||
info = self._load_data_file(data.replace('<@', '', 1))
|
||||
if utils.not_binary(content_type):
|
||||
try:
|
||||
info = str(info, 'UTF-8')
|
||||
except TypeError:
|
||||
info = info.encode('UTF-8')
|
||||
data = info
|
||||
data = six.text_type(info, 'UTF-8')
|
||||
else:
|
||||
return info
|
||||
else:
|
||||
|
|
|
@ -1 +1 @@
|
|||
{"foo": {"bar": 1}}
|
||||
{"foo": {"bár": 1}}
|
||||
|
|
|
@ -34,7 +34,7 @@ tests:
|
|||
content-type: application/json
|
||||
data: <@data.json
|
||||
response_json_paths:
|
||||
foo.bar: 1
|
||||
foo['bár']: 1
|
||||
|
||||
- name: load image file
|
||||
url: /
|
||||
|
@ -42,3 +42,10 @@ tests:
|
|||
request_headers:
|
||||
content-type: image/png
|
||||
data: <@kitten.png
|
||||
|
||||
- name: load encoded text
|
||||
url: /
|
||||
method: POST
|
||||
request_headers:
|
||||
content-type: text/plain
|
||||
data: <@utf8.txt
|
||||
|
|
|
@ -0,0 +1,222 @@
|
|||
<h1>Unicode Demo</h1>
|
||||
|
||||
<p>Taken from <a
|
||||
href="http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt">http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt</a></p>
|
||||
|
||||
<pre>
|
||||
|
||||
$ENVIRON['HOME']
|
||||
|
||||
UTF-8 encoded sample plain-text file
|
||||
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
|
||||
|
||||
Markus Kuhn [ˈmaʳkʊs kuːn] <http://www.cl.cam.ac.uk/~mgk25/> — 2002-07-25
|
||||
|
||||
|
||||
The ASCII compatible UTF-8 encoding used in this plain-text file
|
||||
is defined in Unicode, ISO 10646-1, and RFC 2279.
|
||||
|
||||
|
||||
Using Unicode/UTF-8, you can write in emails and source code things such as
|
||||
|
||||
Mathematics and sciences:
|
||||
|
||||
∮ E⋅da = Q, n → ∞, ∑ f(i) = ∏ g(i), ⎧⎡⎛┌─────┐⎞⎤⎫
|
||||
⎪⎢⎜│a²+b³ ⎟⎥⎪
|
||||
∀x∈ℝ: ⌈x⌉ = −⌊−x⌋, α ∧ ¬β = ¬(¬α ∨ β), ⎪⎢⎜│───── ⎟⎥⎪
|
||||
⎪⎢⎜⎷ c₈ ⎟⎥⎪
|
||||
ℕ ⊆ ℕ₀ ⊂ ℤ ⊂ ℚ ⊂ ℝ ⊂ ℂ, ⎨⎢⎜ ⎟⎥⎬
|
||||
⎪⎢⎜ ∞ ⎟⎥⎪
|
||||
⊥ < a ≠ b ≡ c ≤ d ≪ ⊤ ⇒ (⟦A⟧ ⇔ ⟪B⟫), ⎪⎢⎜ ⎲ ⎟⎥⎪
|
||||
⎪⎢⎜ ⎳aⁱ-bⁱ⎟⎥⎪
|
||||
2H₂ + O₂ ⇌ 2H₂O, R = 4.7 kΩ, ⌀ 200 mm ⎩⎣⎝i=1 ⎠⎦⎭
|
||||
|
||||
Linguistics and dictionaries:
|
||||
|
||||
ði ıntəˈnæʃənəl fəˈnɛtık əsoʊsiˈeıʃn
|
||||
Y [ˈʏpsilɔn], Yen [jɛn], Yoga [ˈjoːgɑ]
|
||||
|
||||
APL:
|
||||
|
||||
((V⍳V)=⍳⍴V)/V←,V ⌷←⍳→⍴∆∇⊃‾⍎⍕⌈
|
||||
|
||||
Nicer typography in plain text files:
|
||||
|
||||
╔══════════════════════════════════════════╗
|
||||
║ ║
|
||||
║ • ‘single’ and “double” quotes ║
|
||||
║ ║
|
||||
║ • Curly apostrophes: “We’ve been here” ║
|
||||
║ ║
|
||||
║ • Latin-1 apostrophe and accents: '´` ║
|
||||
║ ║
|
||||
║ • ‚deutsche‘ „Anführungszeichen“ ║
|
||||
║ ║
|
||||
║ • †, ‡, ‰, •, 3–4, —, −5/+5, ™, … ║
|
||||
║ ║
|
||||
║ • ASCII safety test: 1lI|, 0OD, 8B ║
|
||||
║ ╭─────────╮ ║
|
||||
║ • the euro symbol: │ 14.95 € │ ║
|
||||
║ ╰─────────╯ ║
|
||||
╚══════════════════════════════════════════╝
|
||||
|
||||
Combining characters:
|
||||
|
||||
STARGΛ̊TE SG-1, a = v̇ = r̈, a⃑ ⊥ b⃑
|
||||
|
||||
Greek (in Polytonic):
|
||||
|
||||
The Greek anthem:
|
||||
|
||||
Σὲ γνωρίζω ἀπὸ τὴν κόψη
|
||||
τοῦ σπαθιοῦ τὴν τρομερή,
|
||||
σὲ γνωρίζω ἀπὸ τὴν ὄψη
|
||||
ποὺ μὲ βία μετράει τὴ γῆ.
|
||||
|
||||
᾿Απ᾿ τὰ κόκκαλα βγαλμένη
|
||||
τῶν ῾Ελλήνων τὰ ἱερά
|
||||
καὶ σὰν πρῶτα ἀνδρειωμένη
|
||||
χαῖρε, ὦ χαῖρε, ᾿Ελευθεριά!
|
||||
|
||||
From a speech of Demosthenes in the 4th century BC:
|
||||
|
||||
Οὐχὶ ταὐτὰ παρίσταταί μοι γιγνώσκειν, ὦ ἄνδρες ᾿Αθηναῖοι,
|
||||
ὅταν τ᾿ εἰς τὰ πράγματα ἀποβλέψω καὶ ὅταν πρὸς τοὺς
|
||||
λόγους οὓς ἀκούω· τοὺς μὲν γὰρ λόγους περὶ τοῦ
|
||||
τιμωρήσασθαι Φίλιππον ὁρῶ γιγνομένους, τὰ δὲ πράγματ᾿
|
||||
εἰς τοῦτο προήκοντα, ὥσθ᾿ ὅπως μὴ πεισόμεθ᾿ αὐτοὶ
|
||||
πρότερον κακῶς σκέψασθαι δέον. οὐδέν οὖν ἄλλο μοι δοκοῦσιν
|
||||
οἱ τὰ τοιαῦτα λέγοντες ἢ τὴν ὑπόθεσιν, περὶ ἧς βουλεύεσθαι,
|
||||
οὐχὶ τὴν οὖσαν παριστάντες ὑμῖν ἁμαρτάνειν. ἐγὼ δέ, ὅτι μέν
|
||||
ποτ᾿ ἐξῆν τῇ πόλει καὶ τὰ αὑτῆς ἔχειν ἀσφαλῶς καὶ Φίλιππον
|
||||
τιμωρήσασθαι, καὶ μάλ᾿ ἀκριβῶς οἶδα· ἐπ᾿ ἐμοῦ γάρ, οὐ πάλαι
|
||||
γέγονεν ταῦτ᾿ ἀμφότερα· νῦν μέντοι πέπεισμαι τοῦθ᾿ ἱκανὸν
|
||||
προλαβεῖν ἡμῖν εἶναι τὴν πρώτην, ὅπως τοὺς συμμάχους
|
||||
σώσομεν. ἐὰν γὰρ τοῦτο βεβαίως ὑπάρξῃ, τότε καὶ περὶ τοῦ
|
||||
τίνα τιμωρήσεταί τις καὶ ὃν τρόπον ἐξέσται σκοπεῖν· πρὶν δὲ
|
||||
τὴν ἀρχὴν ὀρθῶς ὑποθέσθαι, μάταιον ἡγοῦμαι περὶ τῆς
|
||||
τελευτῆς ὁντινοῦν ποιεῖσθαι λόγον.
|
||||
|
||||
Δημοσθένους, Γ´ ᾿Ολυνθιακὸς
|
||||
|
||||
Georgian:
|
||||
|
||||
From a Unicode conference invitation:
|
||||
|
||||
გთხოვთ ახლავე გაიაროთ რეგისტრაცია Unicode-ის მეათე საერთაშორისო
|
||||
კონფერენციაზე დასასწრებად, რომელიც გაიმართება 10-12 მარტს,
|
||||
ქ. მაინცში, გერმანიაში. კონფერენცია შეჰკრებს ერთად მსოფლიოს
|
||||
ექსპერტებს ისეთ დარგებში როგორიცაა ინტერნეტი და Unicode-ი,
|
||||
ინტერნაციონალიზაცია და ლოკალიზაცია, Unicode-ის გამოყენება
|
||||
ოპერაციულ სისტემებსა, და გამოყენებით პროგრამებში, შრიფტებში,
|
||||
ტექსტების დამუშავებასა და მრავალენოვან კომპიუტერულ სისტემებში.
|
||||
|
||||
Russian:
|
||||
|
||||
From a Unicode conference invitation:
|
||||
|
||||
Зарегистрируйтесь сейчас на Десятую Международную Конференцию по
|
||||
Unicode, которая состоится 10-12 марта 1997 года в Майнце в Германии.
|
||||
Конференция соберет широкий круг экспертов по вопросам глобального
|
||||
Интернета и Unicode, локализации и интернационализации, воплощению и
|
||||
применению Unicode в различных операционных системах и программных
|
||||
приложениях, шрифтах, верстке и многоязычных компьютерных системах.
|
||||
|
||||
Thai (UCS Level 2):
|
||||
|
||||
Excerpt from a poetry on The Romance of The Three Kingdoms (a Chinese
|
||||
classic 'San Gua'):
|
||||
|
||||
[----------------------------|------------------------]
|
||||
๏ แผ่นดินฮั่นเสื่อมโทรมแสนสังเวช พระปกเกศกองบู๊กู้ขึ้นใหม่
|
||||
สิบสองกษัตริย์ก่อนหน้าแลถัดไป สององค์ไซร้โง่เขลาเบาปัญญา
|
||||
ทรงนับถือขันทีเป็นที่พึ่ง บ้านเมืองจึงวิปริตเป็นนักหนา
|
||||
โฮจิ๋นเรียกทัพทั่วหัวเมืองมา หมายจะฆ่ามดชั่วตัวสำคัญ
|
||||
เหมือนขับไสไล่เสือจากเคหา รับหมาป่าเข้ามาเลยอาสัญ
|
||||
ฝ่ายอ้องอุ้นยุแยกให้แตกกัน ใช้สาวนั้นเป็นชนวนชื่นชวนใจ
|
||||
พลันลิฉุยกุยกีกลับก่อเหตุ ช่างอาเพศจริงหนาฟ้าร้องไห้
|
||||
ต้องรบราฆ่าฟันจนบรรลัย ฤๅหาใครค้ำชูกู้บรรลังก์ ฯ
|
||||
|
||||
(The above is a two-column text. If combining characters are handled
|
||||
correctly, the lines of the second column should be aligned with the
|
||||
| character above.)
|
||||
|
||||
Ethiopian:
|
||||
|
||||
Proverbs in the Amharic language:
|
||||
|
||||
ሰማይ አይታረስ ንጉሥ አይከሰስ።
|
||||
ብላ ካለኝ እንደአባቴ በቆመጠኝ።
|
||||
ጌጥ ያለቤቱ ቁምጥና ነው።
|
||||
ደሀ በሕልሙ ቅቤ ባይጠጣ ንጣት በገደለው።
|
||||
የአፍ ወለምታ በቅቤ አይታሽም።
|
||||
አይጥ በበላ ዳዋ ተመታ።
|
||||
ሲተረጉሙ ይደረግሙ።
|
||||
ቀስ በቀስ፥ ዕንቁላል በእግሩ ይሄዳል።
|
||||
ድር ቢያብር አንበሳ ያስር።
|
||||
ሰው እንደቤቱ እንጅ እንደ ጉረቤቱ አይተዳደርም።
|
||||
እግዜር የከፈተውን ጉሮሮ ሳይዘጋው አይድርም።
|
||||
የጎረቤት ሌባ፥ ቢያዩት ይስቅ ባያዩት ያጠልቅ።
|
||||
ሥራ ከመፍታት ልጄን ላፋታት።
|
||||
ዓባይ ማደሪያ የለው፥ ግንድ ይዞ ይዞራል።
|
||||
የእስላም አገሩ መካ የአሞራ አገሩ ዋርካ።
|
||||
ተንጋሎ ቢተፉ ተመልሶ ባፉ።
|
||||
ወዳጅህ ማር ቢሆን ጨርስህ አትላሰው።
|
||||
እግርህን በፍራሽህ ልክ ዘርጋ።
|
||||
|
||||
Runes:
|
||||
|
||||
ᚻᛖ ᚳᚹᚫᚦ ᚦᚫᛏ ᚻᛖ ᛒᚢᛞᛖ ᚩᚾ ᚦᚫᛗ ᛚᚪᚾᛞᛖ ᚾᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ ᚹᛁᚦ ᚦᚪ ᚹᛖᛥᚫ
|
||||
|
||||
(Old English, which transcribed into Latin reads 'He cwaeth that he
|
||||
bude thaem lande northweardum with tha Westsae.' and means 'He said
|
||||
that he lived in the northern land near the Western Sea.')
|
||||
|
||||
Braille:
|
||||
|
||||
⡌⠁⠧⠑ ⠼⠁⠒ ⡍⠜⠇⠑⠹⠰⠎ ⡣⠕⠌
|
||||
|
||||
⡍⠜⠇⠑⠹ ⠺⠁⠎ ⠙⠑⠁⠙⠒ ⠞⠕ ⠃⠑⠛⠔ ⠺⠊⠹⠲ ⡹⠻⠑ ⠊⠎ ⠝⠕ ⠙⠳⠃⠞
|
||||
⠱⠁⠞⠑⠧⠻ ⠁⠃⠳⠞ ⠹⠁⠞⠲ ⡹⠑ ⠗⠑⠛⠊⠌⠻ ⠕⠋ ⠙⠊⠎ ⠃⠥⠗⠊⠁⠇ ⠺⠁⠎
|
||||
⠎⠊⠛⠝⠫ ⠃⠹ ⠹⠑ ⠊⠇⠻⠛⠹⠍⠁⠝⠂ ⠹⠑ ⠊⠇⠻⠅⠂ ⠹⠑ ⠥⠝⠙⠻⠞⠁⠅⠻⠂
|
||||
⠁⠝⠙ ⠹⠑ ⠡⠊⠑⠋ ⠍⠳⠗⠝⠻⠲ ⡎⠊⠗⠕⠕⠛⠑ ⠎⠊⠛⠝⠫ ⠊⠞⠲ ⡁⠝⠙
|
||||
⡎⠊⠗⠕⠕⠛⠑⠰⠎ ⠝⠁⠍⠑ ⠺⠁⠎ ⠛⠕⠕⠙ ⠥⠏⠕⠝ ⠰⡡⠁⠝⠛⠑⠂ ⠋⠕⠗ ⠁⠝⠹⠹⠔⠛ ⠙⠑
|
||||
⠡⠕⠎⠑ ⠞⠕ ⠏⠥⠞ ⠙⠊⠎ ⠙⠁⠝⠙ ⠞⠕⠲
|
||||
|
||||
⡕⠇⠙ ⡍⠜⠇⠑⠹ ⠺⠁⠎ ⠁⠎ ⠙⠑⠁⠙ ⠁⠎ ⠁ ⠙⠕⠕⠗⠤⠝⠁⠊⠇⠲
|
||||
|
||||
⡍⠔⠙⠖ ⡊ ⠙⠕⠝⠰⠞ ⠍⠑⠁⠝ ⠞⠕ ⠎⠁⠹ ⠹⠁⠞ ⡊ ⠅⠝⠪⠂ ⠕⠋ ⠍⠹
|
||||
⠪⠝ ⠅⠝⠪⠇⠫⠛⠑⠂ ⠱⠁⠞ ⠹⠻⠑ ⠊⠎ ⠏⠜⠞⠊⠊⠥⠇⠜⠇⠹ ⠙⠑⠁⠙ ⠁⠃⠳⠞
|
||||
⠁ ⠙⠕⠕⠗⠤⠝⠁⠊⠇⠲ ⡊ ⠍⠊⠣⠞ ⠙⠁⠧⠑ ⠃⠑⠲ ⠔⠊⠇⠔⠫⠂ ⠍⠹⠎⠑⠇⠋⠂ ⠞⠕
|
||||
⠗⠑⠛⠜⠙ ⠁ ⠊⠕⠋⠋⠔⠤⠝⠁⠊⠇ ⠁⠎ ⠹⠑ ⠙⠑⠁⠙⠑⠌ ⠏⠊⠑⠊⠑ ⠕⠋ ⠊⠗⠕⠝⠍⠕⠝⠛⠻⠹
|
||||
⠔ ⠹⠑ ⠞⠗⠁⠙⠑⠲ ⡃⠥⠞ ⠹⠑ ⠺⠊⠎⠙⠕⠍ ⠕⠋ ⠳⠗ ⠁⠝⠊⠑⠌⠕⠗⠎
|
||||
⠊⠎ ⠔ ⠹⠑ ⠎⠊⠍⠊⠇⠑⠆ ⠁⠝⠙ ⠍⠹ ⠥⠝⠙⠁⠇⠇⠪⠫ ⠙⠁⠝⠙⠎
|
||||
⠩⠁⠇⠇ ⠝⠕⠞ ⠙⠊⠌⠥⠗⠃ ⠊⠞⠂ ⠕⠗ ⠹⠑ ⡊⠳⠝⠞⠗⠹⠰⠎ ⠙⠕⠝⠑ ⠋⠕⠗⠲ ⡹⠳
|
||||
⠺⠊⠇⠇ ⠹⠻⠑⠋⠕⠗⠑ ⠏⠻⠍⠊⠞ ⠍⠑ ⠞⠕ ⠗⠑⠏⠑⠁⠞⠂ ⠑⠍⠏⠙⠁⠞⠊⠊⠁⠇⠇⠹⠂ ⠹⠁⠞
|
||||
⡍⠜⠇⠑⠹ ⠺⠁⠎ ⠁⠎ ⠙⠑⠁⠙ ⠁⠎ ⠁ ⠙⠕⠕⠗⠤⠝⠁⠊⠇⠲
|
||||
|
||||
(The first couple of paragraphs of "A Christmas Carol" by Dickens)
|
||||
|
||||
Compact font selection example text:
|
||||
|
||||
ABCDEFGHIJKLMNOPQRSTUVWXYZ /0123456789
|
||||
abcdefghijklmnopqrstuvwxyz £©µÀÆÖÞßéöÿ
|
||||
–—‘“”„†•…‰™œŠŸž€ ΑΒΓΔΩαβγδω АБВГДабвгд
|
||||
∀∂∈ℝ∧∪≡∞ ↑↗↨↻⇣ ┐┼╔╘░►☺♀ fi<>⑀₂ἠḂӥẄɐː⍎אԱა
|
||||
|
||||
Greetings in various languages:
|
||||
|
||||
Hello world, Καλημέρα κόσμε, コンニチハ
|
||||
|
||||
Box drawing alignment tests: █
|
||||
▉
|
||||
╔══╦══╗ ┌──┬──┐ ╭──┬──╮ ╭──┬──╮ ┏━━┳━━┓ ┎┒┏┑ ╷ ╻ ┏┯┓ ┌┰┐ ▊ ╱╲╱╲╳╳╳
|
||||
║┌─╨─┐║ │╔═╧═╗│ │╒═╪═╕│ │╓─╁─╖│ ┃┌─╂─┐┃ ┗╃╄┙ ╶┼╴╺╋╸┠┼┨ ┝╋┥ ▋ ╲╱╲╱╳╳╳
|
||||
║│╲ ╱│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╿ │┃ ┍╅╆┓ ╵ ╹ ┗┷┛ └┸┘ ▌ ╱╲╱╲╳╳╳
|
||||
╠╡ ╳ ╞╣ ├╢ ╟┤ ├┼─┼─┼┤ ├╫─╂─╫┤ ┣┿╾┼╼┿┫ ┕┛┖┚ ┌┄┄┐ ╎ ┏┅┅┓ ┋ ▍ ╲╱╲╱╳╳╳
|
||||
║│╱ ╲│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╽ │┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▎
|
||||
║└─╥─┘║ │╚═╤═╝│ │╘═╪═╛│ │╙─╀─╜│ ┃└─╂─┘┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▏
|
||||
╚══╩══╝ └──┴──┘ ╰──┴──╯ ╰──┴──╯ ┗━━┻━━┛ ▗▄▖▛▀▜ └╌╌┘ ╎ ┗╍╍┛ ┋ ▁▂▃▄▅▆▇█
|
||||
▝▀▘▙▄▟
|
||||
|
||||
</pre>
|
Loading…
Reference in New Issue