Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | ***************** |
| 2 | Unicode HOWTO |
| 3 | ***************** |
| 4 | |
| 5 | :Release: 1.02 |
| 6 | |
| 7 | This HOWTO discusses Python's support for Unicode, and explains various problems |
| 8 | that people commonly encounter when trying to work with Unicode. |
| 9 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 10 | .. XXX fix it |
| 11 | .. warning:: |
| 12 | |
| 13 | This HOWTO has not yet been updated for Python 3000's string object changes. |
| 14 | |
| 15 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 16 | Introduction to Unicode |
| 17 | ======================= |
| 18 | |
| 19 | History of Character Codes |
| 20 | -------------------------- |
| 21 | |
| 22 | In 1968, the American Standard Code for Information Interchange, better known by |
| 23 | its acronym ASCII, was standardized. ASCII defined numeric codes for various |
| 24 | characters, with the numeric values running from 0 to |
| 25 | 127. For example, the lowercase letter 'a' is assigned 97 as its code |
| 26 | value. |
| 27 | |
| 28 | ASCII was an American-developed standard, so it only defined unaccented |
| 29 | characters. There was an 'e', but no 'é' or 'Í'. This meant that languages |
| 30 | which required accented characters couldn't be faithfully represented in ASCII. |
| 31 | (Actually the missing accents matter for English, too, which contains words such |
| 32 | as 'naïve' and 'café', and some publications have house styles which require |
| 33 | spellings such as 'coöperate'.) |
| 34 | |
| 35 | For a while people just wrote programs that didn't display accents. I remember |
| 36 | looking at Apple ][ BASIC programs, published in French-language publications in |
| 37 | the mid-1980s, that had lines like these:: |
| 38 | |
| 39 | PRINT "FICHER EST COMPLETE." |
| 40 | PRINT "CARACTERE NON ACCEPTE." |
| 41 | |
| 42 | Those messages should contain accents, and they just look wrong to someone who |
| 43 | can read French. |
| 44 | |
| 45 | In the 1980s, almost all personal computers were 8-bit, meaning that bytes could |
| 46 | hold values ranging from 0 to 255. ASCII codes only went up to 127, so some |
| 47 | machines assigned values between 128 and 255 to accented characters. Different |
| 48 | machines had different codes, however, which led to problems exchanging files. |
| 49 | Eventually various commonly used sets of values for the 128-255 range emerged. |
| 50 | Some were true standards, defined by the International Standards Organization, |
| 51 | and some were **de facto** conventions that were invented by one company or |
| 52 | another and managed to catch on. |
| 53 | |
| 54 | 255 characters aren't very many. For example, you can't fit both the accented |
| 55 | characters used in Western Europe and the Cyrillic alphabet used for Russian |
| 56 | into the 128-255 range because there are more than 127 such characters. |
| 57 | |
| 58 | You could write files using different codes (all your Russian files in a coding |
| 59 | system called KOI8, all your French files in a different coding system called |
| 60 | Latin1), but what if you wanted to write a French document that quotes some |
| 61 | Russian text? In the 1980s people began to want to solve this problem, and the |
| 62 | Unicode standardization effort began. |
| 63 | |
| 64 | Unicode started out using 16-bit characters instead of 8-bit characters. 16 |
| 65 | bits means you have 2^16 = 65,536 distinct values available, making it possible |
| 66 | to represent many different characters from many different alphabets; an initial |
| 67 | goal was to have Unicode contain the alphabets for every single human language. |
| 68 | It turns out that even 16 bits isn't enough to meet that goal, and the modern |
| 69 | Unicode specification uses a wider range of codes, 0-1,114,111 (0x10ffff in |
| 70 | base-16). |
| 71 | |
| 72 | There's a related ISO standard, ISO 10646. Unicode and ISO 10646 were |
| 73 | originally separate efforts, but the specifications were merged with the 1.1 |
| 74 | revision of Unicode. |
| 75 | |
| 76 | (This discussion of Unicode's history is highly simplified. I don't think the |
| 77 | average Python programmer needs to worry about the historical details; consult |
| 78 | the Unicode consortium site listed in the References for more information.) |
| 79 | |
| 80 | |
| 81 | Definitions |
| 82 | ----------- |
| 83 | |
| 84 | A **character** is the smallest possible component of a text. 'A', 'B', 'C', |
| 85 | etc., are all different characters. So are 'È' and 'Í'. Characters are |
| 86 | abstractions, and vary depending on the language or context you're talking |
| 87 | about. For example, the symbol for ohms (Ω) is usually drawn much like the |
| 88 | capital letter omega (Ω) in the Greek alphabet (they may even be the same in |
| 89 | some fonts), but these are two different characters that have different |
| 90 | meanings. |
| 91 | |
| 92 | The Unicode standard describes how characters are represented by **code |
| 93 | points**. A code point is an integer value, usually denoted in base 16. In the |
| 94 | standard, a code point is written using the notation U+12ca to mean the |
| 95 | character with value 0x12ca (4810 decimal). The Unicode standard contains a lot |
| 96 | of tables listing characters and their corresponding code points:: |
| 97 | |
| 98 | 0061 'a'; LATIN SMALL LETTER A |
| 99 | 0062 'b'; LATIN SMALL LETTER B |
| 100 | 0063 'c'; LATIN SMALL LETTER C |
| 101 | ... |
| 102 | 007B '{'; LEFT CURLY BRACKET |
| 103 | |
| 104 | Strictly, these definitions imply that it's meaningless to say 'this is |
| 105 | character U+12ca'. U+12ca is a code point, which represents some particular |
| 106 | character; in this case, it represents the character 'ETHIOPIC SYLLABLE WI'. In |
| 107 | informal contexts, this distinction between code points and characters will |
| 108 | sometimes be forgotten. |
| 109 | |
| 110 | A character is represented on a screen or on paper by a set of graphical |
| 111 | elements that's called a **glyph**. The glyph for an uppercase A, for example, |
| 112 | is two diagonal strokes and a horizontal stroke, though the exact details will |
| 113 | depend on the font being used. Most Python code doesn't need to worry about |
| 114 | glyphs; figuring out the correct glyph to display is generally the job of a GUI |
| 115 | toolkit or a terminal's font renderer. |
| 116 | |
| 117 | |
| 118 | Encodings |
| 119 | --------- |
| 120 | |
| 121 | To summarize the previous section: a Unicode string is a sequence of code |
| 122 | points, which are numbers from 0 to 0x10ffff. This sequence needs to be |
| 123 | represented as a set of bytes (meaning, values from 0-255) in memory. The rules |
| 124 | for translating a Unicode string into a sequence of bytes are called an |
| 125 | **encoding**. |
| 126 | |
| 127 | The first encoding you might think of is an array of 32-bit integers. In this |
| 128 | representation, the string "Python" would look like this:: |
| 129 | |
| 130 | P y t h o n |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 131 | 0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00 |
| 132 | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 133 | |
| 134 | This representation is straightforward but using it presents a number of |
| 135 | problems. |
| 136 | |
| 137 | 1. It's not portable; different processors order the bytes differently. |
| 138 | |
| 139 | 2. It's very wasteful of space. In most texts, the majority of the code points |
| 140 | are less than 127, or less than 255, so a lot of space is occupied by zero |
| 141 | bytes. The above string takes 24 bytes compared to the 6 bytes needed for an |
| 142 | ASCII representation. Increased RAM usage doesn't matter too much (desktop |
| 143 | computers have megabytes of RAM, and strings aren't usually that large), but |
| 144 | expanding our usage of disk and network bandwidth by a factor of 4 is |
| 145 | intolerable. |
| 146 | |
| 147 | 3. It's not compatible with existing C functions such as ``strlen()``, so a new |
| 148 | family of wide string functions would need to be used. |
| 149 | |
| 150 | 4. Many Internet standards are defined in terms of textual data, and can't |
| 151 | handle content with embedded zero bytes. |
| 152 | |
| 153 | Generally people don't use this encoding, instead choosing other encodings that |
| 154 | are more efficient and convenient. |
| 155 | |
| 156 | Encodings don't have to handle every possible Unicode character, and most |
| 157 | encodings don't. For example, Python's default encoding is the 'ascii' |
| 158 | encoding. The rules for converting a Unicode string into the ASCII encoding are |
| 159 | simple; for each code point: |
| 160 | |
| 161 | 1. If the code point is < 128, each byte is the same as the value of the code |
| 162 | point. |
| 163 | |
| 164 | 2. If the code point is 128 or greater, the Unicode string can't be represented |
| 165 | in this encoding. (Python raises a :exc:`UnicodeEncodeError` exception in this |
| 166 | case.) |
| 167 | |
| 168 | Latin-1, also known as ISO-8859-1, is a similar encoding. Unicode code points |
| 169 | 0-255 are identical to the Latin-1 values, so converting to this encoding simply |
| 170 | requires converting code points to byte values; if a code point larger than 255 |
| 171 | is encountered, the string can't be encoded into Latin-1. |
| 172 | |
| 173 | Encodings don't have to be simple one-to-one mappings like Latin-1. Consider |
| 174 | IBM's EBCDIC, which was used on IBM mainframes. Letter values weren't in one |
| 175 | block: 'a' through 'i' had values from 129 to 137, but 'j' through 'r' were 145 |
| 176 | through 153. If you wanted to use EBCDIC as an encoding, you'd probably use |
| 177 | some sort of lookup table to perform the conversion, but this is largely an |
| 178 | internal detail. |
| 179 | |
| 180 | UTF-8 is one of the most commonly used encodings. UTF stands for "Unicode |
| 181 | Transformation Format", and the '8' means that 8-bit numbers are used in the |
| 182 | encoding. (There's also a UTF-16 encoding, but it's less frequently used than |
| 183 | UTF-8.) UTF-8 uses the following rules: |
| 184 | |
| 185 | 1. If the code point is <128, it's represented by the corresponding byte value. |
| 186 | 2. If the code point is between 128 and 0x7ff, it's turned into two byte values |
| 187 | between 128 and 255. |
| 188 | 3. Code points >0x7ff are turned into three- or four-byte sequences, where each |
| 189 | byte of the sequence is between 128 and 255. |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 190 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 191 | UTF-8 has several convenient properties: |
| 192 | |
| 193 | 1. It can handle any Unicode code point. |
| 194 | 2. A Unicode string is turned into a string of bytes containing no embedded zero |
| 195 | bytes. This avoids byte-ordering issues, and means UTF-8 strings can be |
| 196 | processed by C functions such as ``strcpy()`` and sent through protocols that |
| 197 | can't handle zero bytes. |
| 198 | 3. A string of ASCII text is also valid UTF-8 text. |
| 199 | 4. UTF-8 is fairly compact; the majority of code points are turned into two |
| 200 | bytes, and values less than 128 occupy only a single byte. |
| 201 | 5. If bytes are corrupted or lost, it's possible to determine the start of the |
| 202 | next UTF-8-encoded code point and resynchronize. It's also unlikely that |
| 203 | random 8-bit data will look like valid UTF-8. |
| 204 | |
| 205 | |
| 206 | |
| 207 | References |
| 208 | ---------- |
| 209 | |
| 210 | The Unicode Consortium site at <http://www.unicode.org> has character charts, a |
| 211 | glossary, and PDF versions of the Unicode specification. Be prepared for some |
| 212 | difficult reading. <http://www.unicode.org/history/> is a chronology of the |
| 213 | origin and development of Unicode. |
| 214 | |
| 215 | To help understand the standard, Jukka Korpela has written an introductory guide |
| 216 | to reading the Unicode character tables, available at |
| 217 | <http://www.cs.tut.fi/~jkorpela/unicode/guide.html>. |
| 218 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 219 | Two other good introductory articles were written by Joel Spolsky |
| 220 | <http://www.joelonsoftware.com/articles/Unicode.html> and Jason Orendorff |
| 221 | <http://www.jorendorff.com/articles/unicode/>. If this introduction didn't make |
| 222 | things clear to you, you should try reading one of these alternate articles |
| 223 | before continuing. |
| 224 | |
| 225 | Wikipedia entries are often helpful; see the entries for "character encoding" |
| 226 | <http://en.wikipedia.org/wiki/Character_encoding> and UTF-8 |
| 227 | <http://en.wikipedia.org/wiki/UTF-8>, for example. |
| 228 | |
| 229 | |
| 230 | Python's Unicode Support |
| 231 | ======================== |
| 232 | |
| 233 | Now that you've learned the rudiments of Unicode, we can look at Python's |
| 234 | Unicode features. |
| 235 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 236 | The String Type |
| 237 | --------------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 238 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 239 | Since Python 3.0, the language features a ``str`` type that contain Unicode |
| 240 | characters, meaning any string created using ``"unicode rocks!"``, ``'unicode |
| 241 | rocks!``, or the triple-quoted string syntax is stored as Unicode. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 242 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 243 | To insert a Unicode character that is not part ASCII, e.g., any letters with |
| 244 | accents, one can use escape sequences in their string literals as such:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 245 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 246 | >>> "\N{GREEK CAPITAL LETTER DELTA}" # Using the character name |
| 247 | '\u0394' |
| 248 | >>> "\u0394" # Using a 16-bit hex value |
| 249 | '\u0394' |
| 250 | >>> "\U00000394" # Using a 32-bit hex value |
| 251 | '\u0394' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 252 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 253 | In addition, one can create a string using the :func:`decode` method of |
| 254 | :class:`bytes`. This method takes an encoding, such as UTF-8, and, optionally, |
| 255 | an *errors* argument. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 256 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 257 | The *errors* argument specifies the response when the input string can't be |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 258 | converted according to the encoding's rules. Legal values for this argument are |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 259 | 'strict' (raise a :exc:`UnicodeDecodeError` exception), 'replace' (add U+FFFD, |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 260 | 'REPLACEMENT CHARACTER'), or 'ignore' (just leave the character out of the |
| 261 | Unicode result). The following examples show the differences:: |
| 262 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 263 | >>> b'\x80abc'.decode("utf-8", "strict") |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 264 | Traceback (most recent call last): |
| 265 | File "<stdin>", line 1, in ? |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 266 | UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 267 | ordinal not in range(128) |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 268 | >>> b'\x80abc'.decode("utf-8", "replace") |
| 269 | '\ufffdabc' |
| 270 | >>> b'\x80abc'.decode("utf-8", "ignore") |
| 271 | 'abc' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 272 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 273 | Encodings are specified as strings containing the encoding's name. Python |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 274 | comes with roughly 100 different encodings; see the Python Library Reference at |
Christian Heimes | 2202f87 | 2008-02-06 14:31:34 +0000 | [diff] [blame] | 275 | :ref:`standard-encodings` for a list. Some encodings |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 276 | have multiple names; for example, 'latin-1', 'iso_8859_1' and '8859' are all |
| 277 | synonyms for the same encoding. |
| 278 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 279 | One-character Unicode strings can also be created with the :func:`chr` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 280 | built-in function, which takes integers and returns a Unicode string of length 1 |
| 281 | that contains the corresponding code point. The reverse operation is the |
| 282 | built-in :func:`ord` function that takes a one-character Unicode string and |
| 283 | returns the code point value:: |
| 284 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 285 | >>> chr(40960) |
| 286 | '\ua000' |
| 287 | >>> ord('\ua000') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 288 | 40960 |
| 289 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 290 | Converting to Bytes |
| 291 | ------------------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 292 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 293 | Another important str method is ``.encode([encoding], [errors='strict'])``, |
| 294 | which returns a ``bytes`` representation of the Unicode string, encoded in the |
| 295 | requested encoding. The ``errors`` parameter is the same as the parameter of |
| 296 | the :meth:`decode` method, with one additional possibility; as well as 'strict', |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 297 | 'ignore', and 'replace', you can also pass 'xmlcharrefreplace' which uses XML's |
| 298 | character references. The following example shows the different results:: |
| 299 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 300 | >>> u = chr(40960) + 'abcd' + chr(1972) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 301 | >>> u.encode('utf-8') |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 302 | b'\xea\x80\x80abcd\xde\xb4' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 303 | >>> u.encode('ascii') |
| 304 | Traceback (most recent call last): |
| 305 | File "<stdin>", line 1, in ? |
| 306 | UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in position 0: ordinal not in range(128) |
| 307 | >>> u.encode('ascii', 'ignore') |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 308 | b'abcd' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 309 | >>> u.encode('ascii', 'replace') |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 310 | b'?abcd?' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 311 | >>> u.encode('ascii', 'xmlcharrefreplace') |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 312 | b'ꀀabcd޴' |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 313 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 314 | The low-level routines for registering and accessing the available encodings are |
| 315 | found in the :mod:`codecs` module. However, the encoding and decoding functions |
| 316 | returned by this module are usually more low-level than is comfortable, so I'm |
| 317 | not going to describe the :mod:`codecs` module here. If you need to implement a |
| 318 | completely new encoding, you'll need to learn about the :mod:`codecs` module |
| 319 | interfaces, but implementing encodings is a specialized task that also won't be |
| 320 | covered here. Consult the Python documentation to learn more about this module. |
| 321 | |
| 322 | The most commonly used part of the :mod:`codecs` module is the |
| 323 | :func:`codecs.open` function which will be discussed in the section on input and |
| 324 | output. |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 325 | |
| 326 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 327 | Unicode Literals in Python Source Code |
| 328 | -------------------------------------- |
| 329 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 330 | In Python source code, specific Unicode code points can be written using the |
| 331 | ``\u`` escape sequence, which is followed by four hex digits giving the code |
| 332 | point. The ``\U`` escape sequence is similar, but expects 8 hex digits, not 4:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 333 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 334 | >>> s = "a\xac\u1234\u20ac\U00008000" |
| 335 | ^^^^ two-digit hex escape |
| 336 | ^^^^^ four-digit Unicode escape |
| 337 | ^^^^^^^^^^ eight-digit Unicode escape |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 338 | >>> for c in s: print(ord(c), end=" ") |
| 339 | ... |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 340 | 97 172 4660 8364 32768 |
| 341 | |
| 342 | Using escape sequences for code points greater than 127 is fine in small doses, |
| 343 | but becomes an annoyance if you're using many accented characters, as you would |
| 344 | in a program with messages in French or some other accent-using language. You |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 345 | can also assemble strings using the :func:`chr` built-in function, but this is |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 346 | even more tedious. |
| 347 | |
| 348 | Ideally, you'd want to be able to write literals in your language's natural |
| 349 | encoding. You could then edit Python source code with your favorite editor |
| 350 | which would display the accented characters naturally, and have the right |
| 351 | characters used at runtime. |
| 352 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 353 | Python supports writing Unicode literals in UTF-8 by default, but you can use |
| 354 | (almost) any encoding if you declare the encoding being used. This is done by |
| 355 | including a special comment as either the first or second line of the source |
| 356 | file:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 357 | |
| 358 | #!/usr/bin/env python |
| 359 | # -*- coding: latin-1 -*- |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 360 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 361 | u = 'abcdé' |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 362 | print(ord(u[-1])) |
| 363 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 364 | The syntax is inspired by Emacs's notation for specifying variables local to a |
| 365 | file. Emacs supports many different variables, but Python only supports |
| 366 | 'coding'. The ``-*-`` symbols indicate that the comment is special; within |
| 367 | them, you must supply the name ``coding`` and the name of your chosen encoding, |
| 368 | separated by ``':'``. |
| 369 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 370 | If you don't include such a comment, the default encoding used will be UTF-8 as |
| 371 | already mentioned. |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 372 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 373 | |
| 374 | Unicode Properties |
| 375 | ------------------ |
| 376 | |
| 377 | The Unicode specification includes a database of information about code points. |
| 378 | For each code point that's defined, the information includes the character's |
| 379 | name, its category, the numeric value if applicable (Unicode has characters |
| 380 | representing the Roman numerals and fractions such as one-third and |
| 381 | four-fifths). There are also properties related to the code point's use in |
| 382 | bidirectional text and other display-related properties. |
| 383 | |
| 384 | The following program displays some information about several characters, and |
| 385 | prints the numeric value of one particular character:: |
| 386 | |
| 387 | import unicodedata |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 388 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 389 | u = chr(233) + chr(0x0bf2) + chr(3972) + chr(6000) + chr(13231) |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 390 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 391 | for i, c in enumerate(u): |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 392 | print(i, '%04x' % ord(c), unicodedata.category(c), end=" ") |
| 393 | print(unicodedata.name(c)) |
| 394 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 395 | # Get numeric value of second character |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 396 | print(unicodedata.numeric(u[1])) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 397 | |
| 398 | When run, this prints:: |
| 399 | |
| 400 | 0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE |
| 401 | 1 0bf2 No TAMIL NUMBER ONE THOUSAND |
| 402 | 2 0f84 Mn TIBETAN MARK HALANTA |
| 403 | 3 1770 Lo TAGBANWA LETTER SA |
| 404 | 4 33af So SQUARE RAD OVER S SQUARED |
| 405 | 1000.0 |
| 406 | |
| 407 | The category codes are abbreviations describing the nature of the character. |
| 408 | These are grouped into categories such as "Letter", "Number", "Punctuation", or |
| 409 | "Symbol", which in turn are broken up into subcategories. To take the codes |
| 410 | from the above output, ``'Ll'`` means 'Letter, lowercase', ``'No'`` means |
| 411 | "Number, other", ``'Mn'`` is "Mark, nonspacing", and ``'So'`` is "Symbol, |
| 412 | other". See |
| 413 | <http://www.unicode.org/Public/UNIDATA/UCD.html#General_Category_Values> for a |
| 414 | list of category codes. |
| 415 | |
| 416 | References |
| 417 | ---------- |
| 418 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 419 | The ``str`` type is described in the Python library reference at |
| 420 | :ref:`typesseq`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 421 | |
| 422 | The documentation for the :mod:`unicodedata` module. |
| 423 | |
| 424 | The documentation for the :mod:`codecs` module. |
| 425 | |
| 426 | Marc-André Lemburg gave a presentation at EuroPython 2002 titled "Python and |
| 427 | Unicode". A PDF version of his slides is available at |
Christian Heimes | dd15f6c | 2008-03-16 00:07:10 +0000 | [diff] [blame] | 428 | <http://downloads.egenix.com/python/Unicode-EPC2002-Talk.pdf>, and is an |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 429 | excellent overview of the design of Python's Unicode features. |
| 430 | |
| 431 | |
| 432 | Reading and Writing Unicode Data |
| 433 | ================================ |
| 434 | |
| 435 | Once you've written some code that works with Unicode data, the next problem is |
| 436 | input/output. How do you get Unicode strings into your program, and how do you |
| 437 | convert Unicode into a form suitable for storage or transmission? |
| 438 | |
| 439 | It's possible that you may not need to do anything depending on your input |
| 440 | sources and output destinations; you should check whether the libraries used in |
| 441 | your application support Unicode natively. XML parsers often return Unicode |
| 442 | data, for example. Many relational databases also support Unicode-valued |
| 443 | columns and can return Unicode values from an SQL query. |
| 444 | |
| 445 | Unicode data is usually converted to a particular encoding before it gets |
| 446 | written to disk or sent over a socket. It's possible to do all the work |
| 447 | yourself: open a file, read an 8-bit string from it, and convert the string with |
| 448 | ``unicode(str, encoding)``. However, the manual approach is not recommended. |
| 449 | |
| 450 | One problem is the multi-byte nature of encodings; one Unicode character can be |
| 451 | represented by several bytes. If you want to read the file in arbitrary-sized |
| 452 | chunks (say, 1K or 4K), you need to write error-handling code to catch the case |
| 453 | where only part of the bytes encoding a single Unicode character are read at the |
| 454 | end of a chunk. One solution would be to read the entire file into memory and |
| 455 | then perform the decoding, but that prevents you from working with files that |
| 456 | are extremely large; if you need to read a 2Gb file, you need 2Gb of RAM. |
| 457 | (More, really, since for at least a moment you'd need to have both the encoded |
| 458 | string and its Unicode version in memory.) |
| 459 | |
| 460 | The solution would be to use the low-level decoding interface to catch the case |
| 461 | of partial coding sequences. The work of implementing this has already been |
| 462 | done for you: the :mod:`codecs` module includes a version of the :func:`open` |
| 463 | function that returns a file-like object that assumes the file's contents are in |
| 464 | a specified encoding and accepts Unicode parameters for methods such as |
| 465 | ``.read()`` and ``.write()``. |
| 466 | |
| 467 | The function's parameters are ``open(filename, mode='rb', encoding=None, |
| 468 | errors='strict', buffering=1)``. ``mode`` can be ``'r'``, ``'w'``, or ``'a'``, |
| 469 | just like the corresponding parameter to the regular built-in ``open()`` |
| 470 | function; add a ``'+'`` to update the file. ``buffering`` is similarly parallel |
| 471 | to the standard function's parameter. ``encoding`` is a string giving the |
| 472 | encoding to use; if it's left as ``None``, a regular Python file object that |
| 473 | accepts 8-bit strings is returned. Otherwise, a wrapper object is returned, and |
| 474 | data written to or read from the wrapper object will be converted as needed. |
| 475 | ``errors`` specifies the action for encoding errors and can be one of the usual |
| 476 | values of 'strict', 'ignore', and 'replace'. |
| 477 | |
| 478 | Reading Unicode from a file is therefore simple:: |
| 479 | |
| 480 | import codecs |
| 481 | f = codecs.open('unicode.rst', encoding='utf-8') |
| 482 | for line in f: |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 483 | print(repr(line)) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 484 | |
| 485 | It's also possible to open files in update mode, allowing both reading and |
| 486 | writing:: |
| 487 | |
| 488 | f = codecs.open('test', encoding='utf-8', mode='w+') |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 489 | f.write('\u4500 blah blah blah\n') |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 490 | f.seek(0) |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 491 | print(repr(f.readline()[:1])) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 492 | f.close() |
| 493 | |
| 494 | Unicode character U+FEFF is used as a byte-order mark (BOM), and is often |
| 495 | written as the first character of a file in order to assist with autodetection |
| 496 | of the file's byte ordering. Some encodings, such as UTF-16, expect a BOM to be |
| 497 | present at the start of a file; when such an encoding is used, the BOM will be |
| 498 | automatically written as the first character and will be silently dropped when |
| 499 | the file is read. There are variants of these encodings, such as 'utf-16-le' |
| 500 | and 'utf-16-be' for little-endian and big-endian encodings, that specify one |
| 501 | particular byte ordering and don't skip the BOM. |
| 502 | |
| 503 | |
| 504 | Unicode filenames |
| 505 | ----------------- |
| 506 | |
| 507 | Most of the operating systems in common use today support filenames that contain |
| 508 | arbitrary Unicode characters. Usually this is implemented by converting the |
| 509 | Unicode string into some encoding that varies depending on the system. For |
| 510 | example, MacOS X uses UTF-8 while Windows uses a configurable encoding; on |
| 511 | Windows, Python uses the name "mbcs" to refer to whatever the currently |
| 512 | configured encoding is. On Unix systems, there will only be a filesystem |
| 513 | encoding if you've set the ``LANG`` or ``LC_CTYPE`` environment variables; if |
| 514 | you haven't, the default encoding is ASCII. |
| 515 | |
| 516 | The :func:`sys.getfilesystemencoding` function returns the encoding to use on |
| 517 | your current system, in case you want to do the encoding manually, but there's |
| 518 | not much reason to bother. When opening a file for reading or writing, you can |
| 519 | usually just provide the Unicode string as the filename, and it will be |
| 520 | automatically converted to the right encoding for you:: |
| 521 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 522 | filename = 'filename\u4500abc' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 523 | f = open(filename, 'w') |
| 524 | f.write('blah\n') |
| 525 | f.close() |
| 526 | |
| 527 | Functions in the :mod:`os` module such as :func:`os.stat` will also accept Unicode |
| 528 | filenames. |
| 529 | |
| 530 | :func:`os.listdir`, which returns filenames, raises an issue: should it return |
| 531 | the Unicode version of filenames, or should it return 8-bit strings containing |
| 532 | the encoded versions? :func:`os.listdir` will do both, depending on whether you |
| 533 | provided the directory path as an 8-bit string or a Unicode string. If you pass |
| 534 | a Unicode string as the path, filenames will be decoded using the filesystem's |
| 535 | encoding and a list of Unicode strings will be returned, while passing an 8-bit |
| 536 | path will return the 8-bit versions of the filenames. For example, assuming the |
| 537 | default filesystem encoding is UTF-8, running the following program:: |
| 538 | |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 539 | fn = 'filename\u4500abc' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 540 | f = open(fn, 'w') |
| 541 | f.close() |
| 542 | |
| 543 | import os |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 544 | print(os.listdir('.')) |
| 545 | print(os.listdir(u'.')) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 546 | |
| 547 | will produce the following output:: |
| 548 | |
| 549 | amk:~$ python t.py |
| 550 | ['.svn', 'filename\xe4\x94\x80abc', ...] |
Georg Brandl | f694518 | 2008-02-01 11:56:49 +0000 | [diff] [blame] | 551 | ['.svn', 'filename\u4500abc', ...] |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 552 | |
| 553 | The first list contains UTF-8-encoded filenames, and the second list contains |
| 554 | the Unicode versions. |
| 555 | |
| 556 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 557 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 558 | Tips for Writing Unicode-aware Programs |
| 559 | --------------------------------------- |
| 560 | |
| 561 | This section provides some suggestions on writing software that deals with |
| 562 | Unicode. |
| 563 | |
| 564 | The most important tip is: |
| 565 | |
| 566 | Software should only work with Unicode strings internally, converting to a |
| 567 | particular encoding on output. |
| 568 | |
| 569 | If you attempt to write processing functions that accept both Unicode and 8-bit |
| 570 | strings, you will find your program vulnerable to bugs wherever you combine the |
| 571 | two different kinds of strings. Python's default encoding is ASCII, so whenever |
| 572 | a character with an ASCII value > 127 is in the input data, you'll get a |
| 573 | :exc:`UnicodeDecodeError` because that character can't be handled by the ASCII |
| 574 | encoding. |
| 575 | |
| 576 | It's easy to miss such problems if you only test your software with data that |
| 577 | doesn't contain any accents; everything will seem to work, but there's actually |
| 578 | a bug in your program waiting for the first user who attempts to use characters |
| 579 | > 127. A second tip, therefore, is: |
| 580 | |
| 581 | Include characters > 127 and, even better, characters > 255 in your test |
| 582 | data. |
| 583 | |
| 584 | When using data coming from a web browser or some other untrusted source, a |
| 585 | common technique is to check for illegal characters in a string before using the |
| 586 | string in a generated command line or storing it in a database. If you're doing |
| 587 | this, be careful to check the string once it's in the form that will be used or |
| 588 | stored; it's possible for encodings to be used to disguise characters. This is |
| 589 | especially true if the input data also specifies the encoding; many encodings |
| 590 | leave the commonly checked-for characters alone, but Python includes some |
| 591 | encodings such as ``'base64'`` that modify every single character. |
| 592 | |
| 593 | For example, let's say you have a content management system that takes a Unicode |
| 594 | filename, and you want to disallow paths with a '/' character. You might write |
| 595 | this code:: |
| 596 | |
| 597 | def read_file (filename, encoding): |
| 598 | if '/' in filename: |
| 599 | raise ValueError("'/' not allowed in filenames") |
| 600 | unicode_name = filename.decode(encoding) |
| 601 | f = open(unicode_name, 'r') |
| 602 | # ... return contents of file ... |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 603 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 604 | However, if an attacker could specify the ``'base64'`` encoding, they could pass |
| 605 | ``'L2V0Yy9wYXNzd2Q='``, which is the base-64 encoded form of the string |
| 606 | ``'/etc/passwd'``, to read a system file. The above code looks for ``'/'`` |
| 607 | characters in the encoded form and misses the dangerous character in the |
| 608 | resulting decoded form. |
| 609 | |
| 610 | References |
| 611 | ---------- |
| 612 | |
| 613 | The PDF slides for Marc-André Lemburg's presentation "Writing Unicode-aware |
| 614 | Applications in Python" are available at |
Christian Heimes | dd15f6c | 2008-03-16 00:07:10 +0000 | [diff] [blame] | 615 | <http://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 616 | and discuss questions of character encodings as well as how to internationalize |
| 617 | and localize an application. |
| 618 | |
| 619 | |
| 620 | Revision History and Acknowledgements |
| 621 | ===================================== |
| 622 | |
| 623 | Thanks to the following people who have noted errors or offered suggestions on |
| 624 | this article: Nicholas Bastin, Marius Gedminas, Kent Johnson, Ken Krugler, |
| 625 | Marc-André Lemburg, Martin von Löwis, Chad Whitacre. |
| 626 | |
| 627 | Version 1.0: posted August 5 2005. |
| 628 | |
| 629 | Version 1.01: posted August 7 2005. Corrects factual and markup errors; adds |
| 630 | several links. |
| 631 | |
| 632 | Version 1.02: posted August 16 2005. Corrects factual errors. |
| 633 | |
| 634 | |
| 635 | .. comment Additional topic: building Python w/ UCS2 or UCS4 support |
| 636 | .. comment Describe obscure -U switch somewhere? |
| 637 | .. comment Describe use of codecs.StreamRecoder and StreamReaderWriter |
| 638 | |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 639 | .. comment |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 640 | Original outline: |
| 641 | |
| 642 | - [ ] Unicode introduction |
| 643 | - [ ] ASCII |
| 644 | - [ ] Terms |
| 645 | - [ ] Character |
| 646 | - [ ] Code point |
| 647 | - [ ] Encodings |
| 648 | - [ ] Common encodings: ASCII, Latin-1, UTF-8 |
| 649 | - [ ] Unicode Python type |
| 650 | - [ ] Writing unicode literals |
| 651 | - [ ] Obscurity: -U switch |
| 652 | - [ ] Built-ins |
| 653 | - [ ] unichr() |
| 654 | - [ ] ord() |
| 655 | - [ ] unicode() constructor |
| 656 | - [ ] Unicode type |
| 657 | - [ ] encode(), decode() methods |
| 658 | - [ ] Unicodedata module for character properties |
| 659 | - [ ] I/O |
| 660 | - [ ] Reading/writing Unicode data into files |
| 661 | - [ ] Byte-order marks |
| 662 | - [ ] Unicode filenames |
| 663 | - [ ] Writing Unicode programs |
| 664 | - [ ] Do everything in Unicode |
| 665 | - [ ] Declaring source code encodings (PEP 263) |
| 666 | - [ ] Other issues |
| 667 | - [ ] Building Python (UCS2, UCS4) |