blob: 65053c7f1bb31a17a80a8a5ca49db4f1f4b717fe [file] [log] [blame]
Skip Montanarob4a04172003-03-20 23:29:12 +00001\section{\module{csv} --- CSV File Reading and Writing}
2
3\declaremodule{standard}{csv}
4\modulesynopsis{Write and read tabular data to and from delimited files.}
Skip Montanaro3bd3c842003-04-24 18:47:31 +00005\sectionauthor{Skip Montanaro}{skip@pobox.com}
Skip Montanarob4a04172003-03-20 23:29:12 +00006
7\versionadded{2.3}
8\index{csv}
9\indexii{data}{tabular}
10
11The so-called CSV (Comma Separated Values) format is the most common import
12and export format for spreadsheets and databases. There is no ``CSV
13standard'', so the format is operationally defined by the many applications
14which read and write it. The lack of a standard means that subtle
15differences often exist in the data produced and consumed by different
16applications. These differences can make it annoying to process CSV files
17from multiple sources. Still, while the delimiters and quoting characters
18vary, the overall format is similar enough that it is possible to write a
19single module which can efficiently manipulate such data, hiding the details
20of reading and writing the data from the programmer.
21
Skip Montanaro5d0136e2003-04-25 15:14:49 +000022The \module{csv} module implements classes to read and write tabular data in
Skip Montanarob4a04172003-03-20 23:29:12 +000023CSV format. It allows programmers to say, ``write this data in the format
24preferred by Excel,'' or ``read data from this file which was generated by
25Excel,'' without knowing the precise details of the CSV format used by
26Excel. Programmers can also describe the CSV formats understood by other
27applications or define their own special-purpose CSV formats.
28
Skip Montanaro5d0136e2003-04-25 15:14:49 +000029The \module{csv} module's \class{reader} and \class{writer} objects read and
Skip Montanarob4a04172003-03-20 23:29:12 +000030write sequences. Programmers can also read and write data in dictionary
31form using the \class{DictReader} and \class{DictWriter} classes.
32
Fred Drake96352682003-04-25 18:02:34 +000033\begin{notice}
34 This version of the \module{csv} module doesn't support Unicode
35 input. Also, there are currently some issues regarding \ASCII{} NUL
David Goodgercb30f972006-04-04 03:05:44 +000036 characters. Accordingly, all input should be UTF-8 or printable
37 \ASCII{} to be safe; see the examples in section~\ref{csv-examples}.
38 These restrictions will be removed in the future.
Fred Drake96352682003-04-25 18:02:34 +000039\end{notice}
Skip Montanarob4a04172003-03-20 23:29:12 +000040
41\begin{seealso}
42% \seemodule{array}{Arrays of uniformly types numeric values.}
43 \seepep{305}{CSV File API}
44 {The Python Enhancement Proposal which proposed this addition
45 to Python.}
46\end{seealso}
47
48
Raymond Hettinger6f6d7b932003-08-31 05:44:54 +000049\subsection{Module Contents \label{csv-contents}}
Skip Montanarob4a04172003-03-20 23:29:12 +000050
Skip Montanaro5d0136e2003-04-25 15:14:49 +000051The \module{csv} module defines the following functions:
Skip Montanarob4a04172003-03-20 23:29:12 +000052
53\begin{funcdesc}{reader}{csvfile\optional{,
Andrew McNamara8231de02005-01-12 11:47:57 +000054 dialect=\code{'excel'}}\optional{, fmtparam}}
Skip Montanarob4a04172003-03-20 23:29:12 +000055Return a reader object which will iterate over lines in the given
56{}\var{csvfile}. \var{csvfile} can be any object which supports the
57iterator protocol and returns a string each time its \method{next}
Andrew McNamara8231de02005-01-12 11:47:57 +000058method is called - file objects and list objects are both suitable.
59If \var{csvfile} is a file object, it must be opened with
Skip Montanaro5e4e39f2003-07-02 15:32:48 +000060the 'b' flag on platforms where that makes a difference. An optional
61{}\var{dialect} parameter can be given
Skip Montanarob4a04172003-03-20 23:29:12 +000062which is used to define a set of parameters specific to a particular CSV
63dialect. It may be an instance of a subclass of the \class{Dialect}
64class or one of the strings returned by the \function{list_dialects}
65function. The other optional {}\var{fmtparam} keyword arguments can be
66given to override individual formatting parameters in the current
67dialect. For more information about the dialect and formatting
Raymond Hettinger6e380cd2003-09-10 18:54:49 +000068parameters, see section~\ref{csv-fmt-params}, ``Dialects and Formatting
Skip Montanarob4a04172003-03-20 23:29:12 +000069Parameters'' for details of these parameters.
70
71All data read are returned as strings. No automatic data type
72conversion is performed.
73\end{funcdesc}
74
75\begin{funcdesc}{writer}{csvfile\optional{,
Andrew McNamara8231de02005-01-12 11:47:57 +000076 dialect=\code{'excel'}}\optional{, fmtparam}}
Skip Montanarob4a04172003-03-20 23:29:12 +000077Return a writer object responsible for converting the user's data into
Skip Montanaro5e4e39f2003-07-02 15:32:48 +000078delimited strings on the given file-like object. \var{csvfile} can be any
79object with a \function{write} method. If \var{csvfile} is a file object,
80it must be opened with the 'b' flag on platforms where that makes a
81difference. An optional
Skip Montanarob4a04172003-03-20 23:29:12 +000082{}\var{dialect} parameter can be given which is used to define a set of
83parameters specific to a particular CSV dialect. It may be an instance
84of a subclass of the \class{Dialect} class or one of the strings
85returned by the \function{list_dialects} function. The other optional
86{}\var{fmtparam} keyword arguments can be given to override individual
87formatting parameters in the current dialect. For more information
88about the dialect and formatting parameters, see
Raymond Hettinger6e380cd2003-09-10 18:54:49 +000089section~\ref{csv-fmt-params}, ``Dialects and Formatting Parameters'' for
Skip Montanarob4a04172003-03-20 23:29:12 +000090details of these parameters. To make it as easy as possible to
91interface with modules which implement the DB API, the value
92\constant{None} is written as the empty string. While this isn't a
93reversible transformation, it makes it easier to dump SQL NULL data values
94to CSV files without preprocessing the data returned from a
95\code{cursor.fetch*()} call. All other non-string data are stringified
96with \function{str()} before being written.
97\end{funcdesc}
98
Andrew McNamara8231de02005-01-12 11:47:57 +000099\begin{funcdesc}{register_dialect}{name\optional{, dialect}\optional{, fmtparam}}
100Associate \var{dialect} with \var{name}. \var{name} must be a string
101or Unicode object. The dialect can be specified either by passing a
102sub-class of \class{Dialect}, or by \var{fmtparam} keyword arguments,
103or both, with keyword arguments overriding parameters of the dialect.
104For more information about the dialect and formatting parameters, see
105section~\ref{csv-fmt-params}, ``Dialects and Formatting Parameters''
106for details of these parameters.
Skip Montanarob4a04172003-03-20 23:29:12 +0000107\end{funcdesc}
108
109\begin{funcdesc}{unregister_dialect}{name}
110Delete the dialect associated with \var{name} from the dialect registry. An
111\exception{Error} is raised if \var{name} is not a registered dialect
112name.
113\end{funcdesc}
114
115\begin{funcdesc}{get_dialect}{name}
116Return the dialect associated with \var{name}. An \exception{Error} is
117raised if \var{name} is not a registered dialect name.
118\end{funcdesc}
119
120\begin{funcdesc}{list_dialects}{}
121Return the names of all registered dialects.
122\end{funcdesc}
123
Andrew McNamara8231de02005-01-12 11:47:57 +0000124\begin{funcdesc}{field_size_limit}{\optional{new_limit}}
125 Returns the current maximum field size allowed by the parser. If
126 \var{new_limit} is given, this becomes the new limit.
127 \versionadded{2.5}
128\end{funcdesc}
129
Skip Montanarob4a04172003-03-20 23:29:12 +0000130
Skip Montanaro5d0136e2003-04-25 15:14:49 +0000131The \module{csv} module defines the following classes:
Skip Montanarob4a04172003-03-20 23:29:12 +0000132
Skip Montanarodffeed32003-10-03 14:03:01 +0000133\begin{classdesc}{DictReader}{csvfile\optional{,
134 fieldnames=\constant{None},\optional{,
Fred Drake96352682003-04-25 18:02:34 +0000135 restkey=\constant{None}\optional{,
136 restval=\constant{None}\optional{,
Skip Montanarob4a04172003-03-20 23:29:12 +0000137 dialect=\code{'excel'}\optional{,
Skip Montanaro10659f22004-04-16 03:21:01 +0000138 *args, **kwds}}}}}}
Skip Montanarob4a04172003-03-20 23:29:12 +0000139Create an object which operates like a regular reader but maps the
Skip Montanarodffeed32003-10-03 14:03:01 +0000140information read into a dict whose keys are given by the optional
141{} \var{fieldnames}
142parameter. If the \var{fieldnames} parameter is omitted, the values in
143the first row of the \var{csvfile} will be used as the fieldnames.
144If the row read has fewer fields than the fieldnames sequence,
Skip Montanarob4a04172003-03-20 23:29:12 +0000145the value of \var{restval} will be used as the default value. If the row
146read has more fields than the fieldnames sequence, the remaining data is
147added as a sequence keyed by the value of \var{restkey}. If the row read
148has fewer fields than the fieldnames sequence, the remaining keys take the
Skip Montanaro10659f22004-04-16 03:21:01 +0000149value of the optional \var{restval} parameter. Any other optional or
150keyword arguments are passed to the underlying \class{reader} instance.
Skip Montanarob4a04172003-03-20 23:29:12 +0000151\end{classdesc}
152
153
154\begin{classdesc}{DictWriter}{csvfile, fieldnames\optional{,
155 restval=""\optional{,
156 extrasaction=\code{'raise'}\optional{,
Skip Montanaro10659f22004-04-16 03:21:01 +0000157 dialect=\code{'excel'}\optional{,
158 *args, **kwds}}}}}
Skip Montanarob4a04172003-03-20 23:29:12 +0000159Create an object which operates like a regular writer but maps dictionaries
160onto output rows. The \var{fieldnames} parameter identifies the order in
161which values in the dictionary passed to the \method{writerow()} method are
162written to the \var{csvfile}. The optional \var{restval} parameter
163specifies the value to be written if the dictionary is missing a key in
164\var{fieldnames}. If the dictionary passed to the \method{writerow()}
165method contains a key not found in \var{fieldnames}, the optional
166\var{extrasaction} parameter indicates what action to take. If it is set
167to \code{'raise'} a \exception{ValueError} is raised. If it is set to
Skip Montanaro10659f22004-04-16 03:21:01 +0000168\code{'ignore'}, extra values in the dictionary are ignored. Any other
169optional or keyword arguments are passed to the underlying \class{writer}
170instance.
Skip Montanarodffeed32003-10-03 14:03:01 +0000171
172Note that unlike the \class{DictReader} class, the \var{fieldnames}
173parameter of the \class{DictWriter} is not optional. Since Python's
174\class{dict} objects are not ordered, there is not enough information
175available to deduce the order in which the row should be written to the
176\var{csvfile}.
177
Skip Montanarob4a04172003-03-20 23:29:12 +0000178\end{classdesc}
179
Skip Montanarob4a04172003-03-20 23:29:12 +0000180\begin{classdesc*}{Dialect}{}
181The \class{Dialect} class is a container class relied on primarily for its
182attributes, which are used to define the parameters for a specific
Fred Drake96352682003-04-25 18:02:34 +0000183\class{reader} or \class{writer} instance.
Skip Montanarob4a04172003-03-20 23:29:12 +0000184\end{classdesc*}
185
Skip Montanarobb0c9dc2005-01-05 06:58:15 +0000186\begin{classdesc}{excel}{}
187The \class{excel} class defines the usual properties of an Excel-generated
188CSV file.
189\end{classdesc}
190
191\begin{classdesc}{excel_tab}{}
192The \class{excel_tab} class defines the usual properties of an
193Excel-generated TAB-delimited file.
194\end{classdesc}
195
Skip Montanaro77892372003-05-19 15:33:36 +0000196\begin{classdesc}{Sniffer}{}
197The \class{Sniffer} class is used to deduce the format of a CSV file.
Fred Drake96352682003-04-25 18:02:34 +0000198\end{classdesc}
199
Skip Montanaro8bdaac72005-12-28 15:56:58 +0000200The \class{Sniffer} class provides two methods:
Fred Drake96352682003-04-25 18:02:34 +0000201
Skip Montanaro77892372003-05-19 15:33:36 +0000202\begin{methoddesc}{sniff}{sample\optional{,delimiters=None}}
203Analyze the given \var{sample} and return a \class{Dialect} subclass
204reflecting the parameters found. If the optional \var{delimiters} parameter
205is given, it is interpreted as a string containing possible valid delimiter
206characters.
Fred Drake96352682003-04-25 18:02:34 +0000207\end{methoddesc}
208
209\begin{methoddesc}{has_header}{sample}
210Analyze the sample text (presumed to be in CSV format) and return
211\constant{True} if the first row appears to be a series of column
212headers.
213\end{methoddesc}
214
215
Skip Montanarob4a04172003-03-20 23:29:12 +0000216The \module{csv} module defines the following constants:
217
Skip Montanaroa1045562003-06-04 15:30:13 +0000218\begin{datadesc}{QUOTE_ALL}
Skip Montanarob4a04172003-03-20 23:29:12 +0000219Instructs \class{writer} objects to quote all fields.
220\end{datadesc}
221
222\begin{datadesc}{QUOTE_MINIMAL}
223Instructs \class{writer} objects to only quote those fields which contain
Andrew McNamara8231de02005-01-12 11:47:57 +0000224special characters such as \var{delimiter}, \var{quotechar} or any of the
225characters in \var{lineterminator}.
Skip Montanarob4a04172003-03-20 23:29:12 +0000226\end{datadesc}
227
228\begin{datadesc}{QUOTE_NONNUMERIC}
Andrew McNamara8231de02005-01-12 11:47:57 +0000229Instructs \class{writer} objects to quote all non-numeric
230fields.
231
232Instructs the reader to convert all non-quoted fields to type \var{float}.
Skip Montanarob4a04172003-03-20 23:29:12 +0000233\end{datadesc}
234
235\begin{datadesc}{QUOTE_NONE}
236Instructs \class{writer} objects to never quote fields. When the current
237\var{delimiter} occurs in output data it is preceded by the current
Andrew McNamara8231de02005-01-12 11:47:57 +0000238\var{escapechar} character. If \var{escapechar} is not set, the writer
239will raise \exception{Error} if any characters that require escaping
240are encountered.
241
242Instructs \class{reader} to perform no special processing of quote characters.
Skip Montanarob4a04172003-03-20 23:29:12 +0000243\end{datadesc}
244
245
246The \module{csv} module defines the following exception:
247
248\begin{excdesc}{Error}
249Raised by any of the functions when an error is detected.
250\end{excdesc}
251
252
Fred Drake96352682003-04-25 18:02:34 +0000253\subsection{Dialects and Formatting Parameters\label{csv-fmt-params}}
Skip Montanarob4a04172003-03-20 23:29:12 +0000254
255To make it easier to specify the format of input and output records,
256specific formatting parameters are grouped together into dialects. A
257dialect is a subclass of the \class{Dialect} class having a set of specific
258methods and a single \method{validate()} method. When creating \class{reader}
259or \class{writer} objects, the programmer can specify a string or a subclass
260of the \class{Dialect} class as the dialect parameter. In addition to, or
261instead of, the \var{dialect} parameter, the programmer can also specify
262individual formatting parameters, which have the same names as the
Raymond Hettinger6f6d7b932003-08-31 05:44:54 +0000263attributes defined below for the \class{Dialect} class.
Skip Montanarob4a04172003-03-20 23:29:12 +0000264
Fred Drake96352682003-04-25 18:02:34 +0000265Dialects support the following attributes:
266
267\begin{memberdesc}[Dialect]{delimiter}
268A one-character string used to separate fields. It defaults to \code{','}.
269\end{memberdesc}
270
271\begin{memberdesc}[Dialect]{doublequote}
Andrew McNamara8231de02005-01-12 11:47:57 +0000272Controls how instances of \var{quotechar} appearing inside a field should
273be themselves be quoted. When \constant{True}, the character is doubled.
274When \constant{False}, the \var{escapechar} is used as a prefix to the
275\var{quotechar}. It defaults to \constant{True}.
276
277On output, if \var{doublequote} is \constant{False} and no
278\var{escapechar} is set, \exception{Error} is raised if a \var{quotechar}
279is found in a field.
Fred Drake96352682003-04-25 18:02:34 +0000280\end{memberdesc}
281
282\begin{memberdesc}[Dialect]{escapechar}
Andrew McNamara8231de02005-01-12 11:47:57 +0000283A one-character string used by the writer to escape the \var{delimiter} if
284\var{quoting} is set to \constant{QUOTE_NONE} and the \var{quotechar}
285if \var{doublequote} is \constant{False}. On reading, the \var{escapechar}
286removes any special meaning from the following character. It defaults
287to \constant{None}, which disables escaping.
Fred Drake96352682003-04-25 18:02:34 +0000288\end{memberdesc}
289
290\begin{memberdesc}[Dialect]{lineterminator}
Andrew McNamara8231de02005-01-12 11:47:57 +0000291The string used to terminate lines produced by the \class{writer}.
292It defaults to \code{'\e r\e n'}.
293
294\note{The \class{reader} is hard-coded to recognise either \code{'\e r'}
295or \code{'\e n'} as end-of-line, and ignores \var{lineterminator}. This
296behavior may change in the future.}
Fred Drake96352682003-04-25 18:02:34 +0000297\end{memberdesc}
298
299\begin{memberdesc}[Dialect]{quotechar}
Andrew McNamara8231de02005-01-12 11:47:57 +0000300A one-character string used to quote fields containing special characters,
301such as the \var{delimiter} or \var{quotechar}, or which contain new-line
302characters. It defaults to \code{'"'}.
Fred Drake96352682003-04-25 18:02:34 +0000303\end{memberdesc}
304
305\begin{memberdesc}[Dialect]{quoting}
Andrew McNamara8231de02005-01-12 11:47:57 +0000306Controls when quotes should be generated by the writer and recognised
307by the reader. It can take on any of the \constant{QUOTE_*} constants
308(see section~\ref{csv-contents}) and defaults to \constant{QUOTE_MINIMAL}.
Fred Drake96352682003-04-25 18:02:34 +0000309\end{memberdesc}
310
311\begin{memberdesc}[Dialect]{skipinitialspace}
312When \constant{True}, whitespace immediately following the \var{delimiter}
313is ignored. The default is \constant{False}.
314\end{memberdesc}
315
Skip Montanarob4a04172003-03-20 23:29:12 +0000316
317\subsection{Reader Objects}
318
Fred Drake96352682003-04-25 18:02:34 +0000319Reader objects (\class{DictReader} instances and objects returned by
Raymond Hettinger6f6d7b932003-08-31 05:44:54 +0000320the \function{reader()} function) have the following public methods:
Skip Montanarob4a04172003-03-20 23:29:12 +0000321
Fred Drake96352682003-04-25 18:02:34 +0000322\begin{methoddesc}[csv reader]{next}{}
Skip Montanarob4a04172003-03-20 23:29:12 +0000323Return the next row of the reader's iterable object as a list, parsed
324according to the current dialect.
325\end{methoddesc}
326
Andrew McNamara8231de02005-01-12 11:47:57 +0000327Reader objects have the following public attributes:
328
329\begin{memberdesc}[csv reader]{dialect}
330A read-only description of the dialect in use by the parser.
331\end{memberdesc}
332
333\begin{memberdesc}[csv reader]{line_num}
334 The number of lines read from the source iterator. This is not the same
335 as the number of records returned, as records can span multiple lines.
336\end{memberdesc}
337
Skip Montanarob4a04172003-03-20 23:29:12 +0000338
339\subsection{Writer Objects}
340
Skip Montanaroba0485a2004-01-21 13:47:04 +0000341\class{Writer} objects (\class{DictWriter} instances and objects returned by
342the \function{writer()} function) have the following public methods. A
343{}\var{row} must be a sequence of strings or numbers for \class{Writer}
344objects and a dictionary mapping fieldnames to strings or numbers (by
345passing them through \function{str()} first) for {}\class{DictWriter}
346objects. Note that complex numbers are written out surrounded by parens.
347This may cause some problems for other programs which read CSV files
348(assuming they support complex numbers at all).
Skip Montanarob4a04172003-03-20 23:29:12 +0000349
Fred Drake96352682003-04-25 18:02:34 +0000350\begin{methoddesc}[csv writer]{writerow}{row}
Skip Montanarob4a04172003-03-20 23:29:12 +0000351Write the \var{row} parameter to the writer's file object, formatted
352according to the current dialect.
353\end{methoddesc}
354
Fred Drake96352682003-04-25 18:02:34 +0000355\begin{methoddesc}[csv writer]{writerows}{rows}
Skip Montanaroba0485a2004-01-21 13:47:04 +0000356Write all the \var{rows} parameters (a list of \var{row} objects as
357described above) to the writer's file object, formatted
Skip Montanarob4a04172003-03-20 23:29:12 +0000358according to the current dialect.
359\end{methoddesc}
360
Andrew McNamara8231de02005-01-12 11:47:57 +0000361Writer objects have the following public attribute:
362
363\begin{memberdesc}[csv writer]{dialect}
364A read-only description of the dialect in use by the writer.
365\end{memberdesc}
366
367
Skip Montanarob4a04172003-03-20 23:29:12 +0000368
David Goodgercb30f972006-04-04 03:05:44 +0000369\subsection{Examples\label{csv-examples}}
Skip Montanarob4a04172003-03-20 23:29:12 +0000370
Andrew McNamara8231de02005-01-12 11:47:57 +0000371The simplest example of reading a CSV file:
Skip Montanarob4a04172003-03-20 23:29:12 +0000372
373\begin{verbatim}
Fred Drake96352682003-04-25 18:02:34 +0000374import csv
Andrew M. Kuchling6f937b12004-08-07 15:11:24 +0000375reader = csv.reader(open("some.csv", "rb"))
Fred Drake96352682003-04-25 18:02:34 +0000376for row in reader:
377 print row
Skip Montanarob4a04172003-03-20 23:29:12 +0000378\end{verbatim}
379
Andrew McNamara8231de02005-01-12 11:47:57 +0000380Reading a file with an alternate format:
Skip Montanaro2b2795a2004-07-08 19:49:10 +0000381
382\begin{verbatim}
383import csv
Andrew McNamara8231de02005-01-12 11:47:57 +0000384reader = csv.reader(open("passwd", "rb"), delimiter=':', quoting=csv.QUOTE_NONE)
Skip Montanaro2b2795a2004-07-08 19:49:10 +0000385for row in reader:
Andrew McNamara8231de02005-01-12 11:47:57 +0000386 print row
Skip Montanaro2b2795a2004-07-08 19:49:10 +0000387\end{verbatim}
388
Andrew McNamara8231de02005-01-12 11:47:57 +0000389The corresponding simplest possible writing example is:
Skip Montanarob4a04172003-03-20 23:29:12 +0000390
391\begin{verbatim}
Fred Drake96352682003-04-25 18:02:34 +0000392import csv
Andrew M. Kuchling6f937b12004-08-07 15:11:24 +0000393writer = csv.writer(open("some.csv", "wb"))
Andrew McNamara8231de02005-01-12 11:47:57 +0000394writer.writerows(someiterable)
Skip Montanarob4a04172003-03-20 23:29:12 +0000395\end{verbatim}
Andrew McNamara8231de02005-01-12 11:47:57 +0000396
397Registering a new dialect:
398
399\begin{verbatim}
400import csv
401
402csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE)
403
404reader = csv.reader(open("passwd", "rb"), 'unixpwd')
405\end{verbatim}
406
407A slightly more advanced use of the reader - catching and reporting errors:
408
409\begin{verbatim}
410import csv, sys
411filename = "some.csv"
412reader = csv.reader(open(filename, "rb"))
413try:
414 for row in reader:
415 print row
416except csv.Error, e:
417 sys.exit('file %s, line %d: %s' % (filename, reader.line_num, e))
418\end{verbatim}
419
420And while the module doesn't directly support parsing strings, it can
421easily be done:
422
423\begin{verbatim}
424import csv
Thomas Woutersbbdf6072006-02-16 14:57:05 +0000425for row in csv.reader(['one,two,three']):
426 print row
Andrew McNamara8231de02005-01-12 11:47:57 +0000427\end{verbatim}
428
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000429The \module{csv} module doesn't directly support reading and writing
David Goodgercb30f972006-04-04 03:05:44 +0000430Unicode, but it is 8-bit-clean save for some problems with \ASCII{} NUL
431characters. So you can write functions or classes that handle the
432encoding and decoding for you as long as you avoid encodings like
433UTF-16 that use NULs. UTF-8 is recommended.
434
435\function{unicode_csv_reader} below is a generator that wraps
436\class{csv.reader} to handle Unicode CSV data (a list of Unicode
437strings). \function{utf_8_encoder} is a generator that encodes the
438Unicode strings as UTF-8, one string (or row) at a time. The encoded
439strings are parsed by the CSV reader, and
440\function{unicode_csv_reader} decodes the UTF-8-encoded cells back
441into Unicode:
442
443\begin{verbatim}
444import csv
445
446def unicode_csv_reader(unicode_csv_data, dialect=csv.excel, **kwargs):
447 # csv.py doesn't do Unicode; encode temporarily as UTF-8:
448 csv_reader = csv.reader(utf_8_encoder(unicode_csv_data),
449 dialect=dialect, **kwargs)
450 for row in csv_reader:
451 # decode UTF-8 back to Unicode, cell by cell:
452 yield [unicode(cell, 'utf-8') for cell in row]
453
454def utf_8_encoder(unicode_csv_data):
455 for line in unicode_csv_data:
456 yield line.encode('utf-8')
457\end{verbatim}
458
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000459For all other encodings the following \class{UnicodeReader} and
460\class{UnicodeWriter} classes can be used. They take an additional
461\var{encoding} parameter in their constructor and make sure that the data
462passes the real reader or writer encoded as UTF-8:
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000463
464\begin{verbatim}
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000465import csv, codecs, cStringIO
466
467class UTF8Recoder:
468 """
469 Iterator that reads an encoded stream and reencodes the input to UTF-8
470 """
471 def __init__(self, f, encoding):
472 self.reader = codecs.getreader(encoding)(f)
473
474 def __iter__(self):
475 return self
476
477 def next(self):
478 return self.reader.next().encode("utf-8")
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000479
480class UnicodeReader:
David Goodgercb30f972006-04-04 03:05:44 +0000481 """
482 A CSV reader which will iterate over lines in the CSV file "f",
483 which is encoded in the given encoding.
484 """
485
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000486 def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000487 f = UTF8Recoder(f, encoding)
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000488 self.reader = csv.reader(f, dialect=dialect, **kwds)
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000489
490 def next(self):
491 row = self.reader.next()
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000492 return [unicode(s, "utf-8") for s in row]
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000493
494 def __iter__(self):
495 return self
496
497class UnicodeWriter:
David Goodgercb30f972006-04-04 03:05:44 +0000498 """
499 A CSV writer which will write rows to CSV file "f",
500 which is encoded in the given encoding.
501 """
502
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000503 def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000504 # Redirect output to a queue
505 self.queue = cStringIO.StringIO()
506 self.writer = csv.writer(self.queue, dialect=dialect, **kwds)
507 self.stream = f
508 self.encoder = codecs.getincrementalencoder(encoding)()
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000509
510 def writerow(self, row):
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000511 self.writer.writerow([s.encode("utf-8") for s in row])
512 # Fetch UTF-8 output from the queue ...
513 data = self.queue.getvalue()
514 data = data.decode("utf-8")
515 # ... and reencode it into the target encoding
516 data = self.encoder.encode(data)
517 # write to the target stream
518 self.stream.write(data)
519 # empty queue
520 self.queue.truncate(0)
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000521
522 def writerows(self, rows):
523 for row in rows:
524 self.writerow(row)
525\end{verbatim}