blob: 0e3294b76eab19af210c9821e39705ef44c90a11 [file] [log] [blame]
Skip Montanarob4a04172003-03-20 23:29:12 +00001\section{\module{csv} --- CSV File Reading and Writing}
2
3\declaremodule{standard}{csv}
4\modulesynopsis{Write and read tabular data to and from delimited files.}
Skip Montanaro3bd3c842003-04-24 18:47:31 +00005\sectionauthor{Skip Montanaro}{skip@pobox.com}
Skip Montanarob4a04172003-03-20 23:29:12 +00006
7\versionadded{2.3}
8\index{csv}
9\indexii{data}{tabular}
10
11The so-called CSV (Comma Separated Values) format is the most common import
12and export format for spreadsheets and databases. There is no ``CSV
13standard'', so the format is operationally defined by the many applications
14which read and write it. The lack of a standard means that subtle
15differences often exist in the data produced and consumed by different
16applications. These differences can make it annoying to process CSV files
17from multiple sources. Still, while the delimiters and quoting characters
18vary, the overall format is similar enough that it is possible to write a
19single module which can efficiently manipulate such data, hiding the details
20of reading and writing the data from the programmer.
21
Skip Montanaro5d0136e2003-04-25 15:14:49 +000022The \module{csv} module implements classes to read and write tabular data in
Skip Montanarob4a04172003-03-20 23:29:12 +000023CSV format. It allows programmers to say, ``write this data in the format
24preferred by Excel,'' or ``read data from this file which was generated by
25Excel,'' without knowing the precise details of the CSV format used by
26Excel. Programmers can also describe the CSV formats understood by other
27applications or define their own special-purpose CSV formats.
28
Skip Montanaro5d0136e2003-04-25 15:14:49 +000029The \module{csv} module's \class{reader} and \class{writer} objects read and
Skip Montanarob4a04172003-03-20 23:29:12 +000030write sequences. Programmers can also read and write data in dictionary
31form using the \class{DictReader} and \class{DictWriter} classes.
32
Fred Drake96352682003-04-25 18:02:34 +000033\begin{notice}
34 This version of the \module{csv} module doesn't support Unicode
35 input. Also, there are currently some issues regarding \ASCII{} NUL
David Goodgercb30f972006-04-04 03:05:44 +000036 characters. Accordingly, all input should be UTF-8 or printable
37 \ASCII{} to be safe; see the examples in section~\ref{csv-examples}.
38 These restrictions will be removed in the future.
Fred Drake96352682003-04-25 18:02:34 +000039\end{notice}
Skip Montanarob4a04172003-03-20 23:29:12 +000040
41\begin{seealso}
42% \seemodule{array}{Arrays of uniformly types numeric values.}
43 \seepep{305}{CSV File API}
44 {The Python Enhancement Proposal which proposed this addition
45 to Python.}
46\end{seealso}
47
48
Raymond Hettinger6f6d7b932003-08-31 05:44:54 +000049\subsection{Module Contents \label{csv-contents}}
Skip Montanarob4a04172003-03-20 23:29:12 +000050
Skip Montanaro5d0136e2003-04-25 15:14:49 +000051The \module{csv} module defines the following functions:
Skip Montanarob4a04172003-03-20 23:29:12 +000052
53\begin{funcdesc}{reader}{csvfile\optional{,
Andrew McNamara8231de02005-01-12 11:47:57 +000054 dialect=\code{'excel'}}\optional{, fmtparam}}
Skip Montanarob4a04172003-03-20 23:29:12 +000055Return a reader object which will iterate over lines in the given
56{}\var{csvfile}. \var{csvfile} can be any object which supports the
57iterator protocol and returns a string each time its \method{next}
Andrew McNamara8231de02005-01-12 11:47:57 +000058method is called - file objects and list objects are both suitable.
59If \var{csvfile} is a file object, it must be opened with
Skip Montanaro5e4e39f2003-07-02 15:32:48 +000060the 'b' flag on platforms where that makes a difference. An optional
61{}\var{dialect} parameter can be given
Skip Montanarob4a04172003-03-20 23:29:12 +000062which is used to define a set of parameters specific to a particular CSV
63dialect. It may be an instance of a subclass of the \class{Dialect}
64class or one of the strings returned by the \function{list_dialects}
65function. The other optional {}\var{fmtparam} keyword arguments can be
66given to override individual formatting parameters in the current
67dialect. For more information about the dialect and formatting
Raymond Hettinger6e380cd2003-09-10 18:54:49 +000068parameters, see section~\ref{csv-fmt-params}, ``Dialects and Formatting
Skip Montanarob4a04172003-03-20 23:29:12 +000069Parameters'' for details of these parameters.
70
71All data read are returned as strings. No automatic data type
72conversion is performed.
Skip Montanaroabd51a32006-07-29 20:06:05 +000073
74\versionchanged[
75If literal newlines are important within a field, users need to read their
76file in a way that preserves the newlines. The behavior before 2.5 would
77introduce spurious characters into quoted fields, with no way for the user
78to control that behavior. The previous behavior caused considerable
79problems, particularly on platforms that did not use the unix line ending
80conventions, or with files that originated on those platforms - users were
81finding mysterious newlines where they didn't expect them.
82]{2.5}
83
Skip Montanarob4a04172003-03-20 23:29:12 +000084\end{funcdesc}
85
86\begin{funcdesc}{writer}{csvfile\optional{,
Andrew McNamara8231de02005-01-12 11:47:57 +000087 dialect=\code{'excel'}}\optional{, fmtparam}}
Skip Montanarob4a04172003-03-20 23:29:12 +000088Return a writer object responsible for converting the user's data into
Skip Montanaro5e4e39f2003-07-02 15:32:48 +000089delimited strings on the given file-like object. \var{csvfile} can be any
90object with a \function{write} method. If \var{csvfile} is a file object,
91it must be opened with the 'b' flag on platforms where that makes a
92difference. An optional
Skip Montanarob4a04172003-03-20 23:29:12 +000093{}\var{dialect} parameter can be given which is used to define a set of
94parameters specific to a particular CSV dialect. It may be an instance
95of a subclass of the \class{Dialect} class or one of the strings
96returned by the \function{list_dialects} function. The other optional
97{}\var{fmtparam} keyword arguments can be given to override individual
98formatting parameters in the current dialect. For more information
99about the dialect and formatting parameters, see
Raymond Hettinger6e380cd2003-09-10 18:54:49 +0000100section~\ref{csv-fmt-params}, ``Dialects and Formatting Parameters'' for
Skip Montanarob4a04172003-03-20 23:29:12 +0000101details of these parameters. To make it as easy as possible to
102interface with modules which implement the DB API, the value
103\constant{None} is written as the empty string. While this isn't a
104reversible transformation, it makes it easier to dump SQL NULL data values
105to CSV files without preprocessing the data returned from a
106\code{cursor.fetch*()} call. All other non-string data are stringified
107with \function{str()} before being written.
108\end{funcdesc}
109
Andrew McNamara8231de02005-01-12 11:47:57 +0000110\begin{funcdesc}{register_dialect}{name\optional{, dialect}\optional{, fmtparam}}
111Associate \var{dialect} with \var{name}. \var{name} must be a string
112or Unicode object. The dialect can be specified either by passing a
113sub-class of \class{Dialect}, or by \var{fmtparam} keyword arguments,
114or both, with keyword arguments overriding parameters of the dialect.
115For more information about the dialect and formatting parameters, see
116section~\ref{csv-fmt-params}, ``Dialects and Formatting Parameters''
117for details of these parameters.
Skip Montanarob4a04172003-03-20 23:29:12 +0000118\end{funcdesc}
119
120\begin{funcdesc}{unregister_dialect}{name}
121Delete the dialect associated with \var{name} from the dialect registry. An
122\exception{Error} is raised if \var{name} is not a registered dialect
123name.
124\end{funcdesc}
125
126\begin{funcdesc}{get_dialect}{name}
127Return the dialect associated with \var{name}. An \exception{Error} is
128raised if \var{name} is not a registered dialect name.
129\end{funcdesc}
130
131\begin{funcdesc}{list_dialects}{}
132Return the names of all registered dialects.
133\end{funcdesc}
134
Andrew McNamara8231de02005-01-12 11:47:57 +0000135\begin{funcdesc}{field_size_limit}{\optional{new_limit}}
136 Returns the current maximum field size allowed by the parser. If
137 \var{new_limit} is given, this becomes the new limit.
138 \versionadded{2.5}
139\end{funcdesc}
140
Skip Montanarob4a04172003-03-20 23:29:12 +0000141
Skip Montanaro5d0136e2003-04-25 15:14:49 +0000142The \module{csv} module defines the following classes:
Skip Montanarob4a04172003-03-20 23:29:12 +0000143
Skip Montanarodffeed32003-10-03 14:03:01 +0000144\begin{classdesc}{DictReader}{csvfile\optional{,
145 fieldnames=\constant{None},\optional{,
Fred Drake96352682003-04-25 18:02:34 +0000146 restkey=\constant{None}\optional{,
147 restval=\constant{None}\optional{,
Skip Montanarob4a04172003-03-20 23:29:12 +0000148 dialect=\code{'excel'}\optional{,
Skip Montanaro10659f22004-04-16 03:21:01 +0000149 *args, **kwds}}}}}}
Skip Montanarob4a04172003-03-20 23:29:12 +0000150Create an object which operates like a regular reader but maps the
Skip Montanarodffeed32003-10-03 14:03:01 +0000151information read into a dict whose keys are given by the optional
152{} \var{fieldnames}
153parameter. If the \var{fieldnames} parameter is omitted, the values in
154the first row of the \var{csvfile} will be used as the fieldnames.
155If the row read has fewer fields than the fieldnames sequence,
Skip Montanarob4a04172003-03-20 23:29:12 +0000156the value of \var{restval} will be used as the default value. If the row
157read has more fields than the fieldnames sequence, the remaining data is
158added as a sequence keyed by the value of \var{restkey}. If the row read
159has fewer fields than the fieldnames sequence, the remaining keys take the
Skip Montanaro10659f22004-04-16 03:21:01 +0000160value of the optional \var{restval} parameter. Any other optional or
161keyword arguments are passed to the underlying \class{reader} instance.
Skip Montanarob4a04172003-03-20 23:29:12 +0000162\end{classdesc}
163
164
165\begin{classdesc}{DictWriter}{csvfile, fieldnames\optional{,
166 restval=""\optional{,
167 extrasaction=\code{'raise'}\optional{,
Skip Montanaro10659f22004-04-16 03:21:01 +0000168 dialect=\code{'excel'}\optional{,
169 *args, **kwds}}}}}
Skip Montanarob4a04172003-03-20 23:29:12 +0000170Create an object which operates like a regular writer but maps dictionaries
171onto output rows. The \var{fieldnames} parameter identifies the order in
172which values in the dictionary passed to the \method{writerow()} method are
173written to the \var{csvfile}. The optional \var{restval} parameter
174specifies the value to be written if the dictionary is missing a key in
175\var{fieldnames}. If the dictionary passed to the \method{writerow()}
176method contains a key not found in \var{fieldnames}, the optional
177\var{extrasaction} parameter indicates what action to take. If it is set
178to \code{'raise'} a \exception{ValueError} is raised. If it is set to
Skip Montanaro10659f22004-04-16 03:21:01 +0000179\code{'ignore'}, extra values in the dictionary are ignored. Any other
180optional or keyword arguments are passed to the underlying \class{writer}
181instance.
Skip Montanarodffeed32003-10-03 14:03:01 +0000182
183Note that unlike the \class{DictReader} class, the \var{fieldnames}
184parameter of the \class{DictWriter} is not optional. Since Python's
185\class{dict} objects are not ordered, there is not enough information
186available to deduce the order in which the row should be written to the
187\var{csvfile}.
188
Skip Montanarob4a04172003-03-20 23:29:12 +0000189\end{classdesc}
190
Skip Montanarob4a04172003-03-20 23:29:12 +0000191\begin{classdesc*}{Dialect}{}
192The \class{Dialect} class is a container class relied on primarily for its
193attributes, which are used to define the parameters for a specific
Fred Drake96352682003-04-25 18:02:34 +0000194\class{reader} or \class{writer} instance.
Skip Montanarob4a04172003-03-20 23:29:12 +0000195\end{classdesc*}
196
Skip Montanarobb0c9dc2005-01-05 06:58:15 +0000197\begin{classdesc}{excel}{}
198The \class{excel} class defines the usual properties of an Excel-generated
199CSV file.
200\end{classdesc}
201
202\begin{classdesc}{excel_tab}{}
203The \class{excel_tab} class defines the usual properties of an
204Excel-generated TAB-delimited file.
205\end{classdesc}
206
Skip Montanaro77892372003-05-19 15:33:36 +0000207\begin{classdesc}{Sniffer}{}
208The \class{Sniffer} class is used to deduce the format of a CSV file.
Fred Drake96352682003-04-25 18:02:34 +0000209\end{classdesc}
210
Skip Montanaro8bdaac72005-12-28 15:56:58 +0000211The \class{Sniffer} class provides two methods:
Fred Drake96352682003-04-25 18:02:34 +0000212
Skip Montanaro77892372003-05-19 15:33:36 +0000213\begin{methoddesc}{sniff}{sample\optional{,delimiters=None}}
214Analyze the given \var{sample} and return a \class{Dialect} subclass
215reflecting the parameters found. If the optional \var{delimiters} parameter
216is given, it is interpreted as a string containing possible valid delimiter
217characters.
Fred Drake96352682003-04-25 18:02:34 +0000218\end{methoddesc}
219
220\begin{methoddesc}{has_header}{sample}
221Analyze the sample text (presumed to be in CSV format) and return
222\constant{True} if the first row appears to be a series of column
223headers.
224\end{methoddesc}
225
226
Skip Montanarob4a04172003-03-20 23:29:12 +0000227The \module{csv} module defines the following constants:
228
Skip Montanaroa1045562003-06-04 15:30:13 +0000229\begin{datadesc}{QUOTE_ALL}
Skip Montanarob4a04172003-03-20 23:29:12 +0000230Instructs \class{writer} objects to quote all fields.
231\end{datadesc}
232
233\begin{datadesc}{QUOTE_MINIMAL}
234Instructs \class{writer} objects to only quote those fields which contain
Andrew McNamara8231de02005-01-12 11:47:57 +0000235special characters such as \var{delimiter}, \var{quotechar} or any of the
236characters in \var{lineterminator}.
Skip Montanarob4a04172003-03-20 23:29:12 +0000237\end{datadesc}
238
239\begin{datadesc}{QUOTE_NONNUMERIC}
Andrew McNamara8231de02005-01-12 11:47:57 +0000240Instructs \class{writer} objects to quote all non-numeric
241fields.
242
243Instructs the reader to convert all non-quoted fields to type \var{float}.
Skip Montanarob4a04172003-03-20 23:29:12 +0000244\end{datadesc}
245
246\begin{datadesc}{QUOTE_NONE}
247Instructs \class{writer} objects to never quote fields. When the current
248\var{delimiter} occurs in output data it is preceded by the current
Andrew McNamara8231de02005-01-12 11:47:57 +0000249\var{escapechar} character. If \var{escapechar} is not set, the writer
250will raise \exception{Error} if any characters that require escaping
251are encountered.
252
253Instructs \class{reader} to perform no special processing of quote characters.
Skip Montanarob4a04172003-03-20 23:29:12 +0000254\end{datadesc}
255
256
257The \module{csv} module defines the following exception:
258
259\begin{excdesc}{Error}
260Raised by any of the functions when an error is detected.
261\end{excdesc}
262
263
Fred Drake96352682003-04-25 18:02:34 +0000264\subsection{Dialects and Formatting Parameters\label{csv-fmt-params}}
Skip Montanarob4a04172003-03-20 23:29:12 +0000265
266To make it easier to specify the format of input and output records,
267specific formatting parameters are grouped together into dialects. A
268dialect is a subclass of the \class{Dialect} class having a set of specific
269methods and a single \method{validate()} method. When creating \class{reader}
270or \class{writer} objects, the programmer can specify a string or a subclass
271of the \class{Dialect} class as the dialect parameter. In addition to, or
272instead of, the \var{dialect} parameter, the programmer can also specify
273individual formatting parameters, which have the same names as the
Raymond Hettinger6f6d7b932003-08-31 05:44:54 +0000274attributes defined below for the \class{Dialect} class.
Skip Montanarob4a04172003-03-20 23:29:12 +0000275
Fred Drake96352682003-04-25 18:02:34 +0000276Dialects support the following attributes:
277
278\begin{memberdesc}[Dialect]{delimiter}
279A one-character string used to separate fields. It defaults to \code{','}.
280\end{memberdesc}
281
282\begin{memberdesc}[Dialect]{doublequote}
Andrew McNamara8231de02005-01-12 11:47:57 +0000283Controls how instances of \var{quotechar} appearing inside a field should
284be themselves be quoted. When \constant{True}, the character is doubled.
285When \constant{False}, the \var{escapechar} is used as a prefix to the
286\var{quotechar}. It defaults to \constant{True}.
287
288On output, if \var{doublequote} is \constant{False} and no
289\var{escapechar} is set, \exception{Error} is raised if a \var{quotechar}
290is found in a field.
Fred Drake96352682003-04-25 18:02:34 +0000291\end{memberdesc}
292
293\begin{memberdesc}[Dialect]{escapechar}
Andrew McNamara8231de02005-01-12 11:47:57 +0000294A one-character string used by the writer to escape the \var{delimiter} if
295\var{quoting} is set to \constant{QUOTE_NONE} and the \var{quotechar}
296if \var{doublequote} is \constant{False}. On reading, the \var{escapechar}
297removes any special meaning from the following character. It defaults
298to \constant{None}, which disables escaping.
Fred Drake96352682003-04-25 18:02:34 +0000299\end{memberdesc}
300
301\begin{memberdesc}[Dialect]{lineterminator}
Andrew McNamara8231de02005-01-12 11:47:57 +0000302The string used to terminate lines produced by the \class{writer}.
303It defaults to \code{'\e r\e n'}.
304
305\note{The \class{reader} is hard-coded to recognise either \code{'\e r'}
306or \code{'\e n'} as end-of-line, and ignores \var{lineterminator}. This
307behavior may change in the future.}
Fred Drake96352682003-04-25 18:02:34 +0000308\end{memberdesc}
309
310\begin{memberdesc}[Dialect]{quotechar}
Andrew McNamara8231de02005-01-12 11:47:57 +0000311A one-character string used to quote fields containing special characters,
312such as the \var{delimiter} or \var{quotechar}, or which contain new-line
313characters. It defaults to \code{'"'}.
Fred Drake96352682003-04-25 18:02:34 +0000314\end{memberdesc}
315
316\begin{memberdesc}[Dialect]{quoting}
Andrew McNamara8231de02005-01-12 11:47:57 +0000317Controls when quotes should be generated by the writer and recognised
318by the reader. It can take on any of the \constant{QUOTE_*} constants
319(see section~\ref{csv-contents}) and defaults to \constant{QUOTE_MINIMAL}.
Fred Drake96352682003-04-25 18:02:34 +0000320\end{memberdesc}
321
322\begin{memberdesc}[Dialect]{skipinitialspace}
323When \constant{True}, whitespace immediately following the \var{delimiter}
324is ignored. The default is \constant{False}.
325\end{memberdesc}
326
Skip Montanarob4a04172003-03-20 23:29:12 +0000327
328\subsection{Reader Objects}
329
Fred Drake96352682003-04-25 18:02:34 +0000330Reader objects (\class{DictReader} instances and objects returned by
Raymond Hettinger6f6d7b932003-08-31 05:44:54 +0000331the \function{reader()} function) have the following public methods:
Skip Montanarob4a04172003-03-20 23:29:12 +0000332
Fred Drake96352682003-04-25 18:02:34 +0000333\begin{methoddesc}[csv reader]{next}{}
Skip Montanarob4a04172003-03-20 23:29:12 +0000334Return the next row of the reader's iterable object as a list, parsed
335according to the current dialect.
336\end{methoddesc}
337
Andrew McNamara8231de02005-01-12 11:47:57 +0000338Reader objects have the following public attributes:
339
340\begin{memberdesc}[csv reader]{dialect}
341A read-only description of the dialect in use by the parser.
342\end{memberdesc}
343
344\begin{memberdesc}[csv reader]{line_num}
345 The number of lines read from the source iterator. This is not the same
346 as the number of records returned, as records can span multiple lines.
347\end{memberdesc}
348
Skip Montanarob4a04172003-03-20 23:29:12 +0000349
350\subsection{Writer Objects}
351
Skip Montanaroba0485a2004-01-21 13:47:04 +0000352\class{Writer} objects (\class{DictWriter} instances and objects returned by
353the \function{writer()} function) have the following public methods. A
354{}\var{row} must be a sequence of strings or numbers for \class{Writer}
355objects and a dictionary mapping fieldnames to strings or numbers (by
356passing them through \function{str()} first) for {}\class{DictWriter}
357objects. Note that complex numbers are written out surrounded by parens.
358This may cause some problems for other programs which read CSV files
359(assuming they support complex numbers at all).
Skip Montanarob4a04172003-03-20 23:29:12 +0000360
Fred Drake96352682003-04-25 18:02:34 +0000361\begin{methoddesc}[csv writer]{writerow}{row}
Skip Montanarob4a04172003-03-20 23:29:12 +0000362Write the \var{row} parameter to the writer's file object, formatted
363according to the current dialect.
364\end{methoddesc}
365
Fred Drake96352682003-04-25 18:02:34 +0000366\begin{methoddesc}[csv writer]{writerows}{rows}
Skip Montanaroba0485a2004-01-21 13:47:04 +0000367Write all the \var{rows} parameters (a list of \var{row} objects as
368described above) to the writer's file object, formatted
Skip Montanarob4a04172003-03-20 23:29:12 +0000369according to the current dialect.
370\end{methoddesc}
371
Andrew McNamara8231de02005-01-12 11:47:57 +0000372Writer objects have the following public attribute:
373
374\begin{memberdesc}[csv writer]{dialect}
375A read-only description of the dialect in use by the writer.
376\end{memberdesc}
377
378
Skip Montanarob4a04172003-03-20 23:29:12 +0000379
David Goodgercb30f972006-04-04 03:05:44 +0000380\subsection{Examples\label{csv-examples}}
Skip Montanarob4a04172003-03-20 23:29:12 +0000381
Andrew McNamara8231de02005-01-12 11:47:57 +0000382The simplest example of reading a CSV file:
Skip Montanarob4a04172003-03-20 23:29:12 +0000383
384\begin{verbatim}
Fred Drake96352682003-04-25 18:02:34 +0000385import csv
Andrew M. Kuchling6f937b12004-08-07 15:11:24 +0000386reader = csv.reader(open("some.csv", "rb"))
Fred Drake96352682003-04-25 18:02:34 +0000387for row in reader:
388 print row
Skip Montanarob4a04172003-03-20 23:29:12 +0000389\end{verbatim}
390
Andrew McNamara8231de02005-01-12 11:47:57 +0000391Reading a file with an alternate format:
Skip Montanaro2b2795a2004-07-08 19:49:10 +0000392
393\begin{verbatim}
394import csv
Andrew McNamara8231de02005-01-12 11:47:57 +0000395reader = csv.reader(open("passwd", "rb"), delimiter=':', quoting=csv.QUOTE_NONE)
Skip Montanaro2b2795a2004-07-08 19:49:10 +0000396for row in reader:
Andrew McNamara8231de02005-01-12 11:47:57 +0000397 print row
Skip Montanaro2b2795a2004-07-08 19:49:10 +0000398\end{verbatim}
399
Andrew McNamara8231de02005-01-12 11:47:57 +0000400The corresponding simplest possible writing example is:
Skip Montanarob4a04172003-03-20 23:29:12 +0000401
402\begin{verbatim}
Fred Drake96352682003-04-25 18:02:34 +0000403import csv
Andrew M. Kuchling6f937b12004-08-07 15:11:24 +0000404writer = csv.writer(open("some.csv", "wb"))
Andrew McNamara8231de02005-01-12 11:47:57 +0000405writer.writerows(someiterable)
Skip Montanarob4a04172003-03-20 23:29:12 +0000406\end{verbatim}
Andrew McNamara8231de02005-01-12 11:47:57 +0000407
408Registering a new dialect:
409
410\begin{verbatim}
411import csv
412
413csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE)
414
415reader = csv.reader(open("passwd", "rb"), 'unixpwd')
416\end{verbatim}
417
418A slightly more advanced use of the reader - catching and reporting errors:
419
420\begin{verbatim}
421import csv, sys
422filename = "some.csv"
423reader = csv.reader(open(filename, "rb"))
424try:
425 for row in reader:
426 print row
427except csv.Error, e:
428 sys.exit('file %s, line %d: %s' % (filename, reader.line_num, e))
429\end{verbatim}
430
431And while the module doesn't directly support parsing strings, it can
432easily be done:
433
434\begin{verbatim}
435import csv
Thomas Woutersbbdf6072006-02-16 14:57:05 +0000436for row in csv.reader(['one,two,three']):
437 print row
Andrew McNamara8231de02005-01-12 11:47:57 +0000438\end{verbatim}
439
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000440The \module{csv} module doesn't directly support reading and writing
David Goodgercb30f972006-04-04 03:05:44 +0000441Unicode, but it is 8-bit-clean save for some problems with \ASCII{} NUL
442characters. So you can write functions or classes that handle the
443encoding and decoding for you as long as you avoid encodings like
444UTF-16 that use NULs. UTF-8 is recommended.
445
446\function{unicode_csv_reader} below is a generator that wraps
447\class{csv.reader} to handle Unicode CSV data (a list of Unicode
448strings). \function{utf_8_encoder} is a generator that encodes the
449Unicode strings as UTF-8, one string (or row) at a time. The encoded
450strings are parsed by the CSV reader, and
451\function{unicode_csv_reader} decodes the UTF-8-encoded cells back
452into Unicode:
453
454\begin{verbatim}
455import csv
456
457def unicode_csv_reader(unicode_csv_data, dialect=csv.excel, **kwargs):
458 # csv.py doesn't do Unicode; encode temporarily as UTF-8:
459 csv_reader = csv.reader(utf_8_encoder(unicode_csv_data),
460 dialect=dialect, **kwargs)
461 for row in csv_reader:
462 # decode UTF-8 back to Unicode, cell by cell:
463 yield [unicode(cell, 'utf-8') for cell in row]
464
465def utf_8_encoder(unicode_csv_data):
466 for line in unicode_csv_data:
467 yield line.encode('utf-8')
468\end{verbatim}
469
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000470For all other encodings the following \class{UnicodeReader} and
471\class{UnicodeWriter} classes can be used. They take an additional
472\var{encoding} parameter in their constructor and make sure that the data
473passes the real reader or writer encoded as UTF-8:
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000474
475\begin{verbatim}
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000476import csv, codecs, cStringIO
477
478class UTF8Recoder:
479 """
480 Iterator that reads an encoded stream and reencodes the input to UTF-8
481 """
482 def __init__(self, f, encoding):
483 self.reader = codecs.getreader(encoding)(f)
484
485 def __iter__(self):
486 return self
487
488 def next(self):
489 return self.reader.next().encode("utf-8")
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000490
491class UnicodeReader:
David Goodgercb30f972006-04-04 03:05:44 +0000492 """
493 A CSV reader which will iterate over lines in the CSV file "f",
494 which is encoded in the given encoding.
495 """
496
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000497 def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000498 f = UTF8Recoder(f, encoding)
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000499 self.reader = csv.reader(f, dialect=dialect, **kwds)
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000500
501 def next(self):
502 row = self.reader.next()
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000503 return [unicode(s, "utf-8") for s in row]
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000504
505 def __iter__(self):
506 return self
507
508class UnicodeWriter:
David Goodgercb30f972006-04-04 03:05:44 +0000509 """
510 A CSV writer which will write rows to CSV file "f",
511 which is encoded in the given encoding.
512 """
513
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000514 def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000515 # Redirect output to a queue
516 self.queue = cStringIO.StringIO()
517 self.writer = csv.writer(self.queue, dialect=dialect, **kwds)
518 self.stream = f
519 self.encoder = codecs.getincrementalencoder(encoding)()
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000520
521 def writerow(self, row):
Walter Dörwaldf7bc5f92006-04-04 17:32:49 +0000522 self.writer.writerow([s.encode("utf-8") for s in row])
523 # Fetch UTF-8 output from the queue ...
524 data = self.queue.getvalue()
525 data = data.decode("utf-8")
526 # ... and reencode it into the target encoding
527 data = self.encoder.encode(data)
528 # write to the target stream
529 self.stream.write(data)
530 # empty queue
531 self.queue.truncate(0)
Skip Montanaro5011c3f2005-03-18 16:56:37 +0000532
533 def writerows(self, rows):
534 for row in rows:
535 self.writerow(row)
536\end{verbatim}