[Python-Dev] [Fwd: PEP 0305 (small problem with the CSV reader)]

[Andrew McNamara]

>First of all, let me say thank you for the CSV module.

Thanks.

>I've been using it and today is the first time I hit a minor bump in the road.
>What happened is I opened this file with genome annotations with a
>long field and the error "field larger than field limit" showed up.
>From what I can see it is in the "static int parse_add_char(ReaderObj
>*self, char c)" function.
>This function uses the static long field_limit = 128 * 1024;	/* max
>parsed field size */
>I'm not sure if this is supposed to be recomputed or if there is
>something I need to do to change it, but for right now it just says my
>row is bigger than 131,072 and stops.
>I don't think Python 2.5 has any such string length limitations and
>this shouldn't be.

This limit was added back in January 2005 to provide some protection
against the situation where the parser is returning fields directly from
a file, and the file contains a mismatched quote character: this would
otherwise result in the entire file being unexpectedly read into memory.

You can change the limit with the csv.field_size_limit() method. As
you note, it defaults to 128K, but you can set it to anything up to
(2**31)-1 or 2147483647 (about 2 billion).

BTW, I've taken the liberty of CC'ing this to the python-dev list, so
the motivation for this feature is recorded - it caused me some head
scratching, and I added it.

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/