[Baypiggies] Baypiggies Digest, Vol 67, Issue 17

nyxtom at gmail.com nyxtom at gmail.com
Tue May 17 23:06:29 CEST 2011 

+1 on that talk
On May 17, 2011 1:39 PM, <baypiggies-request at python.org> wrote:
>
> Send Baypiggies mailing list submissions to
>        baypiggies at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://mail.python.org/mailman/listinfo/baypiggies
> or, via email, send a message with subject or body 'help' to
>        baypiggies-request at python.org
>
> You can reach the person managing the list at
>        baypiggies-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Baypiggies digest..."
>
>
> Today's Topics:
>
>   1. Re:  reading very large files (Alexandre Conrad)
>   2.  Interested in a PyPy talk? (jim)
>   3. Re:  reading very large files (Lucas Wiman)
>   4. Re:  reading very large files (Lucas Wiman)
>   5.  Fwd:  Interested in a PyPy talk? (Abhishek Pratap)
>   6. Re:  Interested in a PyPy talk? (Tony Cappellini)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 17 May 2011 11:21:58 -0700
> From: Alexandre Conrad <alexandre.conrad at gmail.com>
> To: Simeon Franklin <simeonf at gmail.com>
> Cc: Baypiggies <baypiggies at python.org>
> Subject: Re: [Baypiggies] reading very large files
> Message-ID: <BANLkTinZReTDS-uOSz7_aDAXEZPtxkcNRA at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> 2011/5/17 Simeon Franklin <simeonf at gmail.com>:
> > I missed the list too. Curse that reply button :)
>
> For those using gmail, you can activate the reply button to be "reply
> to all" by default. It's a gmail "labs" feature.
>
> --
> Alex | twitter.com/alexconrad
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 17 May 2011 11:47:13 -0700
> From: jim <jim at systemateka.com>
> To: Baypiggies <baypiggies at python.org>
> Subject: [Baypiggies] Interested in a PyPy talk?
> Message-ID: <1305658033.1712.61.camel at jim-LAPTOP>
> Content-Type: text/plain; charset="UTF-8"
>
>
>    Please respond if you're interested in having
> Dan Roberts talk on PyPy at a BayPIGgies meeting.
> No reponses, no PyPy talk.
> With thanks,
> jim
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 17 May 2011 12:26:16 -0700
> From: Lucas Wiman <lucas.wiman at gmail.com>
> To: baypiggies at python.org
> Subject: Re: [Baypiggies] reading very large files
> Message-ID: <BANLkTimVAneJGaE6w6ZB6DKXRbQWF2bK+g at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Tue, May 17, 2011 at 10:56 AM, <baypiggies-request at python.org> wrote:
>
> >
> > I wish to read a large data file (file size is around 1.8 MB) and
> > manipulate
> > the data in this file. Just reading and writing the first 500 lines of
this
> > file is causing a problem. I wrote:
> >
> > fin = open('gene-GS00471-DNA_B01_1101_37-ASM.tsv')
> > count = 0
> > for i in fin.readlines():
> >    print i
> >    count += 1
> >    if count >= 500:
> >        break
> >
> > and got this error msg:
> >
> > Traceback (most recent call last):
> >  File
> >
> >
"H:\genome_4_omics_study\GS000003696-DID\GS00471-DNA_B01_1101_37-ASM\GS00471-DNA_B01\ASM\gene-GS00471-DNA_B01_1101_37-ASM.tsv\test.py",
> > line 3, in <module>
> >    for i in fin.readlines():
> > MemoryError
> >
>
> If your data is actually a tsv (tab-separated value format), you should be
> using the csv module for actually iterating over lines in it.  Just set
the
> delimiter to '\t' and look at the docs at
> http://docs.python.org/library/csv.html
>
> You should also generally use the "with" syntax when dealing with files
> since it handles closing the file object for you (probably not an issue
when
> you're just reading from a single file, but best practices nonetheless).
>  Here's how I would deal with your situation:
>
> import csv
>
> with open('gene-GS00471-DNA_B01_1101_37-ASM.tsv', 'r') as f:
>    r = csv.reader(f, delimiter='\t')
>    for row in r:
>        # row is a list of strings that correspond to the columns in your
> file
>        do_stuff_with_the_row(row)
> # your file object f is now closed
>
> Best wishes,
> Lucas Wiman
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
http://mail.python.org/pipermail/baypiggies/attachments/20110517/d56bc01c/attachment-0001.html
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 17 May 2011 12:34:47 -0700
> From: Lucas Wiman <lucas.wiman at gmail.com>
> To: baypiggies at python.org
> Subject: Re: [Baypiggies] reading very large files
> Message-ID: <BANLkTimNndVkj-U=pKm9hh5OMk9K4BEpEg at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> It's also extremely surprising to me that reading a 1.8MB file is causing
a
> memory error.  That's actually not a particularly large file, and if it is
> causing a memory error, there must be something wrong with the your Python
> configuration or build.
>
> Best,
> Lucas
>
> On Tue, May 17, 2011 at 12:26 PM, Lucas Wiman <lucas.wiman at gmail.com>
wrote:
>
> >
> >
> > On Tue, May 17, 2011 at 10:56 AM, <baypiggies-request at python.org> wrote:
> >
> >>
> >> I wish to read a large data file (file size is around 1.8 MB) and
> >> manipulate
> >> the data in this file. Just reading and writing the first 500 lines of
> >> this
> >> file is causing a problem. I wrote:
> >>
> >> fin = open('gene-GS00471-DNA_B01_1101_37-ASM.tsv')
> >> count = 0
> >> for i in fin.readlines():
> >>    print i
> >>    count += 1
> >>    if count >= 500:
> >>        break
> >>
> >> and got this error msg:
> >>
> >> Traceback (most recent call last):
> >>  File
> >>
> >>
"H:\genome_4_omics_study\GS000003696-DID\GS00471-DNA_B01_1101_37-ASM\GS00471-DNA_B01\ASM\gene-GS00471-DNA_B01_1101_37-ASM.tsv\test.py",
> >> line 3, in <module>
> >>    for i in fin.readlines():
> >> MemoryError
> >>
> >
> > If your data is actually a tsv (tab-separated value format), you should
be
> > using the csv module for actually iterating over lines in it.  Just set
the
> > delimiter to '\t' and look at the docs at
> > http://docs.python.org/library/csv.html
> >
> > You should also generally use the "with" syntax when dealing with files
> > since it handles closing the file object for you (probably not an issue
when
> > you're just reading from a single file, but best practices nonetheless).
> >  Here's how I would deal with your situation:
> >
> > import csv
> >
> > with open('gene-GS00471-DNA_B01_1101_37-ASM.tsv', 'r') as f:
> >     r = csv.reader(f, delimiter='\t')
> >     for row in r:
> >         # row is a list of strings that correspond to the columns in
your
> > file
> >         do_stuff_with_the_row(row)
> > # your file object f is now closed
> >
> > Best wishes,
> > Lucas Wiman
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
http://mail.python.org/pipermail/baypiggies/attachments/20110517/08dd41aa/attachment-0001.html
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 17 May 2011 12:43:49 -0700
> From: Abhishek Pratap <abhishek.vit at gmail.com>
> To: Baypiggies <baypiggies at python.org>
> Subject: [Baypiggies] Fwd:  Interested in a PyPy talk?
> Message-ID: <BANLkTinoD0hWa4iziXc5_6WJwSOKhRmbqw at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Forgot to copy the list.
>
>
> +1. If it can be recorded it will be awesome.
>
> -Abhi
>
> On Tue, May 17, 2011 at 11:47 AM, jim <jim at systemateka.com> wrote:
> >
> > ? ?Please respond if you're interested in having
> > Dan Roberts talk on PyPy at a BayPIGgies meeting.
> > No reponses, no PyPy talk.
> > With thanks,
> > jim
> >
> >
> > _______________________________________________
> > Baypiggies mailing list
> > Baypiggies at python.org
> > To change your subscription options or unsubscribe:
> > http://mail.python.org/mailman/listinfo/baypiggies
> >
>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 17 May 2011 12:55:14 -0700
> From: Tony Cappellini <tony at tcapp.com>
> To: jim <jim at systemateka.com>
> Cc: Baypiggies <baypiggies at python.org>
> Subject: Re: [Baypiggies] Interested in a PyPy talk?
> Message-ID: <BANLkTinhV1F9Ho9jyP37Xk9AWY-qwmgNQg at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> +1
>
> On Tue, May 17, 2011 at 11:47 AM, jim <jim at systemateka.com> wrote:
>
> >
> >    Please respond if you're interested in having
> > Dan Roberts talk on PyPy at a BayPIGgies meeting.
> > No reponses, no PyPy talk.
> > With thanks,
> > jim
> >
> >
> > _______________________________________________
> > Baypiggies mailing list
> > Baypiggies at python.org
> > To change your subscription options or unsubscribe:
> > http://mail.python.org/mailman/listinfo/baypiggies
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
http://mail.python.org/pipermail/baypiggies/attachments/20110517/90f7e1ad/attachment.html
>
>
> ------------------------------
>
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
>
> End of Baypiggies Digest, Vol 67, Issue 17
> ******************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20110517/647f4559/attachment-0001.html>

More information about the Baypiggies mailing list