[Baypiggies] reading files quickly and efficiently

[Mark Voorhies]

On Wednesday, November 17, 2010 01:18:34 pm Tony Cappellini wrote:
> Don't read the entire file into memory.
> 
> readlines() does that.
> 
> Take a look at Dave Beazely's slides on generators and how he
> processes multi-GB sized  files.
> http://www.dabeaz.com/generators/
> 

For NR, it can also be convenient to convert the FASTA to BLAST
database format (via formatdb or downloading the pre-generated
databases from NCBI) and extract sequences with fastacmd
(formatdb and fastacmd are both included in the NCBI BLAST
package).

--Mark