Subsetting a dataset

[Benjamin Kaplan]

On Sun, Jun 12, 2011 at 9:53 PM, Kumar Mainali <kpmainali at gmail.com> wrote:
> I have a huge dataset containing millions of rows and several dozen columns
> in a tab delimited text file.  I need to extract a small subset of rows and
> only three columns. One of the three columns has two word string with header
> “Scientific Name”. The other two columns carry numbers for Longitude and
> Latitude, as below.
> Sci Name Longitude Latitude Column4
> Gen sp1 82.5 28.4 …
> Gen sp2 45.9 29.7 …
> Gen sp1 57.9 32.9 …
> … … … …
> Of the many species listed under the column “Sci Name”, I am interested in
> only one species which will have multiple records interspersed in the
> millions of rows, and I will probably have to use filename.readline() to
> read the rows one at a time. How would I search for a particular species in
> the dataset and create a new dataset for the species with only the three
> columns?
> Next, I have to create such datasets for hundreds of species. All these
> species are listed in another text file. There must be a way to define an
> iterative function that looks at one species at a time in the list of
> species and creates separate dataset for each species. The huge dataset
> contains more species than those listed in the list of my interest.
> I very much appreciate any help. I am a beginner in Python. So, complete
> code would be more helpful.
> - Kumar

Read in the file with the lists of species. For each line in that
list, open up a file and then put it into a dictionary where the key
is the species name and the value is the file. Then, once you have all
your files created, open up the second file. For each line in the
second file, split it on the tabs and check to see if the first item
is in the dict. If it is, grab your necessary values and write it to
that corresponding file. In rough, untested code

animals = dict()
for line in open('species_list) :
    #make a file for that animal and associate it with the name
    animals[line.strip()] = open('%s_data.csv' % line.strip(),'w')


#now open the second file
for line in open('animal_data') :
    data = line.split('\t')
    if data[name_col] in animals :
        animals[data[name_col]].write('%s\t%s\t%s' & (data[name_col],
data[lat_col], data[lon_col])

replacing the respective file names and column numbers as appropriate,
of course.

>
>