dictionary of list from a file

[sjdevnull at yahoo.com]

limodou wrote:
> On 4 Oct 2006 13:11:15 -0700, sjdevnull at yahoo.com <sjdevnull at yahoo.com> wrote:
> > limodou wrote:
> > > here is my program
> > >
> > > d = {}
> > > for line in file('test.txt'):
> > >     line = line.strip()
> > >     if line:
> > >         k, v = line.strip().split()
> > >         d.setdefault(k, []).append(v)
> > > print d
> >
> > Minor nits: you call strip twice, when you don't need to.  just omit
> > the second call.
>
> Yes, I forgot that.
>
> > Also, I'm not sure stripping the line is the right thing per the spec;
> >
> > d = {}
> > for line in [l[:-1] for l in file('test.txt', 'rU') if len(l)>1]:
> >     k,v = line.split()
> >     d.setdefault(k,[]).append(v)
> >
> [l[:-1] for l in file('test.txt', 'rU') if len(l)>1]
>
> this line I think will create a list, and if the file is large, it'll
> consume many memory I think.
>
> And I think using strip is more clear than list comprehension, except
> for the memory consume.

Good point on memory.

You can use strip in a list (or generator) comprehension just fine.

For example (using a generator to avoid memory consumption):

for line in (l.rstrip("\n") for l in file("test.txt", "rU") if l[0] !=
"\n"):
   k, v = line.split()
   d.setdefault(k, []).append(v)

It's a matter of taste, but I think this isolates the bean-counting
"get rid of newlines/blank lines" stuff from the real processing and
makes it a bit clearer what real work you're doing with each line.