[Tutor] Program gets stuck after a creating a list from dictinary items!

[Steven D'Aprano]

Ali Torkamani wrote:
> Dear Tutors,
> I'm trying to write a dictionary into a csv file, in the following code
> FlatData is the global dictionary whose keys are datetime objects, and the
> values are list of dictionaries. sCommonFeatures are key's that exist in
> all of such 'sub'-dictionaries, and sOtherFeatures are the key's that are
> in some of sub-dictionaries.
> 
> The problem is that in line : D=[prog[key1] for key1 in sCommonFeatures]
> (line 10 below), the program stucks!
> 
> I appreciate any help.


Start by spending some time preparing some sample code we can actually run. 
The code you supply has inconsistent indentation, which means we have to 
*guess* what the indentation is supposed to be. Even if we can guess, we have 
to guess what your data is, which is impossible.

You should try to prepare a minimal working example that anyone can run and 
that demonstrates the problem. Take care to keep your maximum line length 
under 79 characters, so that mail clients won't word-wrap it. See here for 
more information:

http://sscce.org/

>         for prog in FD:
>             D=[prog[key1] for key1 in sCommonFeatures]
>             print(prog)

So FD is a list of dictionaries? What does FD mean? You should have more 
descriptive names.

And prog is not actually a prog(ram), but a dictionary? You *really* should 
have more descriptive names.

Assuming that all the data you are using are standard, built-in lists and 
dicts, I can't see anything that could be causing this problem. Are you using 
any custom subclasses? If so, perhaps there is a bug in one of your 
__getitem__ methods which enters an infinite loop.

How many values are in sCommonFeatures? If there are tens of millions, perhaps 
you are running out of memory and your operating system is trying to page data 
in and out of virtual memory. That can take hours or even days(!), depending 
on how little physical memory you have and how big the list gets.

The symptom of that is that your entire computer basically becomes unresponsive.

What happens if you cut your input data all the way back to a trivially small 
set of values, say, two keys only in sCommonFeatures, prog only having exactly 
two values, exactly one prog in FD? Does the code still get stuck?

If you add another print statement just before the list comprehension, you can 
see whether or not it actually is the list comp that is getting stuck.

What happens if you take out *all* the pdb calls? Not just in your code 
sample, but everywhere in the entire program. Maybe something in the debugger 
is interfering with the list comp.



-- 
Steven