unittest weirdness

[Ethan Furman]

On 03/12/2014 06:44 AM, Roy Smith wrote:
> In article <mailman.8082.1394620172.18130.python-list at python.org>,
>   Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> I've tried it both ways, and both ways my process is being killed, presumably
>> by the O/S.
>
> What evidence do you have the OS is killing the process?

I put a bare try/except around the call to unittest.main, with a print statement in the except, and nothing ever prints.


> Some systems have an oom (Out Of Memory) process killer, which nukes
> (semi-random) process when the system exhausts memory.  Is it possible
> this is happening?  If so, you should see some log message in one of
> your system logs.

That would explain why my editor windows were being killed.


> You didn't mention (or maybe I misssed it) which OS you're using.

Ubuntu 13 something or other.

> I'm
> assuming you've got some kind of system call tracer (strace, truss,
> dtrace, etc).

Sadly, I have no experience with those programs yet, and until now didn't even know they existed.

> Try running your tests under that.  If something is
> sending your process a kill signal, you'll see it:
>
> [gazillions of lines elided]
> write(1, ">>> ", 4>>> )                     = 4
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> select(1, [0], NULL, NULL, NULL)        = ? ERESTARTNOHAND (To be
> restarted)
> --- SIGTERM (Terminated) @ 0 (0) ---
> +++ killed by SIGTERM +++
>
> Alternatively, maybe something inside your process is just calling
> sys.exit(), or even os._exit().  You'll see the exit() system call in
> the strace output.

My bare try/except would have caught that.


> And, of course, the standard suggestion to reduce this down to the
> minimum test case.  You posted:
>
>       def test_xxx_1(self):
>           p = self.pp.lockbox_payments[0]
>           # affirm we have what we're expecting
>           self.assertEqual(
>                   (p.payer, p.ck_num, p.credit),
>                   ('a customer', '010101', 10000),
>                   )
>           self.assertEqual(p.invoices.keys(), ['XXXXXXXXXXX'])
>           self.assertEqual(p.invoices.values()[0].amount, 10000)
>           # now make sure we get back what we're expecting
>           np, b = self.pp._match_invoices(p)
>           missing = []
>           for inv_num in ('123456', '789012', '345678'):
>               if inv_num not in b:
>                   missing.append(inv_num)
>           if missing:
>               raise ValueError('invoices %r missing from batch' % missing)
>
> what happens if you reduce that to:
>
>       def test_xxx_1(self):
>            self.fail()

I only get the strange behavior if more than two (or maybe three) of my test cases fail.  Less than that magic number, 
and everything works just fine.  It doesn't matter which two or three, either.


> do you still get this strange behavior?  What if you get rid of your
> setUpClass()?  Keep hacking away at the test suite until you get down to
> a single line of code which, if run, exhibits the behavior, and if
> commented out, does not.  At that point, you'll have a clue what's
> causing this.  If you're lucky :-)

I strongly suspect it's memory.  When I originally wrote the code I tried to include six months worth of EoM data, but 
had to back it down to three as my process kept mysteriously dying at four or more months.  There must be waaaaaaay too 
much stuff being kept alive by the stack traces of the failed tests.

Thanks for your help!

--
~Ethan~