Decoding a huge JSON file incrementally

[Chris Angelico]

On Fri, Dec 21, 2018 at 4:30 AM Paul Moore <p.f.moore at gmail.com> wrote:
> On Thu, 20 Dec 2018 at 17:22, Chris Angelico <rosuav at gmail.com> wrote:
> > Proper error handling is left as an exercise for the reader, both in
> > terms of JSON errors and file errors. Also, the code is completely
> > untested. Have fun :)
>
> Yeah, once you have the insight that you can attempt to parse a block
> at a time, the rest is just a "simple matter of programming" :-)

Perfect! I can leave it all in your capable hands then. Wasn't sure
how much I'd need to explain there, and it sounds like I
overestimated. :) Have fun with it!

Something to be aware of: exception raising in PYthon is somewhat
expensive, so it may be worth tuning your chunk size upwards a bit to
compensate (keep a bit more in memory, fail the parse less
frequently). Would need to measure. Of course, it's entirely possible
that it won't even matter, and your parse time will be utterly
dominated by disk read times.

ChrisA