Error in processing JSON files in Python

[Denis McMahon]

On Mon, 30 Mar 2015 14:27:14 -0700, Karthik Sharma wrote:

> I have the following python program to read a set of JSON files do some
> processing on it and dump them back to the same folder. However When I
> run the below program and then try to see the output of the JSON file
> using
> 
> `cat file.json | python -m json.tool`
> 
> I get the following error
> 
> `extra data: line 1 column 307 - line 1 column 852 (char 306 - 851)`
> 
> What is wrong with my program?
>  
>     #Process 'new' events to extract more info from 'Messages'
>     rootDir = '/home/s_parts'
>     for dirName, subdirList, fileList in os.walk(rootDir):
>         print('Found directory: %s' % dirName)
>         for fname in fileList:
>             fname='s_parts/'+fname with open(fname, 'r+') as f:
>                 json_data = json.load(f)
>                 # do stuff to the data
>                 json.dump(json_data,f)

You're writing back to the same file as you loaded the data from having 
opened the file in append mode.

This probably leads to a file containing two json objects, the original 
one and the one which you have processed.

That's a bad json file.

Note the caution in the python documentation:

"Note - Unlike pickle and marshal, JSON is not a framed protocol, so 
trying to serialize multiple objects with repeated calls to dump() using 
the same fp will result in an invalid JSON file."

Writing back to the same file as you read with json.load is the same 
thing. If you want to use the same file, you need to close it after the 
json.load(), and then open it again in write mode before the json.dump(). 

Write is "w", *NOT* "w+".

-- 
Denis McMahon, denismfmcmahon at gmail.com