[issue24313] json fails to serialise numpy.int64

Vicki Brown report at bugs.python.org
Mon Aug 26 16:49:43 EDT 2019


Vicki Brown <vlb at cfcl.com> added the comment:

This is still broken. With pandas being popular, it's more likely someone might hit it. Can we fix this?

At the very least, the error message needs to be made much more specific.

 I have created a dictionary containing pandas stats. 
```
def summary_stats(s):
    """ 
    Calculate summary statistics for a series or list, s 
    returns a dictionary
    """
    
    stats = {
      'count': 0,
      'max': 0,
      'min': 0,
      'mean': 0,
      'median': 0,
      'mode': 0,
      'std': 0,
      'z': (0,0)
    }
    
    stats['count'] = s.count()
    stats['max'] = s.max()
    stats['min'] = s.min()
    stats['mean'] = round(s.mean(),3)
    stats['median'] = s.median()
    stats['mode'] = s.mode()[0]
    stats['std'] = round(s.std(),3)

    std3 = 3* stats['std']
    low_z = round(stats['mean'] - (std3),3)
    high_z = round(stats['mean'] + (std3),3)
    stats['z'] = (low_z, high_z)
        
    return(stats)
```

Apparently, pandas (sometimes) returns numpy ints and numpy floats. 

Here's a piece of the dictionary:
```
 {'count': 597,
   'max': 0.95,
   'min': 0.01,
   'mean': 0.585,
   'median': 0.58,
   'mode': 0.59,
   'std': 0.122,
   'z': (0.219, 0.951)}
````

It looks fine, but when I try to dump the dict to json
```
with open('Data/station_stats.json', 'w') as fp:
    json.dump(station_stats, fp)
```

I get this error
```
TypeError: Object of type int64 is not JSON serializable
```

**Much searching** led me to discover that I apparently have numpy ints which I have confirmed.

```
for key, value in station_stats['657']['Fluorescence'].items():
    print(key, value, type(value))

count 3183 <class 'numpy.int64'>
max 2.8 <class 'float'>
min 0.02 <class 'float'>
mean 0.323 <class 'float'>
median 0.28 <class 'float'>
mode 0.24 <class 'numpy.float64'>
std 0.194 <class 'float'>
z (-0.259, 0.905) <class 'tuple'>
```

```
#### Problem description

pandas statistics sometimes produce numpy numerics.

numpy ints are not supported by json.dump

#### Expected Output

I expect ints, floats, strings, ... to be JSON srializable.


<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 15.6.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.1.1
setuptools       : 41.0.1
Cython           : 0.29.12
pytest           : 5.0.1
hypothesis       : None
sphinx           : 2.1.2
blosc            : None
feather          : None
xlsxwriter       : 1.1.8
lxml.etree       : 4.3.4
html5lib         : 1.0.1
pymysql          : 0.9.3
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.7.0
pandas_datareader: None
bs4              : 4.7.1
bottleneck       : 1.2.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.3.4
matplotlib       : 3.1.0
numexpr          : 2.6.9
odfpy            : None
openpyxl         : 2.6.2
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.0
sqlalchemy       : 1.3.5
tables           : 3.5.2
xarray           : None
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.1.8
</details>

----------
nosy: +vlbrown
type: enhancement -> behavior
versions: +Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue24313>
_______________________________________


More information about the Python-bugs-list mailing list