[scikit-learn] Need Urgent help please in resolving JobLibMemoryError
Piotr Bialecki
piotr.bialecki at hotmail.de
Fri Dec 9 04:18:05 EST 2016
Hi Debu,
it seems that you run out of memory.
Try using fewer processes.
I don't think that n_jobs = 1000 will perform as you wish.
Setting n_jobs to -1 uses the number of cores in your system.
Greets,
Piotr
On 09.12.2016 08:16, Debabrata Ghosh wrote:
Hi All,
Greetings !
I am getting JoblibMemoryError while executing a scikit-learn RandomForestClassifier code. Here is my algorithm in short:
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import train_test_split
import pandas as pd
import numpy as np
clf = RandomForestClassifier(n_estimators=5000, n_jobs=1000)
clf.fit(p_input_features_train,p_input_labels_train)
The dataframe p_input_features contain 134 columns (features) and 5 million rows (observations). The exact error message is given below:
Executing Random Forest Classifier
Traceback (most recent call last):
File "/home/user/rf_fold.py", line 43, in <module>
clf.fit(p_features_train,p_labels_train)
File "/var/opt/ lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 290, in fit
for i, t in enumerate(trees))
File "/var/opt/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 810, in __call__
self.retrieve()
File "/var/opt/lib /python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 757, in retrieve
raise exception
sklearn.externals.joblib.my_exceptions.JoblibMemoryError: JoblibMemoryError
___________________________________________________________________________
Multiprocessing exception:
...........................................................................
/var/opt/lib/python2.7/site-packages/sklearn/ensemble/forest.py in fit(self=RandomForestClassifier(bootstrap=True, class_wei...te=None, verbose=0,
warm_start=False), X=array([[ 0. , 0. , 0. , .... 0. , 0. ]], dtype=float32), y=array([[ 0.],
[ 0.],
[ 0.],
...,
[ 0.],
[ 0.],
[ 0.]]), sample_weight=None)
285 trees = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,
286 backend="threading")(
287 delayed(_parallel_build_trees)(
288 t, self, X, y, sample_weight, i, len(trees),
289 verbose=self.verbose, class_weight=self.class_weight)
--> 290 for i, t in enumerate(trees))
i = 4999
291
292 # Collect newly grown trees
293 self.estimators_.extend(trees)
294
...........................................................................
Please can you help me to identify a possible resolution to this.
Thanks,
Debu
_______________________________________________
scikit-learn mailing list
scikit-learn at python.org<mailto:scikit-learn at python.org>
https://mail.python.org/mailman/listinfo/scikit-learn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20161209/87ade20d/attachment-0001.html>
More information about the scikit-learn
mailing list