Question

I’ve got an application where I’m loading very large python objects– they’re serialized scikit-learn classifiers and their associated vocabularies.

The classifiers are large enough (on the order of 1-100 MBs) that loading them into memory is a non-trivial task. The actual read is quick, but unpickling takes a long time, around 10 seconds for a 4MB classifier.

Is there a faster way to serialize/deserialize objects than cPickle.dumps/cPickle.loads?

Additional Info:

The classifiers are instances of one-vs-rest random forests of 10 elements. The classifiers were trained on around 1,000 samples, around 500 features, and 52 possible labels. The min_density parameter is set to 0.

---

cProfile output of cPickle.load:

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000  300.168  300.168 <ipython-input-4-9b8a128f290d>:1(loader)
        1    0.899    0.899  301.067  301.067 <string>:1(<module>)
    51380  288.151    0.006  288.151    0.006 __init__.py:93(__RandomState_ctor)
    51380    0.059    0.000    0.404    0.000 fromnumeric.py:1774(amax)
        1   11.613   11.613  300.168  300.168 {cPickle.load}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
    51380    0.344    0.000    0.344    0.000 {method 'max' of 'numpy.ndarray' objects}
        1    0.000    0.000    0.000    0.000 {open}

I’m in the process of opening an issue at github.com/scikit-learn about this.

Answer 1

Have you tried to use the joblib picker? It’s bundled in the sklearn package:

>>> from sklearn.externals import joblib
>>> joblib.dump(model, '/path/to/model.pkl')
>>> model_copy = joblib.load('/path/to/model.pkl')

Edit: actually for random forest, default pickling with HIGHEST protocolo seems to be faster:

>>> from cPickle import dump, load, HIGHEST_PROTOCOL
>>> dump(model, open('/tmp/model_highest.pkl', 'wb'), HIGHEST_PROTOCOL)
>>> load(open('/tmp/model_highest.pkl', 'rb'))

Edit2: based on your profile report, the issue seems to be the unpickling of pseudo random number generator instances. Could you please provide the exact python snippet your are using to train the models along with the shape of the dataset and include that along with the profiling report as a bug on the github issue tracker of the scikit-learn project?