Merge pull request #58 from aigamedev/pretrain

Support to transfer weights from autoencoder to mlp

Author: alexjc

README.rst

@@ -37,8 +37,8 @@ If you want to use the latest official release, you can do so from PYPI directly
 
 This contains its own packaged version of ``pylearn2`` from the date of the release (and tag) but will use any globally installed version if available.
 
-Pulling From Repository
-~~~~~~~~~~~~~~~~~~~~~~~
+Pulling Repositories
+~~~~~~~~~~~~~~~~~~~~
 
 You'll need to first install some dependencies manually.  Unfortunately, ``pylearn2`` isn't yet installable via PyPI and recommends an editable (``pip -e``) installation::
 

docs/guide_advanced.rst

@@ -1,24 +1,7 @@
 Advanced Usage
 ==============
 
-The examples in this section help you get more out of ``scikit-neuralnetwork`` via integration or configuration of the other libraries it works with, such as Theano or scikit-learn.
-
-
-GPU Backend
------------
-
-To setup the library to use your GPU or CPU explicitly in 32-bit or 64-bit mode, you can use the ``backend`` pseudo-module.  It's a syntactic helper to setup ``THEANO_FLAGS`` in a Pythonic way, for example:
-
-.. code:: python
-
-    # Use the GPU in 32-bit mode, falling back otherwise.
-    from sknn.backend import gpu32
-    
-    # Use the CPU in 64-bit mode.
-    from sknn.backend import cpu64
-    
-
-WARNING: This will only work if your program has not yet imported the ``theano`` module, due to the way the library is designed.  If ``THEANO_FLAGS`` are set on the command-line, they are not overwridden.
+The examples in this section help you get more out of ``scikit-neuralnetwork``, in particular via its integration with ``scikit-learn``.
 
 
 .. _example-pipeline:
@@ -45,6 +28,41 @@ Here's how to setup such a pipeline with a multi-layer perceptron as a classifie
 You can then use the pipeline as you would the neural network, or any other standard API from scikit-learn.
 
 
+Unsupervised Pre-Training
+-------------------------
+
+If you have large quantities of unlabeled data, you may benefit from pre-training using an auto-encoder style architecture in an unsupervised learning fashion.
+
+.. code:: python
+
+    from sknn import ae, mlp
+
+    # Initialize auto-encoder for unsupervised learning.
+    myae = ae.AutoEncoder(
+                layers=[
+                    ae.Layer("Tanh", units=128),
+                    ae.Layer("Sigmoid", units=64)],
+                learning_rate=0.002,
+                n_iter=10)
+    
+    # Layerwise pre-training using only the input data.
+    myae.fit(X)
+    
+    # Initialize the multi-layer perceptron with same base layers.
+    mymlp = mlp.Regressor(
+                layers=[
+                    mlp.Layer("Tanh", units=128),
+                    mlp.Layer("Sigmoid", units=64),
+                    mlp.Layer("Linear")])
+    
+    # Transfer the weights from the auto-encoder.
+    myae.transfer(mymlp)
+    # Now perform supervised-learning as usual.
+    mymlp.fit(X, y)
+
+The downside of this approach is that auto-encoders only support activation fuctions ``Tanh`` and ``Sigmoid`` (currently), which excludes the benefits of more modern activation functions like ``Rectifier``.
+
+
 Grid Search
 -----------
 

docs/guide_beginners.rst

@@ -59,38 +59,7 @@ It's also a good idea to normalize or standardize your data in this case too, fo
 This code will run the classification with the neural network, and return a list of labels predicted for each of the example inputs.
 
 
-Verbose Mode
-------------
-
-To see the output of the neural network's training, you need to configure two things: first setting up the Python logger (mandatory), and secondly to specify a verbose mode if you want more information during training (optional).
-
-The first step is to configure either the ``sknn`` logger specifically, or do so globally (easier) as follows:
-
-.. code:: python
-
-    import sys
-    import logging
-
-    logging.basicConfig(
-                format="%(message)s",
-                level=logging.DEBUG,
-                stream=sys.stdout)
-
-Then you can optionally create your neural networks using an additional ``verbose`` parameter to show the output during training:
-
-.. code:: python
-    
-    from sknn.mlp import Regressor, Layer
-
-    nn = Regressor(
-        layers=[Layer("Linear")],
-        n_iter=20,
-        verbose=1,
-        valid_size=0.25)
-    nn.fit(X, y)
-
-This code will output a table containing validation scores at each of the twenty epochs.  The ``valid_size`` parameter is a ratio of the data to be used internally for validation; in short, the ``fit()`` function is automatically splitting the data into ``X_train`` and ``y_train`` as well as ``X_valid`` and ``y_valid``.
-
+.. _example-convolution:
 
 Convolution
 -----------

docs/guide_start.rst renamed to docs/guide_installation.rst

@@ -1,10 +1,13 @@
 Installation
 ============
 
+You have multiple options to get up and running, though using ``pip`` is by far the easiest and most reliable.
+
+
 Downloading Package
 -------------------
 
-To download and setup the last officially released package, you can do so from PYPI directly::
+**Recommended.** To download and setup the last officially released package, you can do so from PYPI directly::
 
     > pip install scikit-neuralnetwork
 
@@ -13,6 +16,23 @@ This contains its own packaged version of ``pylearn2`` from the date of the rele
 If you want to install the very latest from source, please visit the `Project Page <http://github.com/aigamedev/scikit-neuralnetwork>`_ on GitHub for details.
 
 
+Pulling Repositories
+--------------------
+
+**Optional.** To setup a developer version of the project, you'll need to first install some dependencies manually.  Unfortunately, ``pylearn2`` isn't yet installable via PyPI and recommends an editable (``pip -e``) installation::
+
+    > pip install numpy scipy theano
+    > pip install -e git+https://github.com/lisa-lab/pylearn2.git#egg=Package
+
+Once that's done, you can grab this repository and install from ``setup.py`` in the exact same way::
+
+    > git clone https://github.com/aigamedev/scikit-neuralnetwork.git
+    > cd scikit-neuralnetwork
+    > python setup.py develop
+
+This will make the ``sknn`` package globally available within Python as a reference to the current directory.
+
+
 Running Tests
 -------------
 
@@ -26,4 +46,3 @@ Use the additional command-line parameters in the test runner ``--processes=8``
 .. image:: console_tests.png
 
 We strive to maintain 100% test coverage for all code-paths, to ensure that rapid changes in the underlying ``pylearn2`` library are caught automatically.
-

docs/guide_intermediate.rst

@@ -0,0 +1,70 @@
+Misc. Additions
+===============
+
+Verbose Mode
+------------
+
+To see the output of the neural network's training, you need to configure two things: first setting up the Python logger (mandatory), and secondly to specify a verbose mode if you want more information during training (optional).
+
+The first step is to configure either the ``sknn`` logger specifically, or do so globally (easier) as follows:
+
+.. code:: python
+
+    import sys
+    import logging
+
+    logging.basicConfig(
+                format="%(message)s",
+                level=logging.DEBUG,
+                stream=sys.stdout)
+
+Then you can optionally create your neural networks using an additional ``verbose`` parameter to show the output during training:
+
+.. code:: python
+    
+    from sknn.mlp import Regressor, Layer
+
+    nn = Regressor(
+        layers=[Layer("Linear")],
+        n_iter=20,
+        verbose=True,
+        valid_size=0.25)
+    nn.fit(X, y)
+
+This code will output a table containing validation scores at each of the twenty epochs.  The ``valid_size`` parameter is a ratio of the data to be used internally for validation; in short, the ``fit()`` function is automatically splitting the data into ``X_train`` and ``y_train`` as well as ``X_valid`` and ``y_valid``.
+
+
+Saving & Loading
+----------------
+
+To save a trained neural network to disk, you can do the following after having initialized your multi-layer perceptron as the variable ``nn`` and trained it:
+
+.. code:: python
+
+    import pickle
+    pickle.dump(nn, open('nn.pkl', 'wb'))
+
+After this, the file ``nn.pkl`` will be available in the current working directory — which you can reload at any time:
+
+    import pickle
+    nn == pickle.load(open('nn.pkl', 'rb'))
+
+In this case, you can use the reloaded multi-layer perceptron as if it had just been trained.  This will also work on different machines, whether CPU or GPU.
+
+NOTE: You can serialize complex pipelines (for example from this section :ref:`example-pipeline`) using this exact same approach.
+
+
+GPU Backend
+-----------
+
+To setup the library to use your GPU or CPU explicitly in 32-bit or 64-bit mode, you can use the ``backend`` pseudo-module.  It's a syntactic helper to setup ``THEANO_FLAGS`` in a Pythonic way, for example:
+
+.. code:: python
+
+    # Use the GPU in 32-bit mode, falling back otherwise.
+    from sknn.backend import gpu32
+    
+    # Use the CPU in 64-bit mode.
+    from sknn.backend import cpu64
+
+WARNING: This will only work if your program has not yet imported the ``theano`` module, due to the way the library is designed.  If ``THEANO_FLAGS`` are set on the command-line, they are not overwridden.

docs/index.rst

@@ -26,8 +26,9 @@ User Guide
 .. toctree::
     :maxdepth: 2
 
-    guide_start
+    guide_installation
     guide_beginners
+    guide_intermediate
     guide_advanced
 
 

examples/bench_cifar10.py

@@ -1,27 +1,34 @@
 # -*- coding: utf-8 -*-
 from __future__ import (absolute_import, unicode_literals, print_function)
 
+import sys
 import pickle
+import logging
 import numpy as np
 
+logging.basicConfig(format="%(message)s", level=logging.DEBUG, stream=sys.stdout)
+
+PRETRAIN = False
+
+
 def load(name):
     print("\t"+name)
     try:
         with open(name, 'rb') as f:
-            return pickle.load(f, encoding='latin1')
+            return pickle.load(f) # , encoding='latin1')
     except IOError:
         import gzip
         with gzip.open(name+'.gz', 'rb') as f:
-            return pickle.load(f, encoding='latin1')
+            return pickle.load(f) # , encoding='latin1')
 
 print("Loading...")
 dataset1 = load('data_batch_1')
 dataset2 = load('data_batch_2')
 dataset3 = load('data_batch_3')
 print("")
 
-data_train = np.vstack([dataset1['data'], dataset2['data']])
-labels_train = np.hstack([dataset1['labels'], dataset2['labels']])
+data_train = np.vstack([dataset1['data']]) #, dataset2['data']])
+labels_train = np.hstack([dataset1['labels']]) #, dataset2['labels']])
 
 data_train = data_train.astype('float') / 255.
 labels_train = labels_train
@@ -36,17 +43,30 @@ def load(name):
 logging.basicConfig(format="%(message)s", level=logging.DEBUG, stream=sys.stdout)
 
 from sknn import mlp
-net = mlp.Classifier(
-    layers=[
-        mlp.Layer("Rectifier", units=n_feat*2/3),
-        mlp.Layer("Rectifier", units=n_feat*1/3),
-        mlp.Layer("Softmax", units=n_targets)],
-    n_iter=50,
-    n_stable=10,
-    learning_rate=0.001,
-    valid_size=0.1,
-    verbose=1)
-net.fit(data_train, labels_train)
+nn = mlp.Classifier(
+        layers=[
+            mlp.Layer("Sigmoid", units=128),
+            mlp.Layer("Sigmoid", units=128),
+            mlp.Layer("Softmax", units=n_targets)],
+        n_iter=4,
+        n_stable=4,
+        learning_rate=0.001,
+        valid_size=0.5,
+        verbose=1)
+
+if PRETRAIN:
+    from sknn import ae
+    ae = ae.AutoEncoder(
+            layers=[
+                ae.Layer("Sigmoid", units=128),
+                ae.Layer("Sigmoid", units=128)],
+            learning_rate=0.002,
+            n_iter=10,
+            verbose=1)
+    ae.fit(data_train)
+    ae.transfer(nn)
+
+nn.fit(data_train, labels_train)
 
 from sklearn.metrics import classification_report
 from sklearn.metrics import confusion_matrix