Possibilities of Building "Stacked Neural Networks"

[Robert Kern]

On 2013-09-24 19:03, Michael Lamport Commons wrote:
> Dear Members of this list-serve:
>
>         Would it be possible to build “stacked neural networks” like the one shown in the attached document?
>
>         You may have a few questions about the stacked neural network. For example, what is a stacked neural network? What is the difference between stacked neural networks and the existing neural network? A brief description is provided in the attached document.
>
>         Based on this brief description, I would like to know how would one go about building such stacked neural networks cheaply and easily?  Is there any software available that can do this?  How much would it cost?
>
>         Please feel free to contact me if you think that it would be possible or easier to apply stacked neural network into a more practical field? Suggestions are welcome as well.

The term of art for these kind of architectures is "deep learning" (and 
associated terms like "deep architecture", "deep networks", etc.). It's an 
active field of research that is showing promising preliminary results, and we 
are beginning to see its limits as well. Google and other big machine learning 
players are putting a lot of resources into building these systems.

   http://arxiv.org/pdf/1112.6209v3.pdf

A good resource would be the Deep Learning Tutorial which shows you how to build 
these systems using Theano, a Python package for computing with GPUs, one that 
is particularly well-suited to building deep neural networks.

   http://deeplearning.net/tutorial/

Unfortunately, there is nothing cheap or easy about deep networks. They are 
*very* computationally expensive. You will probably need a small cluster of GPUs 
to solve interesting problems, and training one will probably take a couple of 
days of computation (for the final run, *after* you have debugged your code and 
done the initial experiments to find all of the right hyperparameters for your 
problem).

Good luck!

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco