Possibilities of Building "Stacked Neural Networks"

[William Ray Wing]

On Sep 25, 2013, at 5:43 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On 2013-09-24 19:03, Michael Lamport Commons wrote:
>> Dear Members of this list-serve:
>> 
>>        Would it be possible to build “stacked neural networks” like the one shown in the attached document?
>> 
>>        You may have a few questions about the stacked neural network. For example, what is a stacked neural network? What is the difference between stacked neural networks and the existing neural network? A brief description is provided in the attached document.
>> 
>>        Based on this brief description, I would like to know how would one go about building such stacked neural networks cheaply and easily?  Is there any software available that can do this?  How much would it cost?
>> 
>>        Please feel free to contact me if you think that it would be possible or easier to apply stacked neural network into a more practical field? Suggestions are welcome as well.
> 
> The term of art for these kind of architectures is "deep learning" (and associated terms like "deep architecture", "deep networks", etc.). It's an active field of research that is showing promising preliminary results, and we are beginning to see its limits as well. Google and other big machine learning players are putting a lot of resources into building these systems.
> 
>  http://arxiv.org/pdf/1112.6209v3.pdf
> 
> A good resource would be the Deep Learning Tutorial which shows you how to build these systems using Theano, a Python package for computing with GPUs, one that is particularly well-suited to building deep neural networks.
> 
>  http://deeplearning.net/tutorial/
> 
> Unfortunately, there is nothing cheap or easy about deep networks. They are *very* computationally expensive. You will probably need a small cluster of GPUs to solve interesting problems, and training one will probably take a couple of days of computation (for the final run, *after* you have debugged your code and done the initial experiments to find all of the right hyperparameters for your problem).
> 
> Good luck!
> 
> -- 
> Robert Kern


The OP might also want to look at Nvidia's CUDO units (which package GPUs into massive parallel accelerators - currently well over 2500 GPUs in a single fat card) and PyCUDA which makes the CUDA software available to python.

-Bill