Artificial Neural Networks and Connectionnist Computing
is offered as CS267B,a regular CS graduate course at UCLA
The course also serves as a proving ground for an evolving
web-grounded remote course with the title
Exploring Artificial Neural Networks with Mathematica
The material is presented in the form of a collection of Mathematica "Notebooks" that cover most salient procedures and provide incentives for open exploration.
Textbook References
"Digital Neural Networks" by S.Y. Kung, Prentice Hall, 1993, will be the official text book.
The titles listed below have different strengths and weaknesses
and will focus differently
on the three complementary faces of Artificial Neural Networks, namely
biological
metaphors for biological neural nets, graph descriptions for a special
class
of paralllel computing algorithms or blueprints for parallel computer
architectures.
You may want to examine them and find some closer to your
particular interests.
"Complex Systems Dynamics" by Gerard Weisbuch, Addison Wesley, 1990
"Introduction to the Theory of Neural Computation", John Hertz, Anders
Krogh
and Richard G. Palmer, Addison Wesley, 1991
"Fundamentals of Neural Networks" Laurene Fausett,Prentice Hall, 1994.
"Artificial Neural Systems", Jacek M. Zurada, West Publishing Company, 1992
General References:
The Anderson books are collection of classic Neural Network publications,
including historic papers from the sixties and earlier. Arbib handbook
is an
ambitious attempt to provide a comprehensive view of the field and
its more
recent developments.
"Neurocomputing:Foundations of Research", James A. Anderson and Edward
Rosenfeld Eds., MIT Press, 1988.
"Neurocomputing 2:Directions for Research" James A. Anderson, Andras
Pellionisz and Edward Rosenfeld Eds., MITPress, 1990.
"The Handbook of Neural Theory and Neural Networks" Michael Arbib, Ed.,
MITPress, 1995.
Mathematica References:
"Mathematica", Steve Wolfram, Academic Press, 19XX (2d. edition or
later).(This is the language manual.)
"Mathematica for Scientists and Engineers", Thomas B. Bahder, Addison
Tentative Calendar
Wesley, 1995.
------------------------------------------------------------------------
Week 1
Introduction. Discussion of Class Format.
Overview of course. Where is the "neural" come from? Relation and overlap
with other fields. Three rationales for Artificial Neural Networks.
Generic
(AI) versus specific (Engineering) problem solving. Taxonomy of
Neuromimetics Architectures. Feedforward (combinatorial) versus Feedback
(dynamic) networks. Doing versus Learning Relation to Function
Approximation,Pattern Recognition, Nearest Neighbor Classification.
Statistics, Estimation, Search, Fuzzyness.
Overview of Mathematica. Parallel Primitives.
Week 2
Single neuron multivariate function generation (analog), and/or pattern
classification (Boolean). Classical Pattern Classifiers. The single
continuous (analog) neuron. Linear neurons. Batch weight computations:
Pseudo-Inverse The incremental method: Stochastic Approximation and
Widrow-Hopf rule Steepest descent learning. Neurons with non-linear
transfer
functions. Sigmoid output functions. The Delta Rule. Mathematica exploration
and visualization of continuous unit proerties. From minimum mean squared
error function approximation to input classification.
Notebook: ContinuousUnits.ma
Week 3
Discrete neurons (with analog weights). McCullogh-Pitts Neurons. Threshold
logic. Pattern Classification continued. Linear Separability Perceptron
Learning. Perceptron theorem.
Notebook: DiscreteUnits.ma
Week 4
Feedforward Networks of Continuous Neurons. One-shot Pattern Association.
Multi-dimensional Vector Mapping. Pattern Classification. Multiple
linear
separations define compact pattern regions Merging sub-nets combine
disconnected regions. The exor-equivalence problem. Steepest descent
learning and the Backpropagation Algorithm Classical Examples of Back
Propagation Error function complexity in multilayered networks
Notebooks: BackPropagation.ma, BackPropIdentity.ma.
Week 5
Associative Networks with Recurrent Feedback. Unidirectional and
Bi-directional (BAM) content-addressable memories. The Hopfield Model.
Discrete Updates. synchronous vs Asynchronous. Hebbian Correlation,
Stability.
Notebooks: Hopfield1.ma, Hopfield2.ma, Hopfield3.ma, Hopfield4.ma,
KeelerAttractors.ma, Raam.ma,
Quiz #1(1hour) on Monday: Covers all materials up to previous Wednesday.
Quiz #1 review on Wednesday.
Week 6 Self-Organization and Competitive Learning. Kohonen Maps Linear
Vector Quantization (LVQ). Classical Examples of Self-Organization.
Notebooks: Kohonen.ma, Conscience.ma
Week 7
Constraint Relaxation
Notebooks: WeightFree.ma, Flexmap.ma
Week 8
Bidirectional Associative Memories (BAM) Art and Counter Propagation
Notebooks: Outstar.ma
Week 9
Time Sequences. Jordan Networks
Notebooks:
Term projects outlines must be accessible by Monday of this week. BR>
Week 10
Hardware Considerations. Networks of Programmable Logic Elements
Quiz 2, (1 hour) on Monday, Review on Wednesday
Week 11
Term projects presentations must be released complete and Web browsable
by Friday of this week. BR>
------------------------------------------------------------------------
Term Projects
A typical term project is a mathematica notebook that presents a particular
neural network application and illustrates it with a numerical example.
It
should be self-documented and didactic.
------------------------------------------------------------------------
Why Mathematica The mathematica language has been chosen for the computer
assisted class presentations because of its compacity and conceptual
clarity. Mathematica also features a rich array of parallel primitives,
making the language ideal for expressing the parallel distributed
computations of Neural Networks.
All the mathematica "notebooks" used in class, and some others are
or will
be accessible through this page.
Mathematica is also the preferred language for student projects although
Maple and MathLab are acceptable alternatives. Programming and comment
style
should emulate the ideas expressed in Concept-Level Programming.
The idea behind Concept Level Programming is to embed programs within
readable text documentation and to make the code itself such that,
as far as
feasible, reads as logical english propositions. Images and also sound
should be used generously when appropriate.
Pursuing that goal is facilitated by the expressive power and multimedia
capabilities of the "Formula-Understanding" computer languages exemplified
by the Mathematica and Maple . These languages accept familiar mathematical
notations and can even be approached without specialized programmer
training. An invaluable feature of Mathematica is embodied in the live
"Notebooks" document format . The latter define a new class of publishable
and portable multi-media documents that combining active code with
unrestricted text processed comments, graphics output and illustrations,
along with a capability for animating pictures and generating sound.
------------------------------------------------------------------------