
                    Building Neural Networks with BrainMaker

    Neural networks perform a new and very different type of computing.  To
    solve your problems with neural networks, you need to think about data
    and problems in a new way.  This publication is not martketing hype --
    it provides technical answers to common questions for people who are new
    to this field.  After reading this you will understand what neural
    networks do and how you can make them work for you.

    The information here provides you with a simple explanation of what
    neural networks are, gives you an overview of the BrainMaker software,
    and takes you through the design process step by step.

    I. What Are Neural Networks?

    Artificial neural networks are a new kind of software modeled after the
    human brain.  The brain is itself a biological neural network made up of
    hundreds of billions of neurons: cells that operate in a limited way
    like computers.  These neurons send information back and forth to each
    other through connections, and the resulting architecture is capable of
    learning, analysis, prediction and recognition.

    Artificial neural networks utilize this same architecture to mimc many
    of the brain's most useful abilities, including pattern recognition,
    trend analysis, and the general ability to learn simply by observing
    data.  When you train a neural network to solve a problem, these
    abilities are automatically applied to the problem.

    At a fundamental level, all neural networks learn associations - that
    is, when the network sees an input A or something like A, it responds
    with output B, or something much like it.  A very simple example of an
    association is learning that an orange, round, large vegetable is a
    pumpkin.  The input A is "orange", "round" and "large" and the output B
    is "pumpkin".

    The neurons in BrainMaker's neural networks are organized into layers -
    neurons in one layer listen to the neurons in the previous layer, and
    send results to neurons in the next layer.

    Your input data is presented to the input neurons which send the
    information to the hidden layer.  Each neuron in the hidden layer
    connects to every input neuron.  The hidden neurons send their results
    to the output neurons.  Each output neuron connects to every hidden
    neuron.  When you see the output of a network, you are looking at
    information from the output layer.

    There are many kinds of neural networks which learn by different
    methods.  BrainMaker is a Back-Propagation type and learns in much the
    same way that people do, that is by example and repetition.  It is not
    programmed with rules, like artificial intelligence systems.  It is
    trained by presenting input-output pairs over and over again to the
    network.  Each time an input is presented to the network during
    training, it guesses what the output might be.  The output portion of
    the input-output pair ("pumpkin" in the above example) is used to
    determine if the network was right or wrong.  When wrong, the network is
    corrected by changes made to its internal connections.  This training
    process (which is what BrainMaker does for you) is repeated until the
    network gets all of the training information correct.  Once the network
    is trained, you can run it on new input information and have it
    recognize, generalize, or predict the new answer for you.

    Neural networks are extremely tolerant of noisy, partially missing, or
    incorrect data.  They are able to filter out the garbage in order to
    provide clean signals or patterns.  They can learn even when a some of
    the data is wrong.  They are able to figure out the hidden relationships
    in your data without you having to explain, define, write equations or
    in any other way "program" it.  All you have to do is define the problem
    in your own mind so that you know which data to present as inputs and
    outputs, and it figures out the rest.

    How much of the neural network's claim to fame is hype?  Certainly
    there are limits to what neural networks can do.  They are not going to
    replace conventional computers and programs.  They are an additional
    tool to be used side by side with present technology.  Using a neural
    network is not always the best solution for certain types of problems.
    They are poor at precise calculations and sequential processing.  They
    are unable to predict or recognize anything that does not inherently
    contain some sort of pattern, e.g. they cannot predict the lottery
    since this is a random process.


    II. How Do I Design a Neural Network?


    Here's a brief description of the design process you will go through to
    make a neural network.  The BrainMaker software comes with network
    design examples and a tutorial which takes you through the program step
    by step.

    1.  Decide what you want your neural network to predict, generalize or
    recognize.  Examples include the prediction of next month's DOW Jones
    Average, analysis of a loan application, and recognition of a submarine
    from sonar.  You must have a clear idea of what you want your network to
    tell you.  You cannot just throw all your spreadsheet data at the
    network, not knowing what you want it tell you.

    2.  Decide what information you want your neural network to use for its
    predictions, generalizations or recognitions.  Predictions are always
    based on something - past history, observable trends, related
    information, psychic or inspired knowledge, for example.  (Actually,
    BrainMaker is not so hot at these last two.) Generalizations are based
    upon a conglomerate of descriptive items and often you're not sure which
    ones are related and which ones aren't.  It's okay, just include
    everything you suspect is important - for example, how how many years
    the loan applicant has been unemployed as an adult.  The neural network
    will learn to pay attention to the important items and ingore the ones
    that don't matter.

    Here are some more examples:

    The determination of the health of an individual under anesthesia could
    be based upon temperature, respiratory rate, amount of various gases
    in exhalation, etc.

    GM stock price predictions could be based on the stock's previous
    prices, the price of steel, the price of Ford and Chrysler stock, the
    consumer debt level, interest rates, etc.

    The outcome of a chemical experiment might based upon temperature,
    pressure, quantities of chemicals, combining processes, etc.

    The prediction of next week's required inventory could be based on sales
    for last week, sales for this week last year, number of items at sale
    prices, etc.

    The predicted winner of the Rams / Raiders game could be based on the
    performance in yards gained and allowed by each team in previous games,
    the point spread made by each team in previous games, and a subjective
    team health indicator (the Rams look pretty good lately for +3, the
    Raiders are having coach problems and have an injured quarterback for
    -7).

    3.  Get some data.  Get lots of examples of many possible combinations
    of data.  You must have samples with known results to train your neural
    network with.

    4.  Build a network.  There are two things you need to make a neural
    network: a definition of the network and a collection of data.  The
    definition includes things like how many inputs and outputs there are,
    how many layers of hidden neurons there are, etc.

    5.  Train your network.  Without a program such as BrainMaker, you would
    have to keep track of all the neurons and connections and perform a lot
    of math every time some input is presented just to get an output.  Also,
    every time the output was wrong you would have to make changes to the
    connections with more math.  You would need some way to interpret the
    data for the neurons and a way to display and save the results.

    6.  Test your network.  In order to be sure that you have a good
    network, you must present it with data it has never seen before and
    check the results.  If the results are good, you're ready to use your
    network.  If not, you'll have to get more data, or redesign your
    network.  Often the problem is that not enough samples were used during
    training.

    7.  Run your network to make more money, impress your friends, and
    please your spouse.  Running a network consists of presenting it with
    new input data and observing the result.

    Steps number 2 and 3 are often the most difficult or time-consuming for
    a new neural network designer.  Some people seem to struggle with how to
    define and present their problem to a neural network since this is a new
    way of computing.  An average amount of time for a first design with
    BrainMaker is a few days, including the time it takes to learn the
    program.  Once you are experienced it is not unreasonable to expect a
    design to be completed and trained in an afternoon, for a network of
    average complexity.


    III.  How Does the BrainMaker Software Work?

    There are two parts to the BrainMaker Neural Network system.  The first
    half, NetMaker, is an optional front-end to BrainMaker.  If your data is
    numeric, it can be brought into the NetMaker program so that translation
    into neural network types of files will be done automatically for you.
    Netmaker will take your raw data and ask you whether each type of data
    is an input (a basis for learning), an output (a result), or neither
    (not used).  Using an example from above, the respiratory rate is an
    input and the person's health level is an output.  You must know which
    are the inputs and which are the outputs.  NetMaker will create the
    definition and data files BrainMaker needs.

    To use NetMaker, you need to have your data in a Lotus, Excel, or dBase
    file, or in a text file in rows and columns.  The program can also read
    binary files, and you can specify the format.  You can also collect data
    which is symbolic (words or symbols) or picture-type.  Symbolic data can
    be brought into the Professional version of Netmaker for automatic
    translation into neural network files.  If you have picture-type data,
    you will need to create the network files yourself using an ASCII editor
    such as SEE (provided with BrainMaker) or a word processor.

    BrainMaker takes the definition and data files and creates a trained
    network from them.  BrainMaker reads each sample of training data and
    presents the input to the network being trained.  Each sample is
    presented one at a time.  The network guesses what the output is.  This
    output is compared to the correct output which you supplied in your
    sample training data.  If the network output and the training output are
    the same (within a tolerance specified by you) then that sample is
    considered learned.  If there is a difference between the training
    pattern and the network output, BrainMaker makes adjustments to the
    network's internal connections in order to decrease this difference.  It
    does this for each sample, over and over until it gets them all correct.
    Usually tens or hundreds of times through the sample training data is
    required to train completely.

    The trained neural network resides in a file on your computer.  If you
    have the Professional version of BrainMaker, you can get answers from
    the trained network through another program, thereby completely hiding
    the neural network from users.

    Before you go placing bets on your trained network, it's a good idea to
    test it on data it's never seen before.  You use another set of data
    which is just like the training data.  BrainMaker presents each testing
    input to the trained network and receives an output guess from the
    network, just like during training.  BrainMaker compares the network
    output to your testing data output to determine if the network was
    right.  BrainMaker provides various statistics during training and
    testing.  The only difference between training and testing is that there
    are no corrections made to the internal connections during testing.  If
    you use NetMaker to make network data files, you can have any percentage
    of your original data saved for testing purposes so that you don't have
    to gather another set of data.

    To run the trained network you provide new input information to
    BrainMaker, and it runs the data through the trained network, and gives
    you the result.  You do not provide outputs since you don't know the
    answer -- that's why you ask the neural network for its opinion.  You
    can incorporate your trained network into some other application, so
    that the neural network is hidden from users.  A trained neural network
    can be downloaded onto a neural network chip and run thousands of times
    faster than the best that can be done on a PC.

    In addition to using network data files for training, testing and
    running, BrainMaker lets you input data on the screen and immediately
    see the network's response.  You can train, test or run in this
    interactive mode.  You can also train, test or run in batch mode.  New
    data can be added to the training set as it becomes available and the
    network can be trained further, starting from where it left off before,
    or retrained from scratch.

                 Do I need BrainMaker Professional to do my job?

    The $195 BrainMaker system has been used by thousands of people to solve
    problems and make money.  BrainMaker is a real system, not an
    exploration program, which can and will solve real problems.  We chose
    the limits from BrainMaker carefully - 512 neurons per layer is a good
    match for the compute, memory, and disk resources of PC's.  However,
    some people need larger networks, and are willing to pay the price in
    terms of required EMS memory and training time.  BrainMaker Professional
    includes all of the neural network programs and tools we've ever
    written, all integrated into one system.  Most people can solve their
    problem with BrainMaker, but it will be easier to build and use your
    neural network if you use Professional.  If you plan to distribute your
    trained neural network then you need Professional.


    IV. Example Application

    Common uses for neural networks include stock and commodity price
    forecasting, improved business decisionmaking, pattern recognition, and
    medical diagnostic systems.  They have been used to classify undersea
    sonar returns, convert text to speech, recognize speech and handwriting,
    control robot arms, model psychological behavior and much more.

    The kinds of problems best solved by neural networks are those people do
    well such as association, evaluation and pattern recognition.  Problems
    that are difficult to compute and do not require perfect answers, just
    quick very good answers, are also handled well by neural networks.  This
    is especially true in real-time robotics or industrial controller
    applications.  Predictions of behavior and analysis of large amounts of
    data are also good applications.  Examples of this are stock market
    forecasting and consumer loan analysis.  New applications under
    development include simple vision systems, weather forecasting, aids in
    medical diagnoses and estimatations of the worth of claims for attorneys
    and insurance companies.


                      Designing a Financial Predictor

    Suppose you want to make a network which will predict the price of the
    Dow Jones Industrial Average (DOW) on a month to month average basis,
    one month in advance.  The information to provide the network might
    include the Consumer Price Index (CPI), the price of crude oil, the
    inflation rate, the prime interest rate, the Gross National Product
    (GNP), and other indicators.

    It's best to give the network lots of information.  If you are unsure if
    there is a relationship, provide the data (for example between how the
    good the weather is over the U.S.  and the DOW).  The neural network
    will figure out if the information is important and will learn to ignore
    anything irrelevant.  Sometimes a possibly irrelevant piece of
    information can allow the network to make distinctions which we are not
    aware of.  If there's no correlation, the network will just ignore the
    information.

    A simple DOW predictor network might look like this:

    Inputs:                                                Output:

                                     Ŀ
    Which month it is Ĵ           
                                                
    Consumer Price Index Ĵ           
        for this month                  The     
    Price of crude oil Ĵ   Neural   Dow Jones average
        for this month                  Network        next month
    Inflation rate Ĵ           
        for this month                          
    DOW Ĵ           
        for this month                          
    Consumer Price Index Ĵ           
        last month                              
    Price of crude oil Ĵ           
        last month                              
    Inflation rate Ĵ           
        last month                              
    DOW Ĵ           
        last month                              
    Consumer Price Index Ĵ           
        3 months ago                            
    Price of crude oil Ĵ           
        3 months ago                            
    Inflation rate Ĵ           
        3 months ago                            
    DOW Ĵ           
        3 months ago                            
    Overall U.S. weather Ĵ           
        for this month                          
    Political Climate Ĵ           
        for this month                          
                                     

    It is helpful to think of a financial predictor as something which
    simultaneously looks at the present state of affairs and what has
    happened in the recent past in order to find a pattern emerging over
    time.  That is why some of the inputs are for the same item, but for
    different periods in time.  The reason this is done is that a neural
    network has no specific memory of that last input you gave it.  You
    present historical data for any "trendy" items such as the consumer
    price index, the inflation rate, and the DOW monthly average.  Items
    such as the political climate and the U.S.  weather are probably not
    items that require a historical view, so there's only one input for
    these which represent their current states.  Don't worry if your data is
    not arranged in this manner.  NetMaker will easily arrange the
    historical data for you.

    If you're unsure what to present as a trend and what not to, you can
    different designs and see what works best.  Half the fun of designing a
    neural network is becoming expert at figuring out what things really
    matter.  We admit that ours is a pretty simple example.  A heftier
    design could have information from more periods in the past (last year,
    e.g.) or shorter periods of time (daily, e.g.) and a greater variety of
    data.

    The data is collected for a substantial period of time, say the last 5
    or 10 years.  You need historical data for each month for each kind of
    data.  Part of your data collection could look like this:

    Mon   CPI   Oil    Dow   Inflat  (etc.)
    -------------------------------
    Jan   229   20.0   2645   5.0
    Feb   235   19.8   2633   4.7
    Mar   244   19.6   2627   4.8
    Apr   261   19.6   2611   4.9
    May   276   19.5   2630   5.0
    Jun   287   19.5   2637   5.1
    Jul   296   19.3   2640   5.1

    Note that these are ficticious values shown for illustration purposes
    only.  All you need is the kind of data collection above.  With
    NetMaker, the historical view is easily added (by selecting one menu
    item).  The DOW output is also easily created.  As shown below, CPI is a
    certain month's consumer price index, CPI-1 is the index one month
    before, CPI-3 is the the index 3 months before, etc.  NetMaker will
    produce this:

Mo  CPI CPI-1 CPI-3 Oil  Oil-1 Oil-3 Dow  Dow-1 Dow-3  etc.   Dow Ave (output)
-----------------------------------------------------         -------
Jan 229 220   211   20.0 21.9  19.5  2645 2652  2580          2633
Feb 235 229   215   19.8 20.0  18.3  2633 2645  2585          2627
Mar 244 235   220   19.6 19.8  21.9  2627 2633  2652          2611
Apr 261 244   229   19.6 19.6  20.0  2611 2627  2645          2630
May 276 261   235   19.5 19.6  19.8  2630 2611  2633          2637
Jun 287 276   244   19.5 19.5  19.6  2637 2630  2627          2640
Jul 296 287   261   19.3 19.5  19.5  2640 2637  2611          2644

    There are lots of other things you can do with NetMaker.  For example,
    you can add information based upon moving averages.  Creating moving
    averages helps build networks that depend on current numbers and past
    numbers, but ignore extremely short small changes.

    After you have your data ready NetMaker will create the definition and
    training files for you.  Then BrainMaker will train it and test it.
    After the network is trained to your satisfaction, you can give the
    network current information and get a prediction of next month's Dow
    Jones average.
   