By choosing to view this document, you agree to all provisions of the copyright laws protecting it.

Radu Drossu          Zoran Obradovic       Justin Fletcher

rdrossu@eecs.wsu.edu   zoran@eecs.wsu.edu   jfletche@eecs.wsu.edu

The graphical user interface (GUI) for heterogeneous neural network simulators proposed in this article is intended to be of use both for the novice and for the experienced neural network user. For the novice, it provides an easy to use neural network simulation package that insulates the user from the requirements of knowing the simulator implementation details or the configuration file syntax. For the experienced neural network professional it provides an interface that is easily extensible to include any additional neural network simulator in its binary form. To satisfy both academic and personal computer environments, the GUI has been developed by using the free TCL/TK software package, available on workstations running UNIX and on PC's running the free Linux operating system. Although the GUI and the embedded simulators have been successfully tested both in neural network research and training programs, a more extensive testing in undergraduate and graduate level classes is in progress.

I. Introduction

Neural network data modeling techniques are applicable to two general types of problems:

• Classification problems, in which the goal is to classify a number of patterns as belonging to different categories. An example would be to automatically recognize hand written zip codes and classify each symbol in one of ten categories corresponding to the digits from 0 to 9 [8].
• Prediction problems, in which the goal is to predict a future value of a specific process parameter (an example would be the prediction of the U.S. power consumption for the year 1997). The prediction problems can be regarded as limit cases of classification problems in which the number of classes is infinite.

In addition, these techniques are applicable not only to time independent, but also to time dependent problems (time series), where the patterns to be classified are temporal sequences of previous process values.

Neural network computational models consist of a number of relatively simple, interconnected processing units called neurons working in parallel. Different neural network architectures can be used to solve a given classification or prediction problem. In an architecture the neurons are interconnected through synaptic links and are grouped in layers, with synaptic links usually connecting neurons in adjacent layers. Typically three different layer types can be distinguished: input (the layer that external stimuli are applied to), output (the layer that outputs results to the exterior world) and hidden (the intermediate computational layers between input and output layer). The neural network architectures can be, roughly speaking, divided in two categories: feedforward, in which the signal flow between different layers is unidirectional (from input layer towards output layer), or recurrent, in which the flow between different layers is bidirectional (both from input layer towards output layer as well as the reverse).

The most distinctive element in neural networks, as opposed to traditional computational structures, is denoted as learning. Learning represents the adaptation of some characteristic neural network parameters by using a set of examples (known outcomes of the problem for given conditions) so that for given input patterns the outputs reflect the value of a specific function. The final data modeling goal is not to memorize some known patterns, but to be able to generalize from prior examples (to be able to estimate the outcomes of the problem under unknown conditions).

Neural network data modeling represents the combination of an architecture and an appropriate learning technique. Traditional approaches assume learning on a prespecified architecture, whereas constructive learning creates the appropriate architecture during the learning process. Most neural networks have a number of architectural and learning parameters that have to be appropriately specified. Selection of appropriate parameters for large scale applications is an important experimental problem. If parameters are not appropriate, the algorithms may take a long time to converge or may not converge at all [12]. Consequently, in training novice neural network users, an emphasis on techniques and intuition development is necessary for properly setting these parameters.

Currently, most neural network modeling is performed on standard sequential computers (rather than on specialized neural network hardware) using simulation software. Until recently the choice of a simulation software was not a primary issue among neural network professionals, as application of neural network technology has been limited to relatively small problems. However, the choice of an efficient simulation software plays an important role when dealing with large scale, real life applications as desired by industry. The approach proposed in this paper provides developers with a number of simulator modules implementing various learning algorithms, all using a single user interface. In such an environment it is easier to select the model that is appropriate for a particular application than to become accustomed to various independent simulators.

Until recently, most neural network modeling work was performed by highly trained neural network experts. In practice, such experts learn how to use several commercial or freely available packages and in addition develop their own software to deal with special issues not supported by the packages. Experience seems to indicate that to support features not present in the standard packages, most neural network experts prefer building their own software rather than using tools developed by other experts (even inside a single group). This is a result of the significant time required to learn how to use someone else's neural network tool properly. We strongly believe that the software reuse among experts can be significantly improved by using a flexible user interface, as proposed in this paper, that can easily support developers' needs.

As the applicability of artificial neural networks seems to extend far beyond the computer science domain to areas like electrical engineering, biology, medicine and business, the need for rapidly trained personnel able to quickly assimilate neural network techniques and to apply them to the corresponding domain becomes increasingly acute. Thus the necessity arises to acquire some basic neural network knowledge rapidly, without covering underlying mathematical and programming details, but rather concentrating on understanding the conceptual model as well as on designing adequate experiments tuned to a particular problem. Our experience indicates that the choice of a graphical user interface, as proposed in this paper, plays a crucial role when trying to transfer neural network technology rapidly to non-expert users.

The graphical user interface (GUI) proposed in this article is intended to be of use both for the novice and for the experienced user. The software package included with this paper provides interfaces to both the most used traditional backpropagation learning through gradient descent optimization on a prespecified analog architecture [14] and to a novel learning technique developed in our laboratory, known as hyperplane determination from examples (HDE), which automatically grows a problem tailored discrete architecture [4, 5]. The fine-tuning of each neural network learning process is done by adjusting specific learning parameters. For the novice, it provides an easy to use neural network simulation package which supports both backpropagation and HDE, insulating the user from the requirements of knowing the implementation details and the configuration file syntax for running the simulators. For the experienced neural network professional it provides an interface that is easily extensible to include any additional neural network simulator in its binary form.

The paper is organized as follows: Section 2 contains a description of the graphical user interface and its underlying menu structure; Section 3 illustrates the flexibility, portability and extensibility of the GUI by means of embedding two radically different neural network simulators and Section 4 discusses the applications of the GUI in research and teaching as well as suggestions for further improving the GUI.

II. GUI Design and Implementation

The GUI proposed in this paper is developed for UNIX and Linux environments to satisfy both expert and novice users' needs. To ensure portability, the GUI has been written using just the TCL/TK core features without any additional C procedures. The GUI allows an interactive setting of neural network simulator parameters as well as the execution of the simulations. In its current form, the GUI embeds two radically different simulators, a backpropagation simulator and the HDE constructive learning simulator. A basic understanding of TCL/TK allows the user to easily embed additional simulators by "linking" their executable files to the GUI interface.

The GUI provides the following basic features:

1. A text editor for editing plain text files.
2. The color configuration of all the GUI items.
3. Setting the characteristic features and parameters for the underlying neural network simulators.
4. Saving the neural network configuration in the simulator specific file formats, as well as retrieving a neural network configuration from a file for the backpropagation simulator.
5. Viewing the neural network configuration both in a tutorial style form (in which the significance of the different configuration options is briefly explained), as well as in the simulators' configuration file formats.
6. Running the simulators both as foreground and as background processes (for an explanation of foreground and background processes, the reader can consult [10, 13].
7. A help file that provides information on running the simulators both by using the GUI as well as by editing the corresponding configuration files for the simulators.

In order to run the GUI, the contents of either the directory called NNGUI/UNIX (for HP 9000 series workstations), or of NNGUI/LINUX (for PC's running Linux) from the CD-ROM must be copied to a user's directory (such as MYNNGUI). After setting the current directory to MYNNGUI the GUI can then be started by issuing the command {\tt nngui}. The different menus of the GUI can be accessed either by clicking them with the left mouse button or by simultaneously pressing the ALT key and the underlined letter key of the menu name. During application execution, certain GUI options can be accessed directly from the main window through the use of a keyboard accelerator (hot key). The options that have corresponding hot keys have the key combination displayed in the pop-up menus (see the help file for a list of options), where CTRL+A represents pressing the CTRL and A keys simultaneously. The interface is not case sensitive, so lowercase a is the same as uppercase A). In each file selection box, a file is selected by double-clicking it with the left mouse button.

The menu structure of the GUI can be represented hierarchically as in Fig. 1.

A brief explanation of each of the different menu items supported under the current GUI implementation follows. For a more detailed explanation, install the program from the CD-ROM and select the Help option under the Help menu item, or press CTRL+H in the main menu window.

A. The File Menu

• The Edit File menu item provides access to a simple general purpose text editor that allows the GUI user to easily change parameters in a previously created neural network configuration file. A file selection box allows the user to select either an existing or a new file to edit (see Fig. 2). An editor window is then displayed with the editor commands displayed at the bottom of the screen (see Fig. 3).

  

  

• The Load Config menu item allows the loading of a previously saved configuration file. It also allows the more advanced user to load a configuration file that was created directly (without the use of the GUI). In its current implementation, the GUI supports this option only for the backpropagation simulator.
• The Save Config menu item allows the saving of the neural network configuration defined in the GUI to a file for future use.
• The Exit Program menu item exits the application.

B. The Network Menu

• The Backpropagation menu item selects the backpropagation simulator. By selecting this option, the settings specific to the backpropagation simulator will be displayed under the Configuration menu.
• The Constructive HDE menu item selects the HDE constructive learning simulator. This option will also enable the display of constructive learning specific settings under the Configuration menu.

C. The Configuration Menu

D. The Run Menu

• Run from File starts the simulator using a simulator specific configuration file. This is the option that an expert in a particular neural network model would most likely use.
• Run from Desktop starts the simulator using the parameters that have been configured by using the GUI. This is the preferred option for a novice user, who is unfamiliar with a particular neural network simulator.
• Options allows the choice of whether to run the simulator as a foreground or as a background job (in which case the output of the simulation will be redirected to a file). The foreground option is usually preferred in the case of a short simulation run (of the order of minutes), while the background option is used in the case of long simulation runs that may last hours or even days.

E. The View Menu

• View Params displays the settings in a self-explanatory way, thus helping the user to decide whether all the settings are appropriate or not.
• View File displays the settings according to the specific configuration file syntax of the underlying simulators.

F. The Color Menu

The default color configuration can be restored by selecting the Restore Color Configuration menu item, whereas the preferred color configuration can be saved by choosing the Save Color Configuration menu item. Whenever a new color configuration is saved, it will overwrite the previously saved color configuration stored in the file config.crt.

G. The Help Menu

III. The Configuration Menu as a Key to Embedding Heterogeneous Neural Network Simulators

From the perspective of the end user, who should be insulated from implementation details, the only differences that appear between various simulators are reflected in just the Configuration menu. As previously mentioned, this menu depends on the neural network simulator selected in the Network menu. If additional simulators are added to the GUI, these will be reflected in the corresponding Configuration menu options when selected.

A brief description of the configuration options for the two neural network simulators included in this version of the GUI follows. For a detailed description of the parameters that are set in this menu, the reader is advised to install the software from the CD-ROM and to consult the help file (by pressing CTRL+H in the main window). For details of configuring model parameters, the basic concepts for the backpropagation learning technique can be found in [6], while the HDE constructive learning concepts can be found in [4, 5].

The corresponding Configuration menu settings for the backpropagation simulator are:

• Architecture selects either a feedforward or a recurrent neural network, as well as establishes the desired layer structure (number of layers and number of neurons per layer). In the case of recurrent networks it also allows the selection of the feedback type and of the feedback layer, as well as of the scaling parameters for the feedback signals.
• Parameters sets the backpropagation specific learning parameters, known as learning rate, momentum and tolerance.
• Activation allows the setting of the neurons' activation function to either a sigmoidal, a hyperbolic tangent or an identity function. The setting can be done either globally (for all the neurons in the network) or locally, either at layer level or at neuron (unit) level. A screen of the activation function window is presented in Fig. 5.

  

• Update Mode sets one of the two specific modes in which the weights of the synaptic links are updated, either continuously (after each training example) or in batch mode (after each presentation of all examples from the training set).
• Initial Weights sets the initial values of the synaptic links' weights either randomly or with values read from a file.
• Set Weights allows the setting of particular weight values either globally (for all the synaptic links), or locally (for all the links connecting two adjacent layers, for all the links emanating from a given unit, for all the links ending at a given unit, or for a single link).
• Enable Weights allows the enabling or disabling of certain synaptic links in ways similar to setting weight values. A disabled link behaves like a disconnected link.
• Horizon sets the desired testing horizon for time series prediction problems (a testing horizon of one indicates the prediction of just the next process value, whereas a prediction horizon larger than one indicates the prediction of a process value further in future). This menu item also allows the setting of the training horizon, an internal learning parameter that cannot exceed the testing horizon.
• Threshold allows a brute force elimination of training examples with outputs above or below a certain level ({\it None} represents no elimination of training examples).
• Training selects the training file (the file containing all the examples used during the learning process), as well as the number of training epochs (the number of repetitive passes through the example file) and the number of epochs after which to display the learning status.
• Testing selects the testing file (the file containing examples not seen during the learning process on which to test the generalization ability of the neural network).
• Generating allows the generation of a time series.

• Inputs allows to set the number of neural network inputs (the number of outputs is one, since the simulator is used only for binary classification problems).
• Parameters sets HDE characteristic parameters such as learning rate, random seed, tolerance, classification improvement threshold, maximum iterations for output layer, maximum iteration for hyperplane selection, maximum number of boundary points and maximum number of example pairs. The parameters setting screen is shown in Fig. 6.

  

• Training and Testing allow the setting of training and testing files, similar to the case of the backpropagation simulator.
• Graphics Modes allows the setting of the graphic display modes for tracing the constructive learning process. These include the display of training and test patterns, interpolation steps, points on separating hyperplanes, candidate and actual hyperplanes, as well as the final separating surface. An actual graphics screen resulting from a simulation in which all the graphics options except for the test pattern display are set is presented in Fig. 7.

  

• Network Switches allows the setting of different additional HDE options such as hidden unit creation, printing the network configuration and printing the training and the test sets.
• Advanced Options allows the selection of two sophisticated features. The Hybrid System option incorporates prior knowledge to the HDE learning provided in the form of an expert system [4]. The Parallel Computation option allows the parallel execution of the simulation on either a distributed system or on a highly parallel machine [5]. This option requires the presence of the p4 software package on the user's computer, available without cost from Argonne National Laboratories.

IV. Application in Research and Teaching

In summer 1995, the GUI was also used by neural network first-time users during the WSU/NSF Teacher Institute for Science/Mathematics Education through the Engineering Experiences training program. Two mathematics teachers with no prior knowledge of either the UNIX operating system or neural networks spent six weeks in our laboratory with the objective of learning neural network techniques for designing backpropagation experiments. After becoming accustomed with the GUI in a fast learning process, the results obtained by them on a few neural network benchmark classification problems like the Monks problems [11] and the breast cancer diagnosis problem [15] were comparable to the best known results for the corresponding problems. As a final result, they also developed a neural network teaching module to be used in high school.

Although the GUI and the embedded simulators have been successfully tested both in neural network research and training programs, more extensive in-class testing is needed. Hence, the GUI is intended for use in two courses offered in Fall '95 at the School of Electrical Engineering and Computer Science at Washington State University. One is an undergraduate course on Artificial Intelligence while the other one is a graduate course in Neural Networks. From these two diverse groups of students we expect to gain valuable suggestions on future extensions to the current version of the GUI. Finally, any feedback from the readers of this article and users of the software provided on the accompanying CD-ROM would be extremely beneficial for improving the current GUI implementation and embedding additional neural network simulators.

ACKNOWLEDGEMENTS

REFERENCES

[2] R. Drossu et al., ``Single and Multiple Frame Video Traffic Prediction Using Neural Network Models,'' in Computer Networks, Architecture and Applications, S. V. Raghavan and B. N. Jain eds., Chapman \& Hall, pp. 146-158, 1995.

[3] R. Drossu and Z. Obradovic, ``Novel Results on Stochastic Modelling Hints for Neural Network Prediction,'' in World Congress on Neural Networks, Washington D.C., vol. 3, pp. 230-233, 1995.

[4] J. Fletcher and Z. Obradovic, ``Combining Prior Symbolic Knowledge and Constructive Neural Networks,'' in Connection Science: Journal of Neural Computing, Artificial Intelligence and Cognitive Research, vol. 5, nos. 3-4, pp. 365-375, 1993.

[5] J. Fletcher and Z. Obradovic, ``A Discrete Approach to Constructive Neural Network Learning,'' in Neural, Parallel and Scientific Computations, vol. 3, no. 3, pp. 307-320, 1995.

[6] S. Haykin, ``Neural Networks: A Comprehensive Foundation,'' MacMillan Publishing Company, 1994.

[7] M. Mangeas, A. S. Weigend, ``Forecasting Electricity Demand Using Nonlinear Mixture of Experts,'' in World Congress on Neural Networks, Washington D.C., vol. 2, pp. 48-53, 1995.

[8] O. Matan et al., ``Multi-Digit Recognition Using a Space Displacement Neural Network,'' in Advances in Neural Information Processing Systems, vol. 4, pp. 488-495, 1992.

[9] J. K. Ousterhout, ``Tcl and the Tk Toolkit,'' Addison-Wesley, 1994.

[10] M. G. Sobell, ``UNIX System V: A Practical Guide. Third Edition,'' Benjamin Cummings, 1995.

[11] S.B. Thrun et al. ``The MONK's Problems: A Performance Comparison of Different Learning Algorithms,'' Technical Report, Department of Computer Science, Carnegie Mellon University, 1991.

[12] R. Venkateswaran and Z. Obradovic, ``Efficient Learning through Cooperation,'' in World Congress on Neural Networks, San Diego, CA, vol. 3, pp. 390-395, 1994.

[13] M. Welsh, L. Kaufman, ``Running LINUX,'' O'Reilly and Associates, Sebastopol, California, 1995.

[14] P. Werbos, ``Beyond Regression: New Tools for Predicting and Analysis in the Behavioral Sciences,'' Harvard University, Ph.D. thesis, 1974. Reprinted by Willey and Sons, 1995.

[15] W. H. Wolberg and O. L. Mangasarian, ``Multisurface Method of Pattern Separation for Medical Diagnosis Applied to Breast Cytology,'' in Proc. National Academy of Sciences, U.S.A., vol. 87, pp 9193-9196, 1990.

Zoran Obradovic received the B.S. degree in Applied Mathematics, Information and Computer Sciences in 1985, the M.S. degree in Mathematics and Computer Science in 1987, both from the University of Belgrade and the Ph.D. degree in Computer Science from the Pennsylvania State University in 1991. He is currently a research scientist at the Mathematical Institute of the Serbian Academy of Sciences and Arts, Belgrade and an Assistant Professor in the School of Electrical Engineering and Computer Science, Washington State University. The objective of his research is to explore the applicability of neural network technology to large scale classification and time series prediction problems in very noisy domains.

Justin Fletcher received his B.S. in Interdisciplinary Studies in 1980 and his M.S. in Computer Science in 1983 from the University of Idaho. From 1982 until 1991 he worked as software engineer, consultant and software manager in industry. He received his Ph.D. degree in Computer Science from Washington State University in 1994 specializing in Artificial Neural Networks and is currently a software consultant.