Home   >   CSC-OpenAccess Library   >    Manuscript Information
Performance Analysis of Various Activation Functions in Generalized MLP Architectures of Neural Networks
Bekir Karlik, Ahmet Vehbi
Pages - 111 - 122     |    Revised - 31-01-2011     |    Published - 08-02-2011
Volume - 1   Issue - 4    |    Publication Date - December 2010  Table of Contents
Activation Functions, Multi Layered Perceptron, Neural Networks, Performance Analysis
The activation function used to transform the activation level of a unit (neuron) into an output signal. There are a number of common activation functions in use with artificial neural networks (ANN). The most common choice of activation functions for multi layered perceptron (MLP) is used as transfer functions in research and engineering. Among the reasons for this popularity are its boundedness in the unit interval, the function’s and its derivative’s fast computability, and a number of amenable mathematical properties in the realm of approximation theory. However, considering the huge variety of problem domains MLP is applied in, it is intriguing to suspect that specific problems call for single or a set of specific activation functions. The aim of this study is to analyze the performance of generalized MLP architectures which has back-propagation algorithm using various different activation functions for the neurons of hidden and output layers. For experimental comparisons, Bi-polar sigmoid, Uni-polar sigmoid, Tanh, Conic Section, and Radial Bases Function (RBF) were used.
CITED BY (94)  
1 Klimashevich AV Nikolsky, VI, & Bogonina, OV experience in prevention and treatment of post-burn scarring esophageal strictures by stenting. (Repeat after neg reviews). HERALD Surgical Gastroenterology, 68.
2 Murugadoss, R., & Ramakrishnan, M. Nonlinear Approximations in Sigmoid Transfer Function for Improved Statistical Pattern Recognition Based On PNN Bayesian Approach.
3 Murugadoss, R., & Ramakrishnan, M. universal approximation with non-sigmoid hidden layer activation functions by using artificial neural network modeling.
4 Mondal, K. Recognition of Static Hand Gestures of Alphabet in Bangla Sign Language.
5 Tantawy, M., & Zorkany, M. A Suitable Approach for Evaluating Bus Arrival Time Prediction Techniques in Egypt. algorithms, 2, 9.
6 Cazella, S. C. Thiago Nunes Kehl Viviane Todt Maurício Roberto Veronez.
7 Sakthivel, S., & Habeeb, S. K. M. NNvPDB: Neural Network based Protein Secondary Structure Prediction with PDB Validation.
8 Karlik, B., Uncu, U., & Ayhan, T. Neural Network Methodology for Modeling Heat Transfer in Wake Flow.
9 Grd, P. two-dimensional face image classification for distinguishing children from adults based on anthropometry.
10 Tosatto, S. C. Neural-Symbolic Learning: How to play Soccer. In Seventh International Workshop on Neural-Symbolic Learning and Reasoning (p. 36).
11 Klimashevich, A. Nikolsky, VM, BOGONINA, O., & KUVAKOVA, R. (2012). Neural network model in treating and preventing post-burn scar formation of structures esophagus. fundamental research (2-0).
12 Supriyono, H., & Tokhi, M. O. (2012, February). Dynamic Neuro-modelling Using Bacterial Foraging Optimisation with Fuzzy Adaptation. In Intelligent Systems, Modelling and Simulation (ISMS), 2012 Third International Conference on (pp. 109-114). IEEE.
13 Soares, F. A. A. D. M. (2012). Predição recursiva de diâmetros de clones de eucalipto utilizando rede Perceptron de múltiplas camadas para o cálculo de volume (Doctoral dissertation).
14 Tasic, J. (2012). Procesiranje slikovnih analogija neuronskim mrežama.
15 Mohamad, M., Saman, M. Y. M., & Hitam, M. S. (2012, October). A framework for multiprocessor neural networks systems. In ICT Convergence (ICTC), 2012 International Conference on (pp. 44-48). IEEE.
16 Hilbish, N. (2012). Multiple Fundamental Frequency Pitch Detection for Real Time MIDI Applications.
17 Isa, I. S., Fauzi, N. A., Sharif, J. M., Baharudin, R., & Abbas, M. H. (2012, November). Comparisons of MLP transfer functions for different classification classes. In Control System, Computing and Engineering (ICCSCE), 2012 IEEE International Conference on (pp. 110-114). IEEE.
18 Zhou, Q. (2012). Selective omission of road networks in multi-scale representation (Doctoral dissertation, The Hong Kong Polytechnic University).
19 Supriyono, H. (2012). Novel bacterial foraging optimisation algorithms with application to modelling and control of flexible manipulator systems.
20 Anjo, M. D. S., Pizzolato, E. B., & Feuerstack, S. (2012, November). A real-time system to recognize static gestures of Brazilian sign language (libras) alphabet using Kinect. In Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems (pp. 259-268). Brazilian Computer Society.
21 Karan, O., Bayraktar, C., Gümüskaya, H., & Karlik, B. (2012). Diagnosing diabetes using neural networks on small mobile devices. Expert Systems with Applications, 39(1), 54-60.
22 Yücelbas, S. (2013). Hibrit siniflayicilar kullanarak kalpteki ritim bozukluklarinin teshisi (Doctoral dissertation, Selçuk Üniversitesi Fen Bilimleri Enstitüsü).
23 Fera, M., Lambiase, A., Fruggiero, F., Martino, G., & Nenni, M. E. (2013). Production Scheduling Approaches for Operations Management. INTECH Open Access Publisher.
24 Parvin, A. (2013). Application of Neural Networks with CSD Coefficients for Human Face Recognition.
25 Vijean, V., Hariharan, M., Yaacob, S., & Sulaiman, M. N. B. (2013). Stockwell transform and clustering techniques for efficient detection of vision impairments from single trial VEPs. International Journal of Medical Engineering and Informatics, 5(4), 352-371.
26 Klimashevich AV Nikolsky, VI, Bogonina, OV, Akimov, AA, & Shabrov, AV (2013). A method of predicting esophageal stricture scar AFTER burns. Fundamental research (2-1).
27 Velican, v. (2013). teza de doctorat (doctoral dissertation, academia tehnica militara).
28 Saputri, T. R. D., & Lee, S. W. (2013). Using Artificial Neural Networks for Predicting Traffic Conditions: A Learning Algorithm for Long-term Time Series Forecasting. Journal of Convergence Information Technology, 8(14), 121.
29 Abd Aziz, N., Latif, A., Al Kasyaf, M., Abdullah, W. F. H., Md Tahir, N., & Zolkapli, M. (2013, November). Hardware implementation of backpropagation algorithm based on CHEMFET sensor selectivity. In Control System, Computing and Engineering (ICCSCE), 2013 IEEE International Conference on (pp. 387-390). IEEE.
30 Tang, W. (2013). Modeling, Estimation, and Control of Nonlinear Time-Variant Complex Processes (Doctoral dissertation, Texas Tech University).
31 Yeremia, H., Yuwono, N. A., Raymond, P., & Budiharto, W. (2013). Genetic algorithm and neural network for optical character recognition. Journal of Computer Science, 9(11), 1435.
32 KARLIK, B. (2013). Soft Computing Methods in Bioinformatics: A Comprehensive Review. Mathematical and Computational Applications, 18(3), 176-197.
33 Vijean, V., Hariharan, M., Yaacob, S., Sulaiman, M. N. B., & Adom, A. H. (2013). Objective investigation of vision impairments using single trial pattern reversal visually evoked potentials. Computers & Electrical Engineering, 39(5), 1549-1560.
34 Aziz, N. A., Abdullah, W. F. H., Md Tahir, N., Adenan, M. N. H., & Jamil, W. (2013, August). Enhancement of CHEMFET sensor selectivity based on backpropagation algorithm. In System Engineering and Technology (ICSET), 2013 IEEE 3rd International Conference on (pp. 226-231). IEEE.
35 Zaki, M., Ashour, I., Zorkany, M., & Hesham, B. (2013). Online Bus Arrival Time Prediction Using Hybrid Neural Network and Kalman filter Techniques. International Journal of Modern Engineering Research, 3(4), 2035-2041.
36 Yerrabolu, P., Mareddy, L., Bhatt, D., Aggarwal, P., Kumar, A., & Devabhaktuni, V. (2013). Correction Model-Based ANN Modeling Approach for the Estimation of Radon Concentrations in Ohio. Environmental Progress & Sustainable Energy, 32(4), 1223-1233.
37 Devabhaktuni, V., Bunting, C. F., Green, D., Kvale, D., Mareddy, L., & Rajamani, V. (2013). A new ANN-based modeling approach for rapid EMI/EMC analysis of PCB and shielding enclosures. Electromagnetic Compatibility, IEEE Transactions on, 55(2), 385-394.
38 Horng, S. C., & Lin, S. Y. (2013). Evolutionary algorithm assisted by surrogate model in the framework of ordinal optimization and optimal computing budget allocation. Information Sciences, 233, 214-229.
39 AlBakkar, A. (2014). Adaptive Simplified Neuro-Fuzzy Controller as Supplementary Stabilizer for SVC.
40 Rotich, N. K., Backman, J., Linnanen, L., & Daniil, P. (2014). Wind Resource Assessment and Forecast Planning with Neural Networks. Journal of Sustainable Development of Energy, Water and Environment Systems, 2(2), 174-190.
41 Jeong, K. (2014). Learning from e-learning: Testing Intelligent Learning Systems in South Asia.
42 Viswanathan, a., & chitra, s. (2014). optimized radial basis function classifier with hybrid bat algorithm for multi modal biometrics. journal of theoretical & applied information technology, 67(1).
43 Mohan, A. (2014). A New Spatio-Temporal Data Mining Method and its Application to Reservoir System Operation (Doctoral dissertation, University of Nebraska).
44 ULER, H. G., Sahin, M., & Ferikoglu, A. (2014). Feature selection on single-lead ECG for obstructive sleep apnea diagnosis. Turkish Journal of Electrical Engineering & Computer Sciences, 22, 465-478.
45 Zhou, Q., & Li, Z. (2014). Use of Artificial Neural Networks for Selective Omission in Updating Road Networks. The Cartographic Journal, 51(1), 38-51.
46 Al-Khasawneh, A., & Hijazi, H. (2014). A Predictive E-Health Information System: Diagnosing Diabetes Mellitus Using Neural Network Based Decision Support System. International Journal of Decision Support System Technology (IJDSST), 6(4), 31-48.
47 Arvidsson, J. (2014). Forecasting on-demand video viewership ratingsusing neural networks.
48 Asgari, H. (2014). Modelling, Simulation and Control of Gas Turbines Using Artificial Neural Networks.
49 Dogman, A., & Saatchi, R. (2014). Multimedia traffic quality of service management using statistical and artificial intelligence techniques. IET Circuits, Devices & Systems, 8(5), 367-377.
50 Kumar, R., Chand, K., & Lal, S. P. (2014). Gene Reduction for Cancer Classification Using Cascaded Neural Network with Gene Masking. In Advances in Artificial Intelligence (pp. 301-306). Springer International Publishing.
51 Singh, V., & Lai, S. P. (2014, November). Digit recognition using single layer neural network with principal component analysis. In Computer Science and Engineering (APWC on CSE), 2014 Asia-Pacific World Congress on (pp. 1-7). IEEE.
52 Essai, M. H., & Abd Ellah, A. R. (2014, December). M-Estimators based activation functions for robust neural network learning. In Computer Engineering Conference (ICENCO), 2014 10th International (pp. 70-75). IEEE.
53 Nedic, V., Despotovic, D., Cvetanovic, S., Despotovic, M., & Babic, S. (2014). Comparison of classical statistical methods and artificial neural network in traffic noise prediction. Environmental Impact Assessment Review, 49, 24-30.
54 Vijean, V., Hariharan, M., Yaacob, S., & Sulaiman, M. N. B. (2014). Application of clustering techniques for visually evoked potentials based detection of vision impairments. Biocybernetics and Biomedical Engineering, 34(3), 169-177.
55 Valarmathi, P., & Robinson, S. (2014, December). Efficacy of feature selection techniques for Multilayer Perceptron Neural Network to classify mammogram. In Advanced Computing (ICoAC), 2014 Sixth International Conference on (pp. 26-31). IEEE.
56 Golovko, A. (2014). Foreign exchange rate movement prediction using triangle chart patterns and artificial neural networks (Doctoral dissertation, Tartu Ülikool).
57 Amirov, A., Gerget, O., Devjatyh, D., & Gazaliev, A. (2014). Medical Data Processing System Based on Neural Network and Genetic Algorithm. Procedia-Social and Behavioral Sciences, 131, 149-155.
58 García de Soto, B., Adey, B. T., & Fernando, D. (2014). A Process for the Development and Evaluation of Preliminary Construction Material Quantity Estimation Models Using Backward Elimination Regression and Neural Networks. Journal of Cost Analysis and Parametrics, 7(3), 180-218.
59 Ramkishore, S., Madhumitha, P., & Palanichamy, P. (2014, September). Comparison of Logistic Regression and Support Vector Machine for the Classification of Microstructure and Interfacial Defects in Zircaloy-2. In Soft Computing and Machine Intelligence (ISCMI), 2014 International Conference on (pp. 130-134). IEEE.
60 Laqrichi, S., Marmier, F., & Gourc, D. (2014). Software Cost and Duration Estimation Based on Distributed Project Data: A General Framework. In Enterprise Interoperability VI (pp. 213-224). Springer International Publishing.
61 Sun, W., Su, F., & Wang, L. (2014, December). Improving deep neural networks with multilayer maxout networks. In Visual Communications and Image Processing Conference, 2014 IEEE (pp. 334-337). IEEE.
62 Al Doori, M., & Beyrouti, B. (2014). Credit scoring model based on back propagation neural network using various activation and error function. IJCSNS International Journal of Computer Science and Network Security, 14(3), 16-24.
63 Rotich, N. (2014). Forecasting of wind speeds and directions with artificial neural networks.
64 Gurmu, Z. K., & Fan, W. D. (2014). Artificial Neural Network Travel Time Prediction Model for Buses Using Only GPS Data. Journal of Public Transportation, 17(2), 3.
65 Garg, G., & Sharma, P. (2014). An Analysis of Contrast Enhancement using Activation Functions. International Journal of Hybrid Information Technology, 7(5), 235-244.
66 Tan, T. G., Teo, J., & Anthony, P. (2014). A comparative investigation of non-linear activation functions in neural controllers for search-based game AI engineering. Artificial Intelligence Review, 41(1), 1-25.
67 Vukicevic, A. M., Jovicic, G. R., Stojadinovic, M. M., Prelevic, R. I., & Filipovic, N. D. (2014). Evolutionary assembled neural networks for making medical decisions with minimal regret: Application for predicting advanced bladder cancer outcome. Expert Systems with Applications, 41(18), 8092-8100.
68 Zhang, C., Jiang, J., Ma, J., Zhang, X., Yang, Q., Ouyang, Q., & Lei, X. (2015). Evaluating soil reinforcement by plant roots using artificial neural networks. Soil Use and Management, 31(3), 408-416.
69 Feng Chang. (2015) study in depth a positive linear function of neural networks. Computer Engineering and Design, 36 (3), 759-762.
70 Gautam, C., & Ravi, V. (2015). Data imputation via evolutionary computation, clustering and a neural network. Neurocomputing, 156, 134-142.
71 Akar, M., Hekim, M., & Orhan, U. (2015). Mechanical fault detection in permanent magnet synchronous motors using equal width discretization-based probability distribution and a neural network model. Turkish Journal of Electrical Engineering & Computer Sciences, 23(3).
72 Deo, R. C., & Sahin, M. (2015). Application of the Artificial Neural Network model for prediction of monthly Standardized Precipitation and Evapotranspiration Index using hydrometeorological parameters and climate indices in eastern Australia. Atmospheric Research, 161, 65-81.
73 Genç, B. (2015). A methodology for evaluating utilisation of mine planning software and consequent decision-making strategies in South Africa (Doctoral dissertation, Faculty of Engineering and the Built Environment, University of the Witwatersrand, Johannesburg).
74 Laqrichi, S., Marmier, F., Gourc, D., & Nevoux, J. (2015). Integrating uncertainty in software effort estimation using Bootstrap based Neural Networks. IFAC-PapersOnLine, 48(3), 954-959.
75 Zissis, D., Xidias, E. K., & Lekkas, D. (2015). A cloud based architecture capable of perceiving and predicting multiple vessel behaviour. Applied Soft Computing, 35, 652-661.
76 Yakovyna, v. s. (2015). software failures prediction using rbf neural network.
77 Yakovyna Vitaliy, S. (2015). of article.
78 Foster, R. (2015). A comparison of machine learning techniques for hand shape recognition.
79 Gupta, S., & Kashyap, S. (2015). Forecasting inflation in G-7 countries: an application of artificial neural network. foresight, 17(1), 63-73.
80 Kudlácek, J. (2015). Analýza velkých dat v mobilních sítích.
81 Wang, A., An, N., Chen, G., Li, L., & Alterovitz, G. (2015). Predicting hypertension without measurement: A non-invasive, questionnaire-based approach. Expert Systems with Applications, 42(21), 7601-7609.
82 Liu, P. H. (2015). Novel Convolutional Neural Networks for Deep Learning and Its Applications to General Image Classification.
83 Zissis, D., Xidias, E. K., & Lekkas, D. (2015). Real-time vessel behavior prediction. Evolving Systems, 1-12.
84 Lu, J., Xue, S., Zhang, X., & Han, Y. (2015). A Neural Network-Based Interval Pattern Matcher. Information, 6(3), 388-398.
85 Kashyap, Y., Bansal, A., & Sao, A. K. (2015). Solar radiation forecasting with multiple parameters neural networks. Renewable and Sustainable Energy Reviews, 49, 825-835.
86 Zaki, M., Hamouda, A., & Hisham, B. (2015). Travel Time Prediction under Egypt Heterogeneous Traffic Conditions using Neural Network and Data Fusion. Egyptian Computer Science Journal, 39(2).
87 Sarac, B., Karlik, B., Uncu, U., & Ayhan, T. (2015). Neural Network Methodology for Modeling Heat Transfer in Wake Flow. Journal of Heat Transfer, 137(2), 022201.
88 Borrotti, M., Pievatolo, A., Critelli, I., Degiorgi, A., & Colledani, M. (2015). A computer-aided methodology for the optimization of electrostatic separation processes in recycling. Applied Stochastic Models in Business and Industry.
89 Wróbel, J., & Kulawik, A. (2015, March). Using the artificial neural networks in the modelling of the induction heating. In PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON NUMERICAL ANALYSIS AND APPLIED MATHEMATICS 2014 (ICNAAM-2014) (Vol. 1648, p. 850090). AIP Publishing.
90 Jaddi, N. S., Abdullah, S., & Hamdan, A. R. (2015). Optimization of neural network model using modified bat-inspired algorithm. Applied Soft Computing, 37, 71-86.
91 Hussain, F., & Jeong, J. (2015, March). Exploiting deep neural networks for digital image compression. In Web Applications and Networking (WSWAN), 2015 2nd World Symposium on (pp. 1-6). IEEE.
92 Kashyap, Y., Bansal, A., & Sao, A. K. (2015). Spatial Approach of Artificial Neural Network for Solar Radiation Forecasting: Modeling Issues. Journal of Solar Energy, 2015.
93 Jaddi, N. S., Abdullah, S., & Hamdan, A. R. (2015). Multi-population cooperative bat algorithm-based optimization of artificial neural network model. Information Sciences, 294, 628-644.
94 KARAN O?uz, BAYRAKTAR Canan, GÜMÜ?KAYA Haluk, KARLIK Bekir, “Diagnosing Diabetes Using Neural Networks on Small Mobile Devices”, Expert Systems with Applications, vol. 39 (2012), pp. 54-60, 2012
1 Google Scholar 
2 CiteSeerX 
3 refSeek 
4 Scribd 
5 SlideShare 
6 PdfSR 
B DasGupta, G. Schnitger, “The Power of Approximating: A Comparison of Activation Functions”. In Giles, C. L., Hanson, S. J., and Cowan, J. D., editors, Advances in Neural Information Processing Systems, 5, pp. 615-622, San Mateo, CA. Morgan Kaufmann Publishers, 1993
B. Ciocoiu, “Hybrid Feedforward Neural Networks for Solving Classification Problems”.Neural Processing Letters, 16(1):81-91, 2002.
B. Widrow, M.A. Lehr, “30 years of adoptive neural netwoks; perceptron, madaline, and back propagation”. Proc. IEEE, 78: 1415–1442, 1990
G. Cybenko, “Approximation by superposition of a sigmoidal function”. Mathematics of Control, Signals, and Systems, 2(4):303–314, 1987
G. Dorffner, “Unified frameworks for MLP and RBFNs: Introducing Conic Section Function Networks”. Cybernetics and Systems, 25: 511-554, 1994
J. M. Sopena, E. Romero, R. Alqu´ezar, “Neural networks with periodic and monotonic activation functions: a comparative study in classification problems”. In Proceedings of the 9th International Conference on Artificial Neural Networks, pp. 323-328, 1999
M.D. Buhmann, (2003), Radial Basis Functions: Theory and Implementations, Cambridge University Press, ISBN 978-0-521-63338-3.
M.I. Jordan, “Why the logistic function? A tutorial discussion on probabilities and neural networks”. Computational Cognitive Science Technical Report 9503, Massachusetts Institute of Technology, 1995
R. P. Lippmann, “An introduction to computing with neural nets”. IEEE Acoustics, Speech and Signal Processing, 4(2):4–22, 1987
T. Poggio, F. Girosi, “A theory of networks for approximation and learning”. A.I. Memo No.1140, Artificial Intelligence, Laboratory, Massachusetts Institute of Technology, 1989
Y. Bodyanskiy, N. Lamonova, O. Vynokurova, “Double-Wavelet Neuron Based on Analytical Activation Functions”. International Journal Information Theories & Applications, 14: 281-288,2007
Y. Liu, X. Yao, “Evolutionary Design of Artificial Neural Networks with Different Nodes”. In Proceedings of the Third IEEE International Conference on Evolutionary Computation, pp.570-675, 1996
Dr. Bekir Karlik
- Turkey
Mr. Ahmet Vehbi
Fatih University - Turkey