Home   >   CSC-OpenAccess Library   >    Manuscript Information
Full Text Available

This is an Open Access publication published under CSC-OpenAccess Policy.
Publications from CSC-OpenAccess Library are being accessed from over 74 countries worldwide.
Penalized Regressions with Different Tuning Parameter Choosing Criteria and the Application in Economics
Sheng Gao, Mingwei Sun
Pages - 7 - 17     |    Revised - 31-07-2020     |    Published - 31-08-2020
Volume - 8   Issue - 1    |    Publication Date - August 2020  Table of Contents
Penalized Regression, Lasso, Ridge, Elastic Net, AIC, BIC, AICc, Economic Modeling.
Recently a great deal of attention has been paid to modern regression methods such as penalized regressions which perform variable selection and coefficient estimation simultaneously, thereby providing new approaches to analyze complex data of high dimension. The choice of the tuning parameter is vital in penalized regression. In this paper, we studied the effect of different tuning parameter choosing criteria on the performances of some well-known penalization methods including ridge, lasso, and elastic net regressions. Specifically, we investigated the widely used information criteria in regression models such as Bayesian information criterion (BIC), Akaike’s information criterion (AIC), and AIC correction (AICc) in various simulation scenarios and a real data example in economic modeling. We found that predictive performance of models selected by different information criteria is heavily dependent on the properties of a data set. It is hard to find a universal best tuning parameter choosing criterion and a best penalty function for all cases. The results in this research provide reference for the choices of different criteria for tuning parameter in penalized regressions for practitioners, which also expands the nascent field of applications of penalized regressions.
1 H. Akaikei. "Information theory and an extension of maximum likelihood principle," in Proc. 2nd Int. Symp. on Information Theory, 1973, pp. 267-281.
2 N. Anaraki. "A housing market without fannie mae and freddie mac: The effect of housing starts." The Heritage Foundation. Oct. 2012.
3 K. Burnham and D. Anderson. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd edn. NY: Springer, 2002.
4 S. Chand. "On tuning parameter selection of lasso-type methods-a monte carlo study." In Proc. IBCAST, 2012, pp. 120-129.
5 J. Fan and R. Li. "Variable selection via nonconcave penalized likelihood and its oracle properties." Journal of American Statistical Association, vol.96, pp. 1348-1360, Dec. 2001.
6 J.Fan and H. Peng. "Nonconcave penalized likelihood with a diverging number of parameters." The Annals of Statistics, vol.32, pp.928-961, 2004.
7 J. Friedman, T. Hastie, and R. Tibshirani. "Regularization paths for generalized linear models via coordinate descent." Journal of statistical software, vol.33, pp. 1-22. Aug. 2010.
8 F. Gunes. "Penalized Regression Methods for Linear Models in SAS/STAT®." in Proceedings of the SAS Global Forum 2015 Conference. Cary, NC: SAS Institute Inc. [on-line] Available: http://support. sas. com/rnd/app/stat/papers/2015/PenalizedRegression_LinearModels. pdf. 2015.
9 A.E. Hoerl and R.W. Kennard. "Ridge regression: Biased estimation for nonorthogonal problems." Technometrics, vol.12, pp. 55-67, Feb. 1970.
10 C.M. Hurvich and C.L. Tsai. "Regression and time series model selection in small samples." Biometrika, vol.76, pp. 297-307, Jun. 1989.
11 L. Mutikani. "Housing starts near eight-year high, but permits fall." Internet: https://finance.yahoo.com/news/uhousing-starts-near-eight-132259302.html, Aug.15, 2015 [Mar.26, 2020].
12 Y. Ninomiya and S. Kawano. "AIC for the LASSO in generalized linear models." Electronic Journal of Statistics, vol. 10, pp. 22537-2560, 2016.
13 J.O. Ogutu, T. Schulz-Streeck, and H.P. Piepho. "Genomic selection using regularized linear regression models: ridge regression, lasso, elastic net and their extensions." BMC proceedings, vol.6, pp. S10, Dec. 2012.
14 M.J. Pring, The all-season investor: successful strategies for every stage in the business cycle. John Wiley & Sons, 1992.
15 D.N. Schreiber-Gregory. "Ridge Regression and multicollinearity: An in-depth review." Model Assisted Statistics and Applications, vol.13, pp. 359-365, Jan. 2018.
16 G. Schwarz. "Estimating the dimension of a model." The annals of statistics, Vol. 6, pp.461-464. 1978
17 L.K. Sen and M. Shitan. "The performance of AICC as an order selection criterion in ARMA time series models." Pertanika Journal of Science and Technology, vol.10, pp.25-33. Jan. 2002.
18 P. Shi and C.L. Tsai. "Regression model selection-A residual likelihood approach" Journal of the Royal Statistical Society Series B, vol.64, pp. 237-52, May. 2002.
19 N. Sugiura. "Further analysts of the data by akaike's information criterion and the finite corrections: Further analysts of the data by akaike's." Communications in Statistics-Theory and Methods, vol.7, pp. 13-26, Jan. 1978.
20 R. Tibshirani. "Regression Shrinkage and Selection via the Lasso." Journal of the Royal Statistical Society: Series B (Methodological), vol. 58, pp. 267-88, Jan.1996.
21 H. Wang, B. Li, and C. Leng. "Shrinkage tuning parameter selection with a diverging number of parameters." Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 71, pp.671-683, Jun. 2009.
22 H. Zou and T. Hastie. "Regularization and variable selection via the Elastic Net." Journal of the Royal Statistical Society: Series B, vol.67, pp.301-320, Apr. 2005.
Mr. Sheng Gao
Mathematics and Computer Science Department, Samford University, Birmingham, 35229 - United States of America
Dr. Mingwei Sun
Mathematics and Computer Science Department, Samford University, Birmingham, 35229 - United States of America