One of the most important task in machine learning and data science is to solve an optimization problem. We consider to solve a stochastic Optimization problem via Hessian approximation. It is difficult to compute the original Hessian and to inverse Hessian when data is large. We propose to approximate Hessian by Nystrom approximation. Nystrom approximation is Newton step-based stochastic optimization algorithm for large-scale empirical risk minimization of convex functions. Specifically, Nystrom approximation computes a partial column Hessian of size (d × k) with k << d randomly selected parameters, then use the regularized Nyström method to better approximate the full Hessian matrix. In order to make better approximation, it needs to solve a sub-problem. However, the solution of sub-problem by Nystrom approximation becomes large and therefore it becomes unstable. We address this sub-problem by incorporating damping of search direction using regularized Levenberg–Marquardt method.
Recently, second order optimization methods are widely used in the machine learning and data science to train the large scale data.
| 氏名 | 専攻 | 研究室 | 役職/学年 |
|---|---|---|---|
| Hardik Tankaria | 数理工学専攻 | System Optimization Lab | 博士3回生 |
| Dr. Dinesh Singh | その他: その他 | High-dimensional Statistical Modeling Team, RIKEN center for Advanced Intelligence Project (AIP) | 研究員 |
| Prof. Makoto Yamada | 知能情報学専攻 | High-dimensional Statistical Modeling Team, RIKEN center for Advanced Intelligence Project (AIP) / Collective Intelligence Lab, Kyoto University | 准教授 |
| Prof. Nobuo Yamashita | 数理工学専攻 | System Optimization Lab | 教授 |