Balabala
我想讀完Giuseppe C. Calafiore, Laurent El Ghaoui 版本的《Optimization Models》。
1. Introduction
作者列舉了若干個例子，並闡述了最佳化問題的一般形式，最主要的洞見有：
- 一些具有特定性質的問題是 tractable 的：The focus of this book is on tractable models, and a key message is that models that can be formulated in the form of linear algebra problems, or in convex form, are typically tractable.
- 透過一些巧妙的設計可以使問題 tractable：A problem that may seem hard under a certain formulation may well become tractable if we put some more effort and intelligence in the modeling phase.
- 即使是難以進行巧妙的設計的問題，也可能透過近似的方法解決：However, even for intrinsically hard problems, for which exact solutions may be unaffordable, we may often find useful tractable models that provide us with readily computable approximate, or relaxed, solutions.
典型的最佳化問題分類
-
Least squares and linear equations：
\[\min _x \sum_{i=1}^m\left(\sum_{j=1}^n A_{i j} x_j-b_i\right)^2 \]最小二乘的典型應用是求解一組線性方程 \(\sum_{j=1}^n A_{i j} x_j=b_i, \quad i=1, \ldots, m\)
-
Low-rank approximations and maximum variance：
Low-rank approximations：
\[\min _{x \in \mathbb{R}^n, z \in \mathbb{R}^m} \sum_{i=1}^m\left(\sum_{j=1}^n A_{i j}-z_i x_j\right)^2 \]低秩近似的基本思想為用 \(z_ix_j\) 近似 \(A_{ij}\).
相關的 maximum-variance 為：
\[\max _x \sum_{i=1}^m\left(\sum_{j=1}^n A_{i j} x_j\right)^2 \text { s.t.: } \sum_{i=1}^n x_i^2=1 . \]maximum-variance 問題可用於在高維空間中擬合一條直線。這一點應該跟最小二乘相似，只是這裡不計較\(b_i\)，只得到”斜率“即可。
-
Linear and quadratic programming
Linear programming (LP) problem：
\[\min _x \sum_{j=1}^n c_j x_j \text { s.t.: } \sum_{j=1}^n A_{i j} x_j \leq b_i, \quad i=1, \ldots, m, \]Quadratic programming problems：
\[\min _x \sum_{i=1}^r\left(\sum_{j=1}^n C_{i j} x_j\right)^2+\sum_{j=1}^n c_j x_j \text { s.t.: } \sum_{j=1}^n A_{i j} x_j \leq b_i, i=1, \ldots, m, \] -
Convex optimization
- Convex optimization problems are problems of the form (1.2), where the objective and constraint functions have the special property of convexity.
- One key feature of convex problems is that all local minima are actually global.
容易求解的一類問題。
-
Combinatorial optimization
整數規劃很難，整數與連續變數混合的問題（mixed integer programs ）很難。
-
Non-convex optimization
由於可能存在區域性最優點等問題，較難解決。但對於一些特定問題，例如 Low-rank approximations and maximum variance，已有可靠的來自於線性代數的演算法。
歷史
線性代數起源於古代中國 -> 高斯對線性代數的探索 -> 最佳化思想在物理中的應用 ->...
比較有意思的故事：
In the Soviet Union at that time, the focus was more towards optimization theory, perhaps due to more restricted access to computing resources. Since nonlinear problems are hard, Soviet researchers went back to the linear programming model, and asked the following (at that point theoretical) question: what makes linear programs easy? Is it really linearity of the objective and constraint functions, or some other, more general, structure? Are there classes of problems out there that are nonlinear but still easy to solve?
In the late 80s, two researchers in the former Soviet Union, Yurii Nesterov and Arkadi Nemirovski, discovered that a key property that makes an optimization problem "easy" is not linearity, but actually convexity.