비룡의 컴퓨터이야기

장점
1. 들고 다니기 굉장히 편입니다. K8 모델의 경우, 들고 다니면서 사용하기에는 많이 불편한데 이 모델의 경우에는 충분히 들고다니면서 움직일만 한 것 같습니다.
2. 소음이 굉장히 적은 편입니다. 연구실에서 사용하기에 아무리 적축이라고 해도 무리가 있었는데, 펜타그래피 모델은 굉장히 조용해서 연구실이나 조용함을 추구해야하는 환경에서 잘 사용할 수 있는 모델입니다.
단점
1. 상대적으로 K8에 비해서 타건감이 좋지는 않습니다. 단, 이것은 제품 자체의 문제라기보다는 펜타그래피 모델과 기계식 키보드의 차이점인만큼 개인적인 선호가 반영되어있는 단점입니다.

K8 제품과 마찬가지로 키크론 제품의 경우 맥을 사용하고 있는 저에게는 더할나위 없는 좋은 선택지였습니다. 맨 처음 제가 K8을 입문하게 된 계기가 맥과 호환이 되면서 기계식 키보드이고 블루투스 키보드를 잘 제공하고 있다는 점 때문이였는데, 이번 모델 B1 pro 역시 기계식 키보드만 제외하고 맥과의 호환성 및 블루투스라는 점에서는 충분히 매력적인 제품이라고 생각합니다. 아래 사진에서 볼 수 있는 것처럼 맥북과 아이패드에서 정상적으로 작동하는 것을 확인할 수 있습니다.

기존의 키크론 사용자로서 기대가 많은 제품이였는데, 역시나 잘 만든 제품이라고 생각이 되었고 다음과 같은 분들에게 추천드립니다.

맥 관련 디바이스를 사용하고 계신분
펜타그래피 블루투스 키보드를 찾고 계시는 분

이 두 장점을 충족시키는 제품 키크론 B1 자신있게 추천드릴 수 있을 것 같습니다.

[본 후기는 키크론으로부터 제품을 제공받아 작성한 후기 입니다]

Abstract Algebra Notes (Sylow theorem)

비룡 — Sat, 11 May 2024 16:21:01 +0900

Abstract_Algebra_1.pdf

2.00MB

Last update : 2024.5.11

Diffusion model

비룡 — Sat, 30 Mar 2024 19:23:28 +0900

TBD

Last update : 2024/03/30

6. Approximation and fitting

비룡 — Fri, 23 Feb 2024 18:43:00 +0900

Norm approximation

Basic norm approximation problem

Assume that the columns of $A$ are independent.

\text{minimize }\|Ax - b\|

where $A\in \R^{m\times n}$ with $m\ge n$ , $\|\cdot\|$ is a norm of $\R^m$

Geometric interpretation

Geometrically, the solution $x^*$ is the point such that $Ax^*\in \mathcal R(A)$ that closest to $b$ . The vector

r = Ax - b

is called the residual for the problem; its components are sometimes called the individual residuals associated with $x$ .

Estimation interpretation

It can be interpreted as a problem of estimating a parameter vector based on an imperfect linear vector measurement. We consider a linear measurement model

y = Ax + v

where $y\in \R^m$ is a vector measurement, $x\in \R^n$ is a vector of parameters to be estimated, and $v\in \R^m$ is some unknown measurement error, presumed to be small in the norm $\|\cdot\|$ . The estimation problem is to make a sensible guess as to what $x$ is for given $y$ .

If we guess that $x$ has the value $\hat x$ , the most plausible guess for $x$ is

\hat x = \argmin_z \|Ax - y\|

Example

least squares approximation ( $\|\cdot \|_2$ )
The most common norm approximation problem involves the Euclidean or $l_2$ -norm. By squaring the objective, we obtain an equivalent problem which is called the least-squares approximation problem
$\text{minimize }\|Ax-b\|_2^2 = r_1^2 + \cdots + r_m^2$
where the objective is the sum of squares of the residuals. It can be solvable by normal equations
$A^TAx = A^Tb$
If $A$ has a full rank (i.e. the rows of $A$ are independent), the least-squares approximation problem has a unique solution
$x = (A^TA)^{-1}A^Tb$

Chebyshev approximation ( $\|\cdot \|_\infty$ )
When the $l_\infty$ -norm is used, the norm approximation problem
$\text{minimize }\|Ax - b\|_\infty = \max\{|r_1|, \dots, |r_m|\}$
is called the Chebyshev approximation problem or minimax approximation problem since we are to minimize the maximum absolute value of residuals. The Chebyshev approximation problem can be solved as an LP
$\text{minimize }t$
subject to
$-t\bold1 \preceq Ax - b\preceq t\bold 1$
with variables $x\in \R^n$ and $t\in \R$

Sum of absolute residuals approximation ( $\|\cdot \|_1$ )
When the $l_1$ -norm is used, the norm approximation problem
$\text{minimize }\|Ax-b\|_1 = |r_1| + \cdots + |r_m|$
is called the sum of absolute residuals approximation problem, or in the context of estimation, it is called a robust estimator. Like the Chebyshev approximation problem, this problem can be expressed as an LP
$\text{minimize }1^Tt$
subject to
$-t \preceq Ax - b\preceq t$
with variables $x\in \R^n$ and $t\in \R$

Penalty function approximation

In $l_p$ -norm approximation, for $1\le p<\infty$ , the objective is

(|r_1|^p + \cdots + |r_m|^p)^{1/p}

As in least-squares problems, we can consider the equivalent problem with objective

|r_1|^p + \cdots + |r_m|^p

which is a separable and symmetric function of the residuals. In particular, the objective only depends on the amplitude distribution of the residuals (i.e. the residuals in sorted order)

We will consider a useful generalization of the $l_p$ -norm approximation problem that only depends on the amplitude distribution of the residuals.

The penalty function approximation problem has the form

\text{minimize }\phi(r_1) + \cdots + \phi(r_m)

subject to

r = Ax - b

where $\phi:\R\to \R$ is called the residual penalty function.

We assume that

\phi

is convex, so the penalty function approximation problem is a convex optimization problem.

By choosing

x

r

is determined. Moreover, the feasible set of

r

is an affine set.

By using penalty functions, we can give more weight to a specific component.

We can view the penalty function approximation problem as a multi-criterion optimization problem.

Examples

quadratic
$\phi(u) = u^2$

It is the most typical penalty function because we already know the analytical solution of it.

Unlike the L1 norm, the penalty function value increases faster as the residual grows larger.

deadzone linear with width $a$
$\phi(u) = \text{max\{0, |u|\;-\;a\}}$

Neglect the residual if its values is less than $a$

log-barrier with limit $a$
$\phi(u) = \begin{cases}-a^2\log(1-(u/a)^2 & |u|<a \\ \infty & \text{otherwise}\end{cases}$

It goes to infinity if a residual located in the outside of the limit.

If the residual is small, the penalty function behaves similarly to a quadratic function. However, if the residual is large, its value will increase much more rapidly than that of a quadratic function.

We take a matrix $A\in \R^{100\times 30}$ and vector $b\in \R^{100}$ , and compute the $l_1$ -norm and $l_2$ -norm approximate solutions of $Ax \approx b$ , as well as the penalty function approximations with a dead zone linear penalty with $a = 0.5$ and log barrier penalty with $a = 1$ . The following figure shows the four associated penalty functions and the amplitude distributions of the optimal residuals for these four penalty approximations.

Histogram of optimal point of r with respect to 4 different penalty functions

Several features can be derived from the amplitude distributions

For the $l_1$ -optimal solution, many residuals are either zero or very small. The $l_1$ -optimal solution also has relatively larger residuals than the others.

Compared to $l_2$ -norm, this converges to zero for relatively small residuals due to the higher penalty imposed when the residuals are small.

The $l_2$ -norm approximation has many modest residuals, and relatively few larger ones.

When the residual is small, the value of the penalty value is already small enough so that we can quit the procedure.

For the dead zone linear penalty, we see that many residuals have the value $\pm 0.5$ , right at the edge of the free zone.

For the log barrier penalty, we see that no residuals have a magnitude larger than 1, but otherwise the residual distribution is similar to the residual distribution for $l_2$ -norm approximation.

Penalty function approximation with sensitivity to outliers

In the estimation or regression context, an outlier is a measurement $y_i = a_i^Tx + v_i$ for which the noise $v_i$ is relatively large. This is often associated with faculty data or a flawed measurement. When outliers occur, any estimate of $x$ will be associated with a residual vector with some large components.

Ideally, we would like to guess which measurements are outliers, and either remove them from the estimation process or greatly lower their weight in forming the estimate. This could be accomplished using penalty function approximation such as

\phi(u) = \begin{cases}u^2 & |u|\le M \\ M^2 & \text{otherwise}\end{cases}

This penalty function agrees with least-squares for any residual smaller than $M$ , but puts a fixed weight on any residual larger than $M$ , no matter how much larger it is.

By doing so, we can alleviate the impact of the outlier.

The problem is that, like we can see above, it is not a convex function. The sensitivity of a penalty function depends on the value of the penalty function for large residuals. If we restrict ourselves to convex penalty functions, the ones that are least sensitive are those for which $\phi(u)$ grows linearly (like $l_1$ -norm).

Penalty functions with this property are sometimes called robust, since the associated penalty function approximation methods are much less sensitive to outliers than least-squares.

One obvious example of a robust penalty function is $\phi(u) = |u|$ . Another example is the robust least-squares or Huber penalty function given by

\phi_{\text{hub}}(u) = \begin{cases}u^2 & |u| \le M \\ M(2|u|-M) & \text{otherwise}\end{cases}

Actually,

M(2|u|-M)

is a tangent line of the quadratic function at

u = M

and

-M

It can be interpreted as a mixture of

l_1

-norm and

l_2

-norm.

Since

l_1

-norm approximation is among the convex penalty function approximation methods that are most robust to outliers. So it is sometimes called robust estimation or robust regression

Least norm problems

The basic least-norm problem has the form

\text{minimize }\|x\|

subject to

Ax = b

where the data are $A\in \R^{m\times n}$ with $m\le n$ and $b\in \R^m$ , the variable is $x\in \R^n$ , and $\|\cdot \|$ is a norm on $\R^n$ . Assume that the rows of $A$ are independent.

The least-norm problem is a convex optimization problem

Geometric interpretation
$x^*$ is a point in affine set $\{x |\; Ax = b\}$ with minimum distance to 0

Estimation interpretation
$b = Ax$ are perfect measurement of $x$ . $x^*$ is the smallest estimate consistent with measurements.

Assume we don’t have enough measurement to identify a parameter perfectly (the nullity is not zero), but measurements are perfect.

Example

Least-squares solution of linear equations ( $\|\cdot \|_2$ )
By squaring the objective we obtain the equivalent problem
$\text{minimize }\|x\|_2^2$
subject to
$Ax = b$
Like the least-squares approximation problem, this problem can be solved analytically. By introducing the dual variable $\nu\in \R^m$ , the optimality conditions are
$2x^* + A^T\nu^* = 0 \\ Ax ^* = b$
Then
$\nu^* = -2(AA^T)^{-1}b \\ x^* = A^T(AA^T)^{-1}b$

Since $\text{rank }A = m < n$ , the matrix $AA^T$ is invertible

Sparse solutions via least $l_1$ -norm
$l_1$ -norm approximation gives relatively large weight to small residuals so that it produces a solution $x$ with a large number of components equal to zero.

$l_1$ -norm problem tens to produce sparse solutions of $Ax = b$

Least penalty problem
$\text{minimize }\phi(x_1) + \cdots + \phi(x_n)$
subject to
$Ax = b$
where $\phi:\R\to \R$ is a convex penalty function.

Regularized approximation

The goal is to find a vector $x$ that is small, and also makes the residual $Ax - b$ small. This is naturally described as a convex vector optimization problem with two objectives.

\text{minimize (w.r.t }R_+^2) \; (\|Ax-b\|, \|x\|)

where $A\in \R^{m\times n}$ and norms on $\R^m$ and $\R^n$ can be different.

We want to find a good approximation

Ax \approx b

with small

x

Regularization

Regularization is a common scalarization method used to solve the bi-criterion problem. One form of regularization is to minimize the weighted sum of the objectives

\text{minimize }\|Ax -b \| + \gamma \|x\|

where $\gamma >0$ is a problem parameter.

Another common method of regularization is to minimize the weighted sum of squared norms

\text{minimize }\|Ax - b\|^2 + \delta \|x\|^2

The most common form of regularization is

\text{minimize }\|Ax-b\|_2^2 + \delta \|x\|_2^2

It is called Tikhonov regularization. It has the analytical solution

x = (A^TA+\delta I)^{-1}A^Tb

Signal reconstruction

In reconstruction problems, we start with a signal represented by a vector $x\in \R^n$ . The coefficients $x_i$ correspond to the value of some function of time, evaluated at evenly spaced points.

It is usually assumed that the signal doesn’t vary too rapidly. (i.e. We have

x_i\approx x_{i + 1}

)

The signal $x$ is corrupted by an additive noise $v$

x_{cor} = x + v

The goal is to form an estimate $\hat x$ of the original signal $x$ , given the corrupted signal $x_{cor}$ . This process is called signal reconstruction.

It is related to the denoising process. Moreover, most reconstruction methods end up performing some sort of smoothing operation on

x_{cor}

to produce

\hat x

, so it is also called smoothing

One simple formulation of the reconstruction problem is the bi-criterion problem

\text{minimize (w.r.t }\R_{+}^2) \; (\|\hat x - x_{cor}\|, \phi(\hat x))

where $\phi:\R^n\to \R$ is convex, and it is called regularization function or smoothing objective.

It is meant to measure the roughness, or lack or smoothness, of the estimate

\hat x

Example : Quadratic smoothing

The simplest reconstruction method uses the quadratic smoothing function

\phi_{quad}(x) = \sum_{i = 1}^{n - 1}(x_{i + 1} - x_i)^2 =\|Dx\|_2^2

where $D\in \R^{(n - 1)\times n}$ is the bidiagonal matrix.

D = \begin{bmatrix}-1 & 1 & 0 & \cdots & 0 & 0 & 0 \\ 0 & -1 & 1 & \cdots & 0 & 0 & 0 \\ \vdots & \vdots & \vdots & & \vdots & \vdots & \vdots \\ 0 & 0 & 0 & \cdots & -1 & 1 & 0 \\ 0 & 0 & 0 & \cdots & 0 & -1 & 1\end{bmatrix}

We can obtain the optimal trade-off between $\|\hat x - x_{cor}\|_2$ and $\|D\hat x\|_2$ by minimizing

\|\hat x - x_{cor}\|_2^2 + \delta \|D\hat x\|_2^2

where $\delta >0$ parameters the optimal trade-off curve.

Example : Total variation reconstruction

Simple quadratic smoothing works well as a reconstruction method when the original signal is very smooth, and the noise is rapidly varying. But any rapid variations in the original signal will be removed by quadratic smoothing.

Total variation reconstruction method can remove much of the noise, while still preserving rapid variations in the original signal. The method is based on the smoothing funciton

\phi_{tv}(\hat x) = \sum_{i = 1}^{n - 1}|\hat x_{i + 1}- \hat x_i| = \|D\hat x\|_1

Similar to

l_1

-norm, it gives less penalty for rapid variation than the quadratic one.

Therefore, we have to choose the smoothing objective carefully regarding of the characteristic of the noise and signal.

quadratic smoothing smooths out noise and sharp transition signal

total variation smoothing preserves sharp transitions in signal because

l_1

-norm has less sensitive for the large value than

l_2

-norm.

Robust approximation

We consider an approximation problem with basic objective $\|Ax - b\|$ , but also wish to take into account some uncertainty or possible variation in the data matrix $A$ . There are two approches to solve this problem.

stochastic : assume $A$ is random and minimize $\mathbb E (\|Ax-b\|)$

worst-case : set $\mathcal A$ of possible values of $A$ and minimize $\sup_{A\in \mathcal A}\|Ax - b\|$

Stochastic robust approximation

We assume that $A$ is a random variable taking values in $\R^{m\times n}$ , with mean $\bar A$ . Then

A= \bar A + U

where $U$ is a random matrix with zero mean.

It is natural to use the expected value of $\|Ax - b\|$ as the objective

\text{minimize }\mathbb E(\|Ax - b\|)

We refer to this problem as the stochastic robust approximation problem.

It is always a convex optimization problem, but usually not tractable.

Worst-case robust approximation

It is also possible to model the variation in the matrix $A$ using a worst case approach. We describe the uncertainty by a set of possible values for $A$

A\in \mathcal A\sub \R^{m\times n}

which we assume is non-empty and bounded. We define the associated worst-case error of a candidate approximate solution $x\in \R^n$ as

e_{wc}(x) = \sup_{A\in \mathcal A}\|Ax - b\|

which is always a convex function of $x$ .

It is a convex function because it is just a point-wise supremum which is always convex.

The worst-case robust approximation problem is to minimize the worst case error

\text{minimize }e_{wc}(x) = \sup_{A\in \mathcal A}\|Ax - b\|

where the variable is $x$ , and the problem data are $b$ and the set $\mathcal A$ .

When

\mathcal A

is the singleton, the robust approximation problem reduces to the basic norm approximation problem.

It is always a convex optimization problem, but its tractability depends on the norm used and the description of uncertainty of $\mathcal A$ .

Example

To illustrate the difference between the stochastic and worst-case formulations for the robust approximation problem, we consider the least squares problem

\text{minimize }\|A(u)x - b\|_2^2

where $u\in \R$ is an uncertain parameter and $A(u) = A_0 + uA_1$ . We consider a specific instance of the problem, with $A(u)\in \R^{20\times 10}$ , $\|A_0\| = 10, \|A_1\| = 1$ , and $u$ in the interval $[-1, 1]$ .

We find three approximate solutions

Nominal optimal : The optimal solution $x_{nom}$ is found, which minimize $\|A_0x - b\|_2^2$

Stochastic robust approximation : We find $x_{stoch}$ , which minimizes $\mathbb E(\|A(u) x - b\|_2^2$ , assuming the parameter $u$ is uniformly distributed on $[-1, 1]$

Worst-case robust approximation. We find $x_{wc}$ , which minimizes
$\sup_{-1\le u\le 1}\|A(u)x - b\|_2$

Example : stochastic robust Least squares

Consider the stochastic robust least-squares problem

\text{minimize }\|Ax - b\|_2^2

where $A = \bar A + U$ , $U$ is a random matrix with zero mean.

We can express the objective as

\begin{aligned}\mathbb E(\|Ax - b\|_2^2) &= \mathbb E\bigg((\bar Ax -b+Ux)^T(\bar Ax -b+Ux)\bigg) \\ &= (\bar A x - b)^T(\bar Ax-b) + \mathbb E(x^TU^TUx) \\ &=\|\bar Ax-b\|_2^2 + x^TPx\end{aligned}

where $P = \mathbb E(U^TU)$

Therefore, the stochastic robust approximation problem has the form of regularized least-squares problem

\text{minimize }\|\bar Ax - b\|_2^2 + \|P^{1/2}x\|_2^2

with solution

x= (\bar A^T\bar A+P)^{-1}\bar A^Tb

When the matrix $A$ is subject to variation, the vector $Ax$ will have more variation the laarger $x$ is, and Jensen’s inequality tells us that variation in $Ax$ will increase the average value of $\|Ax - b\|_2$ . So we need to balance making $\bar Ax -b$ small with the desire for a small $x$ to keep the variation in $Ax$ small.

For $P = \delta I$ , we can get Tikhonov regularized least squares problem

\text{minimize }\|\bar Ax - b\|_2^2 + \delta \|x\|_2^2

Therefore, regularized least squares problem can be interpreted as a stochastic concept and vice versa.

Example : worst case robust least squares

Let

\mathcal A = \{\bar A + u_1 A_1 + \cdots + u_pA_p|\; \|u\|_2 \le 1\}

Consider the worst case robust least-squares problem

\text{minimize }\sup_{A\in \mathcal A}\|Ax - b\|_2^2 = \sup_{\|u\|_2\le 1} \|P(x)u + q(x)\|_2^2

where $P(x) = \begin{bmatrix}A_1x & A_2x & \cdots & A_px\end{bmatrix}, q(x) = \bar Ax -b$

Note that the strong duality holds between the following problems

Primal problem
$\text{maximize }\|Pu+ q\|_2^2$
subject to
$\|u\|_2^2\le 1$

Intuitively, we can solve this problem by finding a maximum singular value.

Dual problem
$\text{minimize }t+ \lambda$
subject to
$\begin{bmatrix}I & P & q \\P^T & \lambda I & 0 \\ q^T & 0 & t\end{bmatrix}\succeq 0$

It is a very specially case of satisfying the strong duality condition when the problem is not a convex.

Therefore, the Lagrange dual of this problem can be expressed as the SDP

\text{minimize }t+ \lambda

subject to

\begin{bmatrix}I & P(x) & q(x) \\P(x)^T & \lambda I & 0 \\ q(x)^T & 0 & t\end{bmatrix}\succeq 0

with variables $t, \lambda\in \R$ .

For fixed $x$ , we can compute $\sup_{\|u\|_2\le 1} \|P(x)u + q(x)\|_2^2$ by solving the SDP with variables $t$ and $\lambda$ . In other words, optimizing jointly over $t, \lambda$ , and $x$ is equivalent to minimizing worst case error $e_{wc}(x)^2$ .

Therefore, the robust least squares problem is equivalent to the SDP with

x, \lambda , t

as variables.

5. Duality

비룡 — Tue, 13 Feb 2024 01:33:45 +0900

Lagrangian

The basic idea in Lagrangian duality is to take the constraints into account by augmenting the objective function with a weighted sum of the constraint functions.

By using the Lagrangian, we can get a hint to solve the original problem which is not convex nor easy to solve.

Standard form problem (not necessary convex)

\text{minimize }f_0(x)

subject to

f_i(x) \le 0\quad i = 1, \dots, m\\ h_i(x) = 0\quad i = 1, \dots, p

where $x\in \R^n$ , domain $\mathcal D$ , optimal value $p^*$

Lagrangian

\mathcal L(x, \lambda, \nu) = f_0(x) + \sum_{i = 1}^m \lambda_i f_i(x) + \sum_{i = 1}^p \nu_i h_i(x)

where $\mathcal L: \R^n\times \R^m \times \R^p \to \R$ with $\text{dom }L = \mathcal D \times \R^m \times \R^p$

We call

\lambda, \nu

as a dual variable or Lagrange multipliers

Lagrange dual function

Lagrange dual function

\begin{aligned}g(\lambda, \nu) &= \inf_{x\in \mathcal D} \mathcal L(x, \lambda, \nu) \\ &= \inf_{x\in \mathcal D}\bigg(f_0(x) + \sum_{i = 1}^m\lambda_if_i(x) + \sum_{i = 1}^p \nu_ih_i(x)\bigg) \end{aligned}

where $g:\R^m\times \R^p \to \R$

Since the dual function is the point-wise infimum of a family of affine functions of $(\lambda,\mu)$ , it is a concave function. This is because we can view this problem as a point-wise supremum problem by adding a negative term to our objective function. Even though we add a negative term, it is still affine on $(\lambda, \mu)$ (i.e it satisfies the convexity on $(\lambda, \mu)$ for fixed $x$ )

When the Lagrangian is unbounded below in

x

, the dual function takes on the value

-\infty

Proof
We already know that convexity preserving operations.
If $f(x, y)$ is convex in $x$ for every $y\in \mathcal A$ , then
$g(x) = \sup_{y\in \mathcal A}f(x, y)$
is convex
Since $\mathcal L(x, \lambda, \nu)$ is concave in $(\lambda, \nu)$ for every $x\in \mathcal D$ , then
$\inf_{x\in D}\mathcal L(x, \lambda, \nu)$
is concave.

lower bound property

If $\lambda \succeq 0$ , then $g(\lambda, \nu)\le p^*$

Proof
If $\bar x$ is feasible and $\lambda \succeq0$ , then
$f_0(\bar x) \ge \mathcal L(\bar x, \lambda , \nu) \ge \inf_{x\in \mathcal D}\mathcal L(x, \lambda, \nu) = g(\lambda, \nu)$
minimizing over all feasible $\bar x$ gives $p^*\ge g(\lambda, \nu)$

Example

Least-norm solution of linear equations

\text{minimize }x^Tx

subject to

Ax = b

Dual function

Lagrangian is $\mathcal L(x, \nu) = x^Tx + \nu^T(Ax - b)$

to minimize $\mathcal L$ over $x$ , set gradient equal to zero
$\begin{aligned}\nabla_x\mathcal L(x, \nu) &=2x + A^T\nu \\&= 0\end{aligned}$
Therefore,
$x = - \frac{1}{2}A^T\nu$

plug in to $\mathcal L$ to obtain $g$
$g(\nu) = \mathcal L((-1/2)A^T\nu, \nu) = -\frac{1}{4}v^TAA^T\nu -b^T\nu$
a concave function of $\nu$

lower bound property
$p^* \ge -(1/4)\nu^TAA^T\nu - b^T\nu$
for all $\nu$

It might seem useless at a first glance, but it is really useful. Let’s say we have a million equality constraints above. In this case, it is impossible to use analytical solution directly. Instead, we have to use iterative approach. Then we have to decide when should we stop. In this situation, we can use the lower bound.

Standard form LP

\text{minimize }c^Tx

subject to

Ax = b \quad x\succeq0

The Lagrangian is

\begin{aligned}\mathcal L(x, \lambda, \nu) &= c^Tx + \nu^T(Ax - b) - \lambda^Tx \\ &= -b^T\nu + (c+A^T\nu-\lambda)^Tx\end{aligned}

since it is affine in $x$ ,

g(\lambda,\nu) = \inf_x \mathcal L(x, \lambda, \nu) = \begin{cases}-b^T\nu & A^T\nu-\lambda + c = 0 \\ -\infty &\text{otherwise}\end{cases}

Therefore $p^*\ge -b^T\nu$ if $A^T\nu + c\succeq 0$

Equality constrained norm minimization

\text{minimize }\|x\|

subject to

Ax = b

The dual function is

g(\nu) = \inf_x (\|x\|-\nu^T(Ax) + b^T\nu) = \begin{cases}b^T\nu & \|A^T\nu\|_* \le 1 \\ -\infty & \text{otherwise}\end{cases}

where $\|v\|_* = \sup_{\|u\|\le 1 u^Tv}$ is dual norm of $\|\cdot \|$

Dual norm intuition

The dual of a norm $\|\cdot \|$ is defined as: $$\|z\|_* = \sup \{ z^Tx \text{ } | \text{ } \|x\| \le 1\}$$ Could anybody give me an intuition of this concept? I know the definition, I am using i...

https://math.stackexchange.com/questions/903484/dual-norm-intuition

Therefore $p^* \ge b^T\nu$ if $\|A^T\nu\|_* \le 1$

Two-way partitioning

\text{minimize }x^TWx

subject to

x_i^2 = 1\quad i = 1, \dots, n

We can express the objective as

\sum x_ix_jW_{ij}

Therefore, we can interpret $W_{ij}$ as a measure of hostility since if $W_{ij}$ is large, we have to allocate the opposite sign to $x_i$ and $x_j$ respectively.

The dual function is

\begin{aligned}g(\nu) &= \inf_x (x^TWx + \sum_i \nu_i(x_i^2 - 1)) \\ &= \inf_x x^T(W + \text{diag}(v))x - 1^T\nu \\ &= \begin{cases}-1^T\nu & W + \text{diag}(\nu)\succeq 0 \\ -\infty & \text{otherwise}\end{cases}\end{aligned}

Therefore $p^* \ge -1^T\nu$ if $W + \text{diag}(v) \succeq0$

Lagrange dual and conjugate function

\text{minimize }f_0(x)

subject to

Ax\preceq b \\ Cx = d

The dual function is

\begin{aligned}g(\lambda, \nu) &= \inf_{x\in \text{dom }f_0}(f_0(x) + (A^T\lambda + C^T\nu)^Tx - b^T\lambda -d^T\nu) \\ &= -f_0^*(-A^T\lambda - C^T\nu) - b^T\lambda - d^T\nu\end{aligned}

where $f^*(y) = \sup_{x\in \text{dom }f}(y^Tx - f(x))$

How to physically interpret conjugate functions?

If given a convex function $f: \mathbb{R} \to \mathbb{R}$, then the conjugate function $f^*$ is defined as $$f^*(s) = \sup_{t \in \mathbb{R}} (st-f(t))$$ Now i want to understand what is the physi...

https://math.stackexchange.com/questions/2225932/how-to-physically-interpret-conjugate-functions

Example : entropy maximization

f_0(x) = \sum_{i = 1}^n x_i \log x_i \\ f_0^*(y) = \sum_{i = 1}^n e^{y_i - 1}

In this case, we already know the conjugate of

f_0

The Lagrange dual problem

For each pair $(\lambda, \nu)$ with $\lambda \succeq0$ , the Lagrange dual function gives us a lower bound on the optimal value $p^*$ of the optimization problem. Thus we have a lower bound that depends on some parameters $\lambda, \nu$ .

The natural question is: What is the best lower bound that can be obtained from the Lagrange dual function?

This leads to the optimization problem

\text{maximize }g(\lambda, \nu)

subject to

\lambda \succeq 0

This problem is called the Lagrange dual problem. The term dual feasible, to describe a pair $(\lambda, \nu)$ with $\lambda \succeq 0$ and $g(\lambda ,\nu)>-\infty$ .

Note that the Lagrange dual problem is a convex optimization problem that doesn’t depend on the convexity of the original problem since the objective function to be maximized is concave and the constraint is convex

Example

standard form LP and its dual

standard form
$\text{minimize }c^Tx$
subject to
$Ax = b\\x\succeq 0$

dual problem
$\text{maximize }-b^T\nu$
subject to
$A^T\nu + c\succeq 0$

Weak and strong duality

weak duality : $d^* \le p^*$ (where $d^*$ is the optimal value of the dual problem)
- it always hold for convex and non-convex problems.
- it can be used to find non-trivial lower bounds for difficult problem

strong duality : $d^* = p^*$
- it does not hold in general
- (usually) holds for convex problems
- conditions that guarantee strong duality in convex problems are called constraint qualifications

Slater’s constraint qualification

One simple constraint qualification is Slater's condition

Let our problem is as follows

\text{minimize }f_0(x)

subject to

f_i(x) \le 0 \quad i = 1, \dots, m\\ Ax = b

where $f_0, \dots, f_m$ is a convex function.

If there exists an $x\in \text{relint }\mathcal D$ such that

f_i(x) < 0 \quad i = 1,\dots, m \\ Ax = b

Such a point is sometimes called strictly feasible

Slater’s theorem states that strong duality holds if Slater’s condition holds and the problem is convex.

Slater’s condition must require the convexity of the problem

Example

Inequality from LP

Primal problem
$\text{minimize }c^Tx$
subject to
$Ax\preceq b$

dual function
$g(\lambda ) = \inf_x((c+A^T\lambda)^Tx - b^T\lambda) = \begin{cases}-b^T\lambda & A^T\lambda + c = 0 \\ -\infty & \text{otherwise}\end{cases}$

dual problem
$\text{maximize }-b^T\lambda$
subject to
$A^T\lambda + c = 0 \\ \lambda \succeq 0$

Formally, it is not a dual problem. It is just a problem that equivalent to the dual problem.

From the Slater’s condition, $p^* = d^*$ if $A\bar x \prec b$ for some $\bar x$ .

In fact, $p^* = d^*$ except when primal and dual are infeasible.

The strong duality holds if polyhedron has non-empty interior.

Quadratic program

Primal problem (assume $P\in S_{++}^n$ )
$\text{minimize }x^TPx$
subject to
$Ax\preceq b$

dual function
$g(\lambda ) = \inf_x (x^TPx + \lambda ^T(Ax - b)) = -\frac{1}{4}\lambda ^TAP^{-1}A^T\lambda - b^T\lambda$

Since we already know that $P$ is positive definite, we can use the first optimality condition to find the minimum point.

dual problem
$\text{maximize }-\frac{1}{4}\lambda ^TAP^{-1}A^T\lambda - b^T\lambda$
subject to
$\lambda \succeq 0$

From the Slater’s condition, $p^* = d^*$ if $A\bar x \prec b$ for some $\bar x$

In fact,

p^* = d^*

always holds.

Geometric interpretation

for simplicity, consider problem with one constraint $f_1(x) \le 0$

g(\lambda) = \inf_{(u, t)\in \mathcal G}(t + \lambda u)

where $\mathcal G = \{(f_1(x), f_0(x)) | \; x\in \mathcal D\}$

$\lambda u + t = g(\lambda )$ is supporting hyperplane to $\mathcal G$

hyperplane intersects $t$ -axis at $t = g(\lambda)$

Complementary slackness

Suppose that the primal and dual optimal values are attained and equal. Let $x^*$ be a primal optimal and $(\lambda^*, \nu^*)$ is dual optimal

\begin{aligned}f_0(x^*) = g(\lambda^*, \nu^*) &= \inf_x \bigg(f_0(x) + \sum_{i = 1}^m \lambda_i^*f_i(x) + \sum_{i = 1}^pv_i^*h_i(x)\bigg) \\ &\le f_0(x^*) + \sum_{i = 1}^m \lambda_i^*f_i(x^*) + \sum_{i = 1}^p\nu_i^*h_i(x^*) \\ &\le f_0(x^*)\end{aligned}

Therefore, the two inequalities holds with equality.

So, $x^*$ minimizes $\mathcal L(x, \lambda^*, \nu^*)$ and

\lambda_i^*f_i(x^*) = 0 \quad i = 1, \dots, m

More specifically,

\lambda_i^* >0 \Rightarrow f_i(x^*) = 0 \\ f_i(x^*) < 0 \Rightarrow \lambda_i^* = 0

It is known as complementary slackness

Karush-Kuhn-Tucker (KKT) conditions

We now assume that the functions $f_i, h_i$ are differentiable and therefore have open domains.

Let $x^*$ and $(\lambda^*, \nu^*)$ be any primal and dual optimal points with zero duality gap. Since $x^*$ minimizes $\mathcal L(x, \lambda^*, \nu^*)$ over $x$ , it follows that its gradient must vanish at $x^*$

\nabla f_0(x^*) + \sum_{i = 1}^m \lambda_i^* \nabla f_i(x^*) + \sum_{i = 1}^p \nu_i^*\nabla h_i(x^*) = 0

Thus the KKT conditions is as follows

primal constraints
$f_i(x^*) \le 0 \quad i = 1, \dots,m \\ h_i(x^*) = 0\quad i = 1, \dots , m$

dual constraints
$\lambda^* \succeq 0$

Note that there is no constraint on $\nu$ .

complementary slackness
$\lambda_if_i(x^*) = 0\quad i =1, \dots, m$

stationary
$\nabla f_0(x^*) + \sum_{i = 1}^m \lambda_i^* \nabla f_i(x^*) + \sum_{i = 1}^p \nu_i^*\nabla h_i(x^*) = 0$

if strong duality holds and $x^*, \lambda^*, \nu^*$ are optimal, then they must satisfy the KKT conditions.

In other words, if strong duality holds, KKT conditions hold for any pair of primal optimal and dual optimal

Note that it is possible that even though

x, \lambda, \nu

satisfy the KKT conditions, we can't guarantee that it is an optimal point for primal and dual problems respectively.

KKT conditions for convex problems

When the primal problem is convex, the KKT conditions are also sufficient for the points to be primal and dual optimal. In other words, if $f_i$ are convex and $h_i$ are affine, and $\tilde x, \tilde \lambda, \tilde \nu$ are any points that satisfy the KKT conditions, then $\tilde x$ and $(\tilde \lambda, \tilde \nu)$ are primal and dual optimal, with zero duality gap.

Since $\tilde \lambda_i \ge 0$ , $\mathcal L(x, \tilde \lambda, \tilde \nu)$ is convex in $x$ . Moreover, from the complementary slackness condition, $\tilde x$ minimizes $\mathcal L(x, \tilde \lambda, \tilde \nu)$ over $x$ .

Therefore,

\begin{aligned}g(\tilde \lambda, \tilde \nu) &= \mathcal L(\tilde x, \tilde \lambda, \tilde \nu) \\ &= f_0(\tilde x) + \sum_{i = 1}^m \tilde \lambda_i f_i(\tilde x) + \sum_{i = 1}^p \tilde \nu_ih_i(\tilde x) \\ &= f_0(\tilde x)\end{aligned}

This shows that $\tilde x$ and $(\tilde \lambda, \tilde \nu)$ have zero duality gap, and therefore are primal and dual optimal.

For any convex optimization problem with differentiable objective functions, any points that satisfy the KKT conditions are primal and dual optimal, and have zero duality gap

For convex problems, primal and dual optimality satisfy if and only if KKT condition holds

But we can not guarantee that we can always find

(\tilde x, \tilde \lambda, \tilde \nu)

that satisfy the KKT conditions. We can just say that if we find that point, it is primal and dual optimal respectively.

KKT conditions for convex problems with Slater’s condition

If a convex optimization problem with differentiable objective and constraint functions satisfies Slater’s condition, then the KKT conditions provide necessary and sufficient conditions for optimality. Since Slater’s condition implies that the optimal duality gap is zero and the dual optimum is attained, so if $x$ is optimal we can always find $(\lambda, \nu)$ that satisfy the KKT conditions.

Of course, if we find

(x, \lambda, \nu)

satisfy the KKT conditions, we can guarantee that

x

is an optimal point in a primal problem and

(\lambda, \nu)

is an optimal point in a dual problem.

Example : Water-filling

Assume $\alpha_i > 0$ ( $\alpha_i$ is a given value)

\text{minimize }-\sum_{i = 1}^n \log (x_i + \alpha_i)

subject to

x\succeq 0 \\ 1^Tx = 1

Note that it is a convex optimization problem that satisfies the Slater’s condition

if $x$ is optimal, there exist $\lambda \in \R^n, \nu \in \R$ such that

primal feasibility
$x\succeq 0 \\ 1^Tx = 1$

dual feasibility
$\lambda \succeq 0$

complementary slackness
$\lambda_i x_i = 0 \quad i = 1, \dots, n$

stationary condition
$\begin{aligned}\nabla_x \mathcal L(x, \lambda, \nu) &= \nabla_x \bigg(-\sum_{i = 1}^n\log(x_i + \alpha_i) - \lambda^Tx + \nu^T(1^Tx - 1)\bigg) \\ &= \begin{bmatrix} - \frac{1}{x_1 + \alpha_1} - \lambda_1 + \nu\\ \vdots \\ - \frac{1}{x_n + \alpha_n} - \lambda_n + \nu\end{bmatrix} \\ & = 0\end{aligned}$
Therefore,
$- \frac{1}{x_i + \alpha_i} - \lambda_i + \nu = 0 \quad i = 1, \dots, n\\ \Rightarrow \nu = \frac{1}{x_i + \alpha_i} +\lambda_i \quad i = 1, \dots, n$

By using these conditions,

if $\nu < 1/\alpha_i$
$\lambda_i = 0 \text{ and }x_i = 1/\nu - \alpha_i$

if $\nu \ge 1/\alpha_i$
$\lambda_i = \nu - 1/\alpha_i \text{ and }x_i = 0$

Therefore, we can express $x$ as a

1^Tx = \sum_{i = 1}^n \max\{0, 1/\nu - \alpha_i\} = 1

Therefore, we can determine $\nu$ .

Perturbation and sensitivity analysis

When strong duality holds, the optimal dual variables give very useful information about the sensitivity of the optimal value with respect to perturbations of the constraints.

For some problems, it is easier to solve a dual problem rather than a primal problem. In this case, we can use the optimal dual variable to get information about the primal problem.

Unperturbed optimization problem and its dual

Primal problem
$\text{minimize }f_0(x)$
subject to
$f_i(x) \le 0 \quad i = 1, \dots, m \\ h_i (x) = 0 \quad i = 1, \dots, p$

Dual problem
$\text{maximize }g(\lambda, \nu)$
subject to
$\lambda \succeq 0$

Perturbed problem and its dual

Primal problem
$\text{minimize }f_0(x)$
subject to
$f_i(x) \le u_i \quad i = 1, \dots, m \\ h_i(x) = v_i \quad i = 1, \dots, p$

Dual problem
$\text{maximize }g(\lambda, \nu) - u^T\lambda -v^T\nu$
such that
$\lambda \succeq 0$

It increases the feasible set

where $x$ is a primal variable, $u, v$ are parameters

We define $p^*(u, v)$ as the optimal value of the perturbed problem

p^*(u, v) = \inf\{f_0(x) | \exist x \in \mathcal D, f_i(x) \le u_i, h_i(x) = v_i\}

When the original problem is convex, the function $p^*$ is a convex function of $u$ and $v$

Whether

p^*

is convex or not is benign on the convexity of the original function because

x

is affected by

u

and

v

Global sensitivity result

Assume that strong duality holds and that the dual optimum is attained. (This is the case if the original problem is convex, and Slater’s condition is satisfied)

Let $(\lambda^*, \nu^*)$ be optimal for the dual of the unperturbed problem. Then for all $u$ and $v$ we have

\begin{aligned}p^*(u, v) &\ge g(\lambda^*, \nu^*) - u^T\lambda^* - v^T\nu^* &\\&= p^*(0, 0) - u^T\lambda^* - v^T\nu^*\end{aligned}

The first inequality came from the fact that the dual function value of the perturbed problem is a lower bound of its primal.

We conclude that for any $x$ feasible for the perturbed problem, we have

f_0(x) \ge p^*(0, 0) - u^T\lambda^* - v^T\nu^*

Sensitivity interpretation

If $\lambda_i^*$ is large and we tighten $i$ th constraint (i.e. choose $u_i< 0$ ), then the optimal value $p^*(u, v)$ is guaranteed to increase greatly

If $\nu_i^*$ is large and positive and we take $v_i<0$ , or if $\nu_i^*$ is large and negative and we take $v_i>0$ , then the optimal value $p^*(u, v)$ is guaranteed to increase greatly

If $\lambda_i^*$ is small, and we loosen the $i$ th constraint $(u_i > 0)$ , then the optimal value $p^*(u, v)$ will not decrease too much

If $\nu_i^*$ is small and positive, and $v_i>0$ , or if $\nu_i^*$ is small and negative and $v_i <0$ , then the optimal value $p^*(u, v)$ will not decrease too much.

Local sensitivity

Suppose now that $p^*(u, v)$ is differentiable at $u= 0, v = 0$ . Then, provided strong duality holds, the optimal dual variables $\lambda^*, \nu^*$ are related to the gradient of $p^*$ at $u = 0, v= 0$

\lambda_i^* = - \frac{\partial p^*(0, 0)}{\partial u_i} \\ \nu_i^* = - \frac{\partial p^*(0, 0)}{\partial v_i}

Thus, when $p^*(u, v)$ is differentiable at $u = 0, v= 0$ , and strong duality holds, the optimal Lagrange multipliers are exactly the local sensitivities of the optimal value with respect to constraint perturbations.

Proof
Since strong duality holds and the dual optimal $(\lambda^*, \nu^*)$ is attained,
$p^*(u, v) \ge p^*(0, 0) - u^T\lambda^* - v^T\nu^*$
for all $u, v$
Take $u = te_i$ and $v = 0$ , then
$p^*(te_i, 0) \ge p^*(0, 0) - t\lambda_i^*$
If $t \ge 0$ ,
$\frac{p^*(te_i, 0) - p^*(0, 0)}{t} \ge - \lambda_i^* \\ \Rightarrow \lim_{t\to 0+} \frac{p^*(te_i, 0) - p^*(0, 0)}{t} \ge - \lambda_i^* \\ \Rightarrow \frac{\partial p^*(0, 0)}{\partial u_i}\ge -\lambda_i^*$
Similarly, if $t\le 0$ ,
$\frac{\partial p^*(0, 0)}{\partial u_i}\le -\lambda_i^*$
Therefore,
$\frac{\partial p^*(0, 0)}{\partial u_i}= -\lambda_i^*$
The same method can be used to establish
$\frac{\partial p^*(0, 0)}{\partial v_i}=-\nu_i^*$

The local sensitivity result gives us a quantitative measure of how active a constraint is at the optimum $x^*$ . If $f_i(x^*) < 0$ , then the constraint is inactive, and it follows that the constraint can be tightened or loosened a small amount without affecting the optimal value since by complementary slackness, the associated optimal Lagrange multiplier must be zero.

But now suppose that $f_i(x^*)=0$ . The $i$ th optimal Lagrange multiplier tells us how active the constraint is.

If $\lambda_i^*$ is small : the constraint can be loosened or tightened a bit without much effect on the optimal value

if $\lambda_i^*$ is large : if the constraint is loosened or tightened a bit, the effect on the optimal value will be great

Duality and problem reformulations

Equivalent formulations of a problem can lead to very different duals. Reformulating the primal problem can be useful when the dual is difficult to derive, or uninteresting.

Common reformulations

introduce new variables and equality constraints

make explicit constraints implicit or vice-versa

transform objective or constraint functions
→ replace $f_0(x)$ by $\phi(f_0(x))$ with $\phi$ convex, increasing

Introducing new variables and equality constraints

Consider an unconstrained problem of the form

\text{minimize }f_0(Ax + b)

The dual function is constant

g(\lambda, \nu) = \inf_x \mathcal L(x, \lambda, \nu) = \inf_x f_0(Ax +b ) = p^*

We have strong duality, but dual is quite useless.

Now let’s reformulate the above problem as

\text{minimize }f_0(y)

subject to

Ax + b = y

The dual function of the transformed problem is

\begin{aligned}g(\nu) &= \inf_{x,y} \mathcal L(x, \nu) \\ &=\inf_{x,y} f_0(y) +\nu^T(Ax + b - y) \\ &=\begin{cases}-f^*(\nu) + \nu^Tb & A^T\nu = 0 \\-\infty & \text{otherwise}\end{cases} \end{aligned}

Implicit constraints

Include some of the constraints in the objective function by modifying the objective function to be infinite when the constraint is violated

We consider the linear program

\text{minimize }c^Tx

subject to

Ax = b \\ l\preceq x \preceq u

where $A\in \R^{p\times n}$ and $l \prec u$ .

The dual problem is

\text{maximize } -b^T\nu - \lambda_1^Tu + \lambda_2^Tl

subject to

A^T\nu + \lambda_1 - \lambda_2 +c = 0 \\ \lambda_1 \succeq 0 \\ \lambda_2 \succeq 0

Reformulate the primal problem as

\text{minimize }f_0(x) = \begin{cases}c^Tx & l \preceq x \preceq u \\ \infty & \text{otherwise}\end{cases}

Then the dual function for the reformulated problem is

\begin{aligned}g(\nu) &= \inf_{l \preceq x \preceq u}(c^Tx + \nu^T(Ax-b)) \\ &= -b^T\nu - u^T(A^T\nu+c)^{-} + l^T(A^T\nu + c)^+\end{aligned}

where $y_i^+ = \max\{y_i, 0\}, y_i^- = \max\{-y_i, 0\}$

Therefore the dual problem is the unconstrained problem

\text{maximize }-b^T\nu - u^T(A^T\nu + c)^- + l^T(A^T\nu + c)^+

Generalized inequalities

We can extend the Lagrange duality to a problem with generalized inequality constraints.

\text{minimize }f_0(x)

subject to

f_i(x) \preceq_{K_i}0 \quad i = 1, \dots, m \\ h_i(x) = 0 \quad i = 1, \dots, p

where $K_i\subset \R^{k_i}$ are proper cones.

There is no assumption about the convexity of the problem.

The Lagrange dual

With each generalized inequality $f_i(x) \preceq_{K_i}0$ , we associate a Lagrange multiplier vector $\lambda_i\in \R^{k_i}$ and define the associated Lagrangian as

\mathcal L(x, \lambda_1, \cdots, \lambda_m, \nu) = f_0(x) + \sum_{i = 1}^m \lambda_i^Tf_i(x) + \sum_{i = 1}^p \nu_ih_i(x)

The dual function is defined exactly as in a problem with scalar inequalities

\begin{aligned}g(\lambda_1, \dots, \lambda_m , \nu) &= \inf_{x\in \mathcal D}\mathcal L(x, \lambda_1, \dots, \lambda_m, \nu)\\ &= \inf_{x\in \mathcal D}\bigg(f_0(x) + \sum_{i = 1}^m \lambda_i^Tf_i(x) + \sum_{i = 1}^p \nu_ih_i(x)\bigg)\end{aligned}

Since the Lagrangian is affine in the dual variables $(\lambda, \nu)$ , and the dual function is a point-wise infimum of the Lagrangian, the dual function is concave.

As in a problem with scalar inequalities, the dual function gives lower bounds on $p^*$ . Similar to $\lambda_i \ge 0$ the scalar case, it requires the non-negativity requirement on the dual variables for the generalized inequality case.

\lambda_i\succeq_{K_i^*} 0 \quad i = 1, \dots, m

where $K_i^*$ denotes the dual cone of $K_i$ .

Weak duality follows immediately from the definition of dual cone because if $\lambda_i\succeq_{K_i^*}0$ and $f_i(\tilde x) \preceq_{K_i}0$ , then $\lambda_i^Tf_i(\tilde x) \le 0$ . Therefore for any primal feasible point $\tilde x$ and any $\lambda_i \succeq_{K_i^*}0$ for all $i$ , then

\begin{aligned}f_0(x) &\ge f_0(x) + \sum_{i = 1}^m\lambda_i^Tf_i(x) + \sum_{i = 1}^p \nu_ih_i(x) \\ &\ge \inf_{x\in \mathcal D}\mathcal L(x, \lambda_1, \dots, \lambda_m, \nu) \\ &= g(\lambda_1, \dots, \lambda_m, \nu)\end{aligned}

Therefore,

p^* \ge g(\lambda_1, \dots, \lambda_m, \nu)

The Lagrange dual optimization problem is

\text{maximize }g(\lambda_1, \dots, \lambda_m, \nu)

subject to

\lambda_i \succeq _{K_i}0 \quad i = 1, \dots, m

Slater’s condition and strong duality

Strong duality holds when the primal problem is convex and satisfies an appropriate constraint qualification as the scalar case such as Slater’s condition.

Example : Semidefinite program

We consider a semi-definite program in inequality form

\text{minimize }c^Tx

subject to

x_1 F_1 + \cdots + x_nF_n \preceq G

where $F_i, G\in S^{k}$ . (Here $f_1$ is affine, and $K_1$ is $S_{+}^k$ , the positive semi-definite cone.)

Since $S_{+}^k$ is self-dual, the Lagrange multiplier $Z$ is in $S^k$ . Then the Lagrangian is

\begin{aligned}\mathcal L(x, Z) &= c^Tx + \text{tr}(Z(x_1F_1 + \cdots + x_nF_n - G)) \\ &= x_1(c_1 + \text{tr}(ZF_1)) + \cdots + x_n(c_n + \text{tr}(ZF_n)) - \text{tr}(ZG)\end{aligned}

Therefore, the dual function is

g(Z) = \inf_x \mathcal L(x, Z)=\begin{cases}-\text{tr}(ZG) & c_i + \text{tr}(ZF_i) = 0,\forall i \\-\infty & \text{otherwise}\end{cases}

The dual problem can be expressed as

\text{maximize }-\text{tr}(GZ)

subject to

c_i + \text{tr}(ZF_i) = 0 \quad i= 1, \dots, n \\ Z\succeq0

Note that we use

S_{+}^k

inequality

Strong duality obtains if there is a strictly feasible point because it satisfies Slater’s condition. In other words, there exists an $x$ with

x_1 F_1 + \cdots+ x_nF_n -G \prec 0

Note that an inner product of two symmetric matrices can be defined as a trace of its multiplication. Check the dual generalized inequality in convex set section.

4. Convex optimization problems

비룡 — Tue, 6 Feb 2024 17:30:22 +0900

Optimization problem in standard form

\text{minimize } f_0(x)

subject to

f_i(x) \le 0, \; i = 1, \dots, m \\ h_i(x) = 0, \; i = 1, \dots, p

$x\in \R^n$ is the optimization variable

$f_0:\R^n\to \R$ is the objective or cost function

$f_i:\R^n\to \R, \; i = 1, \dots, m$ are the inequality constraint functions

$h_i:\R^n\to \R, \; i = 1, \dots, p$ are the inequality constraint functions

Optimal value

p^* = \inf\{f_o(x) |\; f_i(x)\le 0, \; i = 1, \dots, m, h_i(x) = 0, \; i = 1, \dots, p\}

$p^* = \infty$ if a problem is infeasible (no $x$ satisfies the constraints)

$p^* = -\infty$ if a problem is unbounded below

Optimal and locally optimal points

$x$ is feasible if $x\in \text{dom }f_0$ and it satisfies all constraints

a feasible $x$ is optimal $f_0(x) = p^*$ ; $X_{opt}$ is the set of optimal points

$x$ is locally optimal if there is an $R>0$ such that $x$ is optimal for
$\text{minimize } f_0(z)$
subject to
$f_i(z) \le 0, \; i = 1, \dots, m \\ h_i(z) = 0, \; i = 1, \dots, p \\ \|z -x\|_2\le R$

It can be interpreted as the restricted domain of $f_0$

Examples

$f_0(x) = 1/x, \text{dom }f_0 = \R_{++}$ : $p^* = 0$ , but it has no optimal point.

We have to distinguish between optimal point and optimal value

$f_0(x) = -\log x, \text{dom }f_0 = \R_{++}$ : $p^* = -\infty$

Implicit constraints

the standard form optimization problem has an implicit constraint

x\in \mathcal D = \bigcap_{i = 0}^m\text{dom }f_i \cap \bigcap_{i = 1}^p\text{dom }h_i

we call $\mathcal D$ the domain of the problem.

Feasibility problem

\text{find }x

subject to

f_i(x) \le 0, \; i = 1, \dots, m \\ h_i(x) = 0, \; i = 1, \dots, p

can be considered a special case of the general problem with $f_0(x) = 0$

Convex optimization problem

\text{minimize }f_0(x)

subject to

f_i(x) \le 0, \; i = 1, \dots, m \\ h_i(x) = 0, \; i = 1, \dots, p

$f_0, \dots, f_m$ are convex, $h_1, \dots, h_p$ are affine

problem is quasi-convex if $f_0$ is quasi-convex and $f_1, \dots, f_m$ convex

often written as

\text{minimize }f_0(x)

subject to

f_i(x) \le 0, \; i = 1, \dots, m \\ Ax = b

feasible set of a convex optimization problem is a convex set

The convex optimization problem is not an attribute of the feasible set. It is an attribute of the problem description.

equivalent problem : by solving one, we can construct the other solution

identical problem : objective and constraints are identical

Local and global optima

any locally optimal point of a convex problem is globally optimal

Optimality criterion for differentiable objective function

Suppose that the objective $f_0$ in a convex optimization problem is differentiable, so that for all $x, y\in \text{dom }f_0$ ,

f_0(y) \ge f_0(x) + \nabla f_0(x)^T(y-x)

Then $x$ is optimal if and only if it is feasible and

\nabla f_0(x)^T(y-x)\ge 0

for all feasible $y$

Proof
We already know that $f_0(y) \ge f_0(x) + \nabla f_0(x)^T(y-x)$
If $\nabla f_0(x)^T(y- x) \ge 0$ for all feasible $y$ , then $f_0(y) \ge f_0(x)$ for all feasible $y$ . Therefore, $x$ is optimal.
Conversely, suppose $x$ is optimal and there exists $y$ is feasible region such that
$\nabla f_0(x)^T(y-x) < 0$
That means $y$ is located in the half-space that corresponds to the negative sign of the $\nabla f_0(x)$ . It contradicts the assumption.
Therefore, $\nabla f_0(x)^T(y-x) \ge 0$ for all $y$ in the feasible region.

if non-zero,

\nabla f_0(x)

defines a supporting hyperplane to a feasible set

X

x

unconstrained problem : $x$ is optimal if and only if
$x\in \text{dom }f _0\quad \nabla f_0(x) = 0$

In an unconstrained problem case, we can put arbitrary $y$ so the only option that we can choose is $\nabla f_0(x) = 0$

equality constrained problem
$\text{minimize }f_0(x)$
subject to
$Ax = b$
(Assume that it is non-empty; otherwise the problem is unfeasible)
$x$ is optimal if and only if there exists a $\nu$ such that
$x\in \text{dom }f_0 \quad Ax = b \quad \nabla f_0(x) + A^T\nu = 0$
- Proof
  Since $x, y$ is in the feasible region,
  $Ax = b \text{ and }Ay = b$
  That means
  $x- y \in \mathcal N(A)$
  Therefore, the first optimality condition can be written as
  $\nabla f_0(x)^Tz\ge 0, \forall z\in \mathcal N(A)$
  Since $\mathcal N(A)$ is a subspace,
  $\nabla f_0(x)^T(-z) \ge 0, \forall \in \mathcal N(A)$
  That means
  $\nabla f_0(x)^Tz = 0 , \forall z\in \mathcal N(A)$
  So,
  $\nabla f_0(x) \perp \mathcal N(A)$
  Since $\mathcal N(A) \perp \mathcal R(A^T)$ , (equivalently, $\mathcal R(A^T)$ is row space of $A$ )
  3F theorem (Linear algebra and its application written by Gilbert Strang)
  $\nabla f_0(x) \in \mathcal R(A^T) \\ \Rightarrow \exists v\in \text{dom }A, \nabla f_0(x) = A^Tv$
  Since $\text{dom }A$ is a vector space, it is equivalent to say that
  $\exist v\in \text{dom }A, \nabla f_0(x) + A^Tv = 0$
This is the classical Lagrange multiplier optimality condition

minimization over nonnegative orthant
$\text{minimize }f_0(x)$
subject to
$x\succeq 0$
$x$ is optimal if and only if
$x\in \text{dom }f_0 \quad x\succeq 0 \quad \begin{cases}\nabla f_0(x)_i \ge 0 & x_i = 0 \\ \nabla f_0(x)_i = 0 & x_i > 0 \end{cases}$
- Proof
  The first order optimality condition is
  $\nabla f_0(x)^T(y-x)\ge 0$
  for all feasible $y$
  Since $y$ is in the feasible region, $y\succeq 0$ . So $\nabla f_0(x)\succeq 0$ , it not $\nabla f_0(x)^Ty$ is not unbounded below. Then $\nabla f_0(x)^Ty \ge 0$ for all feasible $y$ .
  Therefore
  $- \nabla f_0(x)^Tx \ge 0$
  Since $\nabla f_0(x) \succeq 0, x\succeq 0$ ,
  $\begin{cases}\nabla f_0(x)_i \ge 0 & x_i = 0 \\ \nabla f_0(x)_i = 0 & x_i > 0 \end{cases}$

Equivalent convex problems

two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa

some common transformations that preserve convexity

Eliminating equality constraints

\text{minimize }f_0(x)

subject to

f_i(x) \le 0, \quad i = 1, \dots, m \\ Ax = b

is equivalent to

\text{minimize }_z f_0(Fz+x_0)

subject to

f_i(Fz + x_0) \le 0 \quad i = 1, \dots, m

where $F$ and $x_0$ are such that

Ax = b\Leftrightarrow x = Fz + x_0\text{ for some }z

Actually let $y$ is in the feasible region (i.e. $Ay = b$ and $f_i(y)\le 0, \forall i$ )

Ax = b, Ay = b\\\Rightarrow x - y\in \mathcal N(A)

Let $\mathcal N$ is a matrix which its columns is the linearly independent vector in $\mathcal N(A)$

However, this problem requires at least one feasible point

x_0

If the original problem is a convex problem, the transformed problem is also a convex problem since it is just the composition of the affine function.

Introducing equality constraints

\text{minimize }f_0(A_0x + b_0)

subject to

f_i(A_ix + b_i) \le 0, \quad i = 1, \dots, m

is equivalent to

\text{minimize }_{x, y_i}f_0(y_0)

subject to

f_i(y_i)\le 0, \quad i = 1, \dots, m\\y_i = A_ix + b_i, \quad i = 1, \dots, m

Since this approach eventually increases the number of constraints, there is no progress at first glance. However, this approach can lead to significant progress.

Introducing slack variables for linear inequalities

\text{minimize }f_0(x)

subject to

a_i^Tx \le b_i \quad i = 1, \dots, m

is equivalent to

\text{minimize }_{x, s}f_0(x)

subject to

a_i^Tx + s_i = b_i, \quad i = 1, \dots, m \\ s_i \ge 0 \quad i = 1, \dots, m

This approach can be applicable when we solve the linear programming problem.

Epigraph form

standard form convex problem is equivalent to

\text{minimize }_{x, t}t

subject to

f_0(x) - t\le 0\\f_i(x)\le 0, \quad i = 1, \dots, m\\ Ax = b

By using this, we can change our objective function as a linear objective.

Minimizing over some variables

\text{minimize }f_0(x_1, x_2)

subject to

f_i(x_1) \le 0 \quad i = 1, \dots, m

is equivalent to

\text{minimize }\bar f_0(x_1)

subject to

f_i(x_1) \le 0 \quad i = 1, \dots, m

where $\bar f_0(x_1) = \inf_{x_2}f_0(x_1, x_2)$

We already know that

\bar f_0

is a convex function.

Dynamic programming preserves the convexity of the problem.

Quasi-convex optimization

\text{minimize }f_0(x)

subject to

f_i(x) \le 0, \quad i = 1, \dots, m\\Ax = b

with $f_0:\R^n\to \R$ quasi-convex, $f_1, \dots, f_m$ convex.

Unlike the convex optimization case, it can have locally optimal points that are not globally optimal.

Nevertheless, a variation of the first-order optimality condition does hold for quasi-convex optimization problems with differentiable objective functions.

In quasi-convex optimization problem, $x$ is optimal if

x\in X, \quad \nabla f_0(x)^T(y- x) > 0, \forall y\in X \setminus \{x\}

Unlike the convex optimization case, it is only sufficient for optimality.

Representation via a family of convex functions

It will be convenient to represent the sublevel sets of a quasi-convex function $f$ via inequalities of convex functions. We seek a family of convex functions $\phi_t$ such that

f_0(x) \le t\Leftrightarrow \phi_t(x) \le 0

Evidently $\phi_t$ must satisfy the property that for all $x\in \R^n$ ,

\phi_s(x) \le 0 \Rightarrow \phi_t(x) \le 0

for $s\ge t$ . This is satisfied if for each $x$ , $\phi_t(x)$ is a non-increasing function of $t$ (i.e. $\phi_s(x) \le \phi_t(x)$ whenever $s\ge t$ .

If we take

\phi_t(x) = \begin{cases}0 & f(x) \le t \\ \infty &\text{otherwise}\end{cases}

it satisfies the above conditions.

We call it the indicator function of the t-sublevel of

f

Note that this representation is not unique, for example, if the sublevel sets of $f$ are closed, we can take

\begin{aligned}\phi_t(x) &= \text{dist}(x, \{z|f(z) \le t\}) \\ &= \sup_z\{d(z, x) | \; f(z) \le t\}\end{aligned}

Example

f_0(x) = \frac{p(x)}{q(x)}

with $p$ convex, $q$ concave, and $p(x) \ge 0, q(x) > 0$ on $\text{dom }f_0$ can take $\phi_t(x) = p(x) - tq(x)$

for $t\ge 0, \phi_t$ convex in $x$

$p(x)/q(x) \le t$ if and only if $\phi_t(x) \le 0$

Quasi-convex optimization via convex feasibility problems

Let $\phi_t:\R^n\to \R, t\in \R$ be a family of convex functions that satisfy

f_0(x) \le t \Leftrightarrow \phi_t(x) \le 0

and also, for each $x$ , $\phi_t(x)$ is a non-increasing function of $t$ , i.e.

s \ge t \Rightarrow \phi_s(x) \le \phi_t(x)

Let $p^*$ is the optimal value of the quasi-convex optimization problem.

If the feasibility problem

\text{find }x

subject to

\phi_t(x) \le 0 \\ f_i(x) \le 0, \quad i = 1, \dots, m \\ Ax = b

is feasible, then we have $p^*\le t$ . Conversely, if the problem is infeasible, then we can conclude $p^*\ge t$ .

Note that we want to find the minimum

t

such that satisfy the above feasibility problem. Since

\phi_t(x)

is a non-increasing of

t

, if

t

satisfy the feasibility for

s\ge t

s

is also satisfy the feasibility.

Note that the above problem is a convex feasibility problem, since the inequality constraint functions are all convex, and the equality constraints are linear.

This can be used as the basis of a simple algorithm for solving the quasi-convex optimization problem using bisection (solving a convex feasibility problem at each step)

Assume that the problem is feasible, and start with an interval $[l, u]$ known to contain the optimal value $p^*$ . When then solve the convex feasibility problem at its midpoint $t = (l+u) /2$ , to determine wheter the optimal value is in the lower or upper half of the interval, and update the interval accordingly. This produces a new interval, which also contains the optimal value, but has half the width of the initial interval.

Basically, the idea is same as the parametric search algorithm.

The blue area in the above figure represents the range of t that satisfies feasibility.

Canonical Problems

Depending on the types of objective function and constraints, optimization problems can be categorized. We will delve into the following six subcategories

Linear Programming (LP)

Quadratic Programming (QP)

Quadratically Constrained Quadratic Programming (QCQP)

Second-Order Cone Programming (SOCP)

Semidefinite Programming (SDP)

Conic Programming (CP)

The above problems have the following inclusion relationships, and moving to the right, they can be seen as more generalized forms.

LP\subset QP\subset QCQP\subset SOCP\subset SDP\subset CP

Linear program (LP)

\text{minimize }c^Tx + d

subject to

Gx\preceq h \\ Ax = b

Convex problem with affine objective and constraint functions (everything in a linear programming is affine)

feasible set is a polyhedron

Note that the level set of the objective function is just a hyperplane.

But be cautious, as it is not as simple as it might think at first glance because the number of vertices will increase exponentially.

Examples

Piecewise-linear minimization

\text{minimize }_x\max_{i = 1, \dots, m}a_i^Tx + b

equivalent to an LP

\text{minimize }t

subject to

a_i^Tx + b_i \le t \quad i = 1, \dots, m

Actually it is just a epigraph form.

Piecewise linear function

In mathematics and statistics, a piecewise linear, PL or segmented function is a real-valued function of a real variable, whose graph is composed of straight-line segments.[1]

https://en.wikipedia.org/wiki/Piecewise_linear_function

Chebyshev center of a polyhedron

Chebyshev center of

\mathcal P = \{x|\; a_i^Tx\le b_i, \; i = 1, \dots, m\}

is center of largest inscribed ball

\mathcal B = \{x_c + u|\; \|u\|_2 \le r\}

$a_i^Tx\le b_i$ for all $x\in \mathcal B$ if and only if

\sup\{x_i^T(x_c + u)|\; \|u\|_2\le r\} = a_i^Ts_c + r\|a_i\|_2 \le b_i

It is linear in

s_c

and

r

Hence $x_c, r$ can be determined by solving the LP

\text{maximize }r

subject to

a_i^Tx_c + r\|a_i\|_2\le b_i \quad i = 1, \dots , m

Linear-fractional program

\text{minimize }f_0(x)

subject to

Gx\preceq h \\ Ax = b

where

f_0(x) = \frac{c^Tx + d}{e^Tx + f} \quad \text{dom }f_0(x) = \{x| \; e^Tx + f > 0\}

We already know that linear fractional function is a quasi-convex function

It can be solved by bisection like this

f_0(x) \le t \Leftrightarrow c^Tx + d \le t(e^Tx + f)

If the feasible set is non-empty, the linear-fractional program can be transformed to an equivalent linear program

\text{minimize } c^Ty + dz

subject to

Gy\preceq hz \\ Ay = bz \\ e^Ty + fz = 1 \\ z \ge 0

with variables $y, z$

Proof
If $x$ is feasible in our original problem, then
$y = \frac{x}{e^Tx +f}, \quad z = \frac{1}{e^Tx + f}$
is feasible in the LP problem, with the same objective value $c^Ty + dz = f_0(x)$ . Therefore the optimal value of the original problem is greater than or equal to the optimal value of the LP problem. It is has a $y, z$ corresponds to the optimal point of the original problem.
Conversely,
1. if $(y, z)$ is feasible in the LP problem with $z\ne 0$ , then $x = y/z$ is feasible in the original problem.
1. If $(y,z)$ is feasible in the LP problem with $z = 0$ and $x_0$ is feasible for the original problem. Then $x = x_0 + ty$ is feasible in the original problem for all $t\ge 0$ . Moreover, $\lim{t\to \infty}f_0(x_0 + ty) = c^Ty + dz$ , so we can find feasible points in the original function with objective values arbitrarily close to the objective value of $(y, z)$
The basic idea is that if we add the additional variable $z$ , we can normalize it by using $x$ and $z$ . Therefore we can eliminate the denominator. So if $z$ is positive, we can think it as a normalization.

If $z = 0$ , we can arbitrary add $y$ to $x_0$ . This is a main idea to prove above.

Generalized linear-fractional program

f_0(x) = \max_{i = 1, \dots, r} \frac{c_i^Tx + d_i}{e_i^Tx + f_j}

where $\text{dom }f_0(x) = \{x|\; e_i^Tx + f_i > 0, \; i = 1, \dots, r\}$

a quasi-convex optimization problem. Unlike the linear fractional problem, there is no way to directly convert to LP. However, since it is a quasi-convex problem, it can be solved by bisection.

The sublevel set is just the intersection of sublevel sets which is convex. Therefore, the objective function is a quasi-convex function.

Quadratic program (QP)

The convex optimization problem is called a quadratic program(QP) if the objective function is convex quadratic, and the constraint functions are affine.

\text{minimize }\frac{1}{2}x^TPx + q^Tx + r

subject to

Gx\preceq h \\ Ax = b

where $P\in S_{+}^n, G\in \R^{m\times n}$ and $A \in \R^{p\times n}$

In a quadratic program, we minimize a convex quadratic function over a polyhedron.

Note that if $P\in S_{++}^n$ , a sublevel set is a elipsoid

Proof
Since $P$ is symmetric and positive definite matrix, it has a orthonomal eigenvector. Let say eigenvectors $\{v_1, \dots, v_n\}$ , and eigenvalue $\{\lambda_1, \dots, \lambda_n\}$ .
$\{x|\; f(x) \le \alpha\} = \{x|\; \frac{1}{2}x^TPx + q^Tx + r\le \alpha\}$
Let $x = x_1 v_1 + \cdots + x_nv_n$
$\begin{aligned}\{x|\; f(x) \le \alpha\} &= \{ x|\; \frac{1}{2}(\lambda_1\|v_1\|x_1^2 + \cdots + \lambda_n\|v_n\|x_n^2 + q_1x_1 + \cdots + q_nx_n + r)\le \alpha\} \\ &= \{x|\frac{1}{2}((\lambda_1\|v_1 \|x_1^2 + q_1x_1) + \cdots + (\lambda_n \|v_n\|x_n^2 + q_nx_n)) \le \alpha - r\} \end{aligned}$
Therefore, it is a ellipsoid if it is not a empty set.

P

is not a positive semi-definite (i.e. it has a negative eigenvalue), the problem is belongs to NP hard.

Quadratically constrained quadratic program (QCQP)

\text{minimize }\frac{1}{2}x^TP_0x + q_0^Tx + r_0

subject to

\frac{1}{2}x^TP_i x + q_i^Tx + r_i \le 0, \quad i = 1, \dots, m \\ Ax = b

where $P_i\in S_{+}^n$ .

If $P_1, \dots, P_m\in S_{++}^n$ , the feasible region is a intersection of $m$ ellipsoids and an affine set.

Examples

Lest-squares

We already know that

\|Ax - b\|_2^2 = x^TA^TAx - 2b^TAx + b^Tb

and since

\begin{aligned}x^TA^TAx &= (Ax)^T(Ax) \\ &= \langle Ax, Ax\rangle \\ &= \|Ax\| \ge 0\end{aligned}

for all $x$ , $A^TA$ is a positive semi-definite matrix.

Therefore

\text{minimize }\|Ax - b\|_2^2

it is a Quadratic programming.

Actually, this problem is simple enough to have the well known analytical solution $x^* = A^\dagger b$ ( $A^\dagger$ is a pseudo-inverse of $A$ )

When linear inequality constraints are added, the problem is called constrained regression or constrained least-squares

There are no analytical solution for these cases.

As an example we can consider regression with lower and upper bounds on the variables

\text{minimize } \|Ax - b\|_2^2

subject to

l_i\le x_i \le u_i \quad i = 1, \dots, n

There is another example like this

\text{minimize } \|Ax - b\|_2^2

where $x_1 \le x_2 \le \cdots \le x_n$

Actually, it is just adding a set of linear inequalities. Therefore, it is a quadratic programming.

Second-order cone programming (SOCP)

\text{minimize }f^Tx

subject to

\|A_ix + b_i\|_2 \le c_i^Tx + d_i, \quad i = 1, \dots, m \\ Fx = g

where $x\in \R^n, A_i\in \R^{n_i\times n}, F\in \R^{p\times n}$ .

We call a constraint of the form

\|Ax + b\|_2 \le c^Tx + d

where $A\in \R^{k\times n}$ , a second-order cone constraint because

(A_ix + b_i, c_i^Tx + d_i) \in \text{second-order cone in }\R^{n_i + 1}

If $c_i = 0$ for all $i = 1, \dotsm, m$ , then the SOCP is equivalent to a QCQP.

If $A_i = 0$ for all $i = 1, \dots, m$ , then the SOCP reduces to a general LP

It is more general than QCQP and LP

Let $f_i(x) = \|A_ix + b_i\|_2 - c_i^Tx -d_i$ . Then, it is not differentiable for all $x$ since it is not differentiable at $A_i x + b_i = 0$ . It might seems it is okay to neglect that points at the first glance, but we can’t because normally that the solution is very often at these points.

Currently, we can solve SOCP as fast as LP.

Examples

Robust linear programming

the parameters in optimization problems are often uncertain

\text{minimize }c^Tx

subject to

a_i^Tx \le b_i\quad i = 1, \dots, m

there can be uncertainty in $c, a_i, b_i$

two common approaches to handling uncertainty (for simplicity, $a_i$ has only uncertainty)

deterministic model: constraints must hold for all $a_i\in \mathcal E_i$
$\text{minimize }c^Tx$
subject to
$a_i^Tx \le b_i \text{ for all }a_i\in \mathcal E_i \quad i = 1, \dots, m$

Since constraints must satisfy for all uncertainty region, this approach is called worst case model.
Let an ellipsoid as $\mathcal E_i$
$\mathcal E_i = \{\bar a_i + P_i u |\; \|u\|_2 \le 1\}$
where $\bar a_i\in \R^n, P_i\in S_+^n$
Then our constraints can be expressed as
$\sup\{a_i^Tx|\; a_i\in \mathcal E_i\}\le b_i \\ \Leftrightarrow \sup\{(\bar a_i + P_iu)^Tx|\; \|u\|_2\le 1\} \le b_i \\ \Leftrightarrow \bar a_i^Tx + \sup\{u^TP_i^Tx|\; \|u\|_2 \le 1\} \le b_i \\ \Leftrightarrow \bar a_i^T + \|P_i^Tx\|_2 \le b_i$
Therefore it can be represented as a SCOP
$\text{minimize }c^Tx$
subject to
$\bar a_i^Tx + \|P_i^Tx \|_2 \le b_i \quad i = 1, \dots, m$

Note that the additional norm terms act as regularization terms ; they prevent $x$ from being large in directions with considerable uncertainty in the parameters $a_i$ .

Since SOCP is as fast as LP, it can be applicable when we want to solve the finance problem.

stochastic model: $a_i$ is random variable; constraints must hold with probability $\eta$
$\text{minimize }c^Tx$
subject to
$\text{prob}(a_i^Tx\le b_i)\ge \eta \quad i = 1, \dots, m$

Alleviate the condition by using probability
Assume $a_i$ is Gaussian with mean $\bar a_i$ , covariance $\Sigma_i$ (i.e. $a_i\sim \mathcal N(\bar a_i, \Sigma_i)$ )
Then $a_i^Tx \sim \mathcal N(\bar a_i^Tx, x^T\Sigma_ix)$ . So,
$\text{prob}(a_i^Tx\le b_i) = \Phi\bigg(\frac{b_i - \bar a_i^Tx}{\|\Sigma_i^{1/2}x\|_2}\bigg)$
where $\Phi(x)$ is CDF of $\mathcal N(0, 1)$
Then it can be written as
$\text{minimize }c^Tx$
subject to
$\bar a_i^Tx + \Phi^{-1}(\eta) \|\Sigma_i^{1/2}x\|_2 \le b_i \quad i = 1, \dots, m$
If $\Phi^{-1}(\eta) \ge 0$ (i.e. $\eta \ge 1/2$ ), it is equivalent to the SOCP.

Geometric program (GP)

Monomial function

f(x) = cx_1^{a_1}x_2^{a_2}\cdots x_n^{a_n} \quad \text{dom }f = \R_{++}^n

with $c>0$ and $a_i\in \R$ , is called a monomial function

In mathematics, monomial has non-negative integers as its exponents. So the above definition is not a standard definition.

Posynomial function

it is just a sum of monomials

f(x) = \sum_{k = 1}^K c_k x_1^{a_{1k}}x_2^{a_{2k}}\cdots x_n^{a_{nk}} \quad \text{dom }f = \R_{++}^n

Geometric program (GP)

\text{minimize }f_0(x)

subject to

f_i(x) \le 1, \quad i = 1, \dots, m\\ h_i(x) = 1 \quad i = 1, \dots, p

where $f_0$ is a posynomial function, $h_i$ is a monomial function.

Geometric program in convex form

Geometric programs are not generally convex optimization problem, but they can be transformed to convex problems by a change of variables and a transformation of the objective and constraint functions.

monomial $f(x) = cx_1^{a_1}\cdots x_n^{a_n}$ transforms to
$\log f(e^{y_1}, \dots, e^{y_n}) = a^Ty + b$
where $b = \log c$

posynomial $f(x) = \sum_{k = 1}^K c_k x_1^{a_{1k}}\cdots x_n^{a_{nk}}$ transforms to
$\log f(e^{y1}, \dots, e^{y_n}) = \log \bigg (\sum_{k = 1}^K e^{e^{a_k^Ty + b_k}}\bigg)$
where $b_k = \log c_k$

We already know that log-sum-exp is a convex function in $y$

Therefore, geometric program transforms to a convex problem

\text{minimize }\log\bigg (\sum_{k = 1}^K \exp(a_{0k}^Ty + b_{0k})\bigg)

subject to

\log\bigg(\sum_{k = 1}^K \exp(a_{ik}^Ty + b_{ik})\bigg) \le 0 \\ g_i^Ty + h_i = 0

Generalized inequality constraints

One very useful generalization of the standard form convex optimization problem is obtained by allowing the inequality constraint functions to be vector valued , and using generalized inequalities in the constraints.

\text{minimize }f_0(x)

subject to

f_i(x) \preceq_{K_i} 0 \quad i = 1, \dots, m \\ Ax = b

where $f_0:\R^n\to \R, K_i\subset R^{k_i}$ are proper cones, and $f_i:\R^n\to\R^{k_i}$ $K_i$ -convex.

Later, we will see that convex optimization problems with generalized inequality constraints can often be solved as easily as ordinary convex optimization problems.

Conic form problems

special case with affine objectives and constraints

\text{minimize }c^Tx

subject to

Fx + g \preceq_K 0 \\ Ax = b

extends linear programming ( $K= \R_{+}^m$ ) to non-polyhedral cones

Semidefinite programming

\text{minimize }c^Tx

subject to

x_1 F_1 + \cdots + x_nF_n + G\preceq0 \\ Ax = b

where $G, F_1, \dots, F_n\in S^k$ , and $A\in \R^{p \times n}$

inequality constraint is called linear matrix inequality (LMI). That means the left-hand side is negative semi-definite.

Note that it includes problems with multiple LMI constraints

x_1\bar F_1+ \cdots + x_n\bar F_n + \bar G \preceq 0 \\ x_1\tilde F_1+ \cdots + x_n\tilde F_n + \tilde G \preceq 0

is equivalent to single LMI

x_1\begin{bmatrix}\bar F_1 & 0 \\ 0 & \tilde F_1\end{bmatrix} + x_2\begin{bmatrix}\bar F_2 & 0 \\ 0 & \tilde F_2\end{bmatrix} + \cdots + x_n \begin{bmatrix}\bar F_n & 0 \\ 0 & \tilde F_n\end{bmatrix} + \begin{bmatrix}\bar G & 0 \\ 0 & \tilde G\end{bmatrix} \preceq 0

Example

LP and equivalent SDP
- LP
  $\text{minimize }c^Tx$
  subject to
  $Ax\preceq b$
- SDP
  $\text{minimize }c^Tx$
  subject to
  $\text{diag}(Ax - b)\preceq0$
Use the fact that if the given diagonal matrix has only non-positive elements, it is a negative semi-definite matrix.

SOCP and equivalent SDP
- SOCP
  $\text{minimize }f^Tx$
  subject to
  $\|A_ix + b_i\|_2 \le c_i^T x +d_i \quad i = 1, \dots, m$
- SDP
  $\text{minimize } f^Tx$
  subject to
  $\begin{bmatrix}(c_i^Tx + d_i)I & A_ix + b_i \\ (A_ix + b_i)^T & c_i^Tx + d_i\end{bmatrix}\succeq 0 \quad i = 1, \dots, m$
- Proof
  Before we start, we have to know the following fact (Schur complement of positive semidefinite matrix)
  $X = \begin{bmatrix}A & B \\ B^T & D\end{bmatrix} \\ S = D - B^TA^{-1}B$
  Then,
  if $D\succ 0$ then
  $X \succeq 0 \text{ iff }S \succeq 0$
  We already know that $c_i^Tx + d_i \succeq 0$ by non-negativity of the norm. Therefore,
  $\begin{bmatrix}(c_i^Tx + d_i)I & A_ix + b_i \\ (A_ix + b_i)^T & c_i^Tx + d_i\end{bmatrix}\succeq 0 \text{ iff } c_i^Tx + d_i - \frac{(A_ix+b_i)^T(A_ix + b_i)}{c_i^Tx + d_i} \succeq 0\\ \Rightarrow \begin{bmatrix}(c_i^Tx + d_i)I & A_ix + b_i \\ (A_ix + b_i)^T & c_i^Tx + d_i\end{bmatrix}\succeq 0 \text{ iff } \|A_ix+b_i\|_2^2 \le (c_i^Tx + d_i)^2$
  Since $c_i^Tx + d_i \ge 0$ ,
  $\begin{bmatrix}(c_i^Tx + d_i)I & A_ix + b_i \\ (A_ix + b_i)^T & c_i^Tx + d_i\end{bmatrix}\succeq 0 \text{ iff } \|A_ix+b_i\|_2 \le c_i^Tx + d_i$

Eigenvalue minimization
$\text{minimize }\lambda_{\max}(A(x))$
where $A(x) = A_0 + x_1 A_1 + \cdots + x_nA_n$ with given $A_i\in S^k$
we can express it as a SDP form
$\text{minimize }t$
subject to
$A(x) \preceq tI$
where $x\in \R^n, t\in \R$
it follows from the fact that
$\lambda_{\max}(A) \le t \Leftrightarrow A\preceq tI$
- Proof
  $\lambda$ is an eigenvalue of $A(x)$ if and only if $\lambda - t$ is an eigenvalue of $A(x) - tI$ .
  Therefore, $\lambda_{\max} (A(x)) \le t$ if and only if all eigenvalues of $A(x) -tI$ are non-positive. (i.e. $A(x) -tI \preceq 0$ )

Matrix norm
$\text{minimize }\|A(x)\|_2 := (\lambda_{\max}(A(x)^TA(x)))^{1/2}$
where $x\in \R^n, t\in \R , A(x) = A_0 + x_1A_1 + \cdots + x_nA_n$ with given $A_i\in \R^{p\times q}$

We already know that matrix norm is a convex function.
we can express it as a SDP form
$\text{minimize }t$
subject to
$\begin{bmatrix}tI & A(x) \\ A(x)^T & tI\end{bmatrix}\succeq 0$
the above matrix is positive semi-definite if
$tI\ge 0 \\ tI - \frac{A(x)^TA(x)}{t} \succeq 0$
Since $t$ is positive,
$t^2I \succeq A(x)^TA(x)$
Therefore, all eigenvalues of $A(x)^TA(x)$ are less than equal to $t^2$

Vector optimization

General vector optimization problem

\text{minimize }f_0(x) \text{ (with respect to K)}

subject to

f_i(x) \le 0 \quad i = 1, \dots, m\\ h_i(x) = 0 \quad i = 1, \dots, p

vector objective $f_0:\R^n\to \R^q$ minimized with respect to proper cone $K\in \R^q$

How to compare the value of vectors? There’s a lot of ways to do that. (i.e. lexicographical order)

Convex vector optimization problem

\text{minimize }f_0(x) \text{ (with respect to K)}

subject to

f_i(x) \le 0 \quad i = 1, \dots, m\\ Ax = b

with $f_0$ $K$ -convex, $f_1, \dots, f_m$ convex.

Optimal and Pareto optimal points

set of achievable objective values

\mathcal O = \{f_0(x) |\; x \text{ feasible} \}

feasible $x$ is optimal if $f_0(x)$ is a minimum value of $\mathcal O$
$x^*$ is optimal since it is comparable to any other points in $\mathcal O$ and it is also less than any other point in $\mathcal O$

feasible $x$ is Pareto optimal if $f_0(x)$ is a minimal value of $\mathcal O$
$x^{po}$ is Pareto optimal since there is non-comparable point in $\mathcal O$

Multi-criterion optimization

vector optimization problem with $K = \R_{+}^q$

f_0(x) = (F_1(x), \dots, F_q(x))

where we want all $F_i$ ’s to be small

feasible $x^*$ is optimal if
$y\text{ feasible} \Rightarrow f_0(x^*) \preceq f_0(y)$
if there exists an optimal point, the objectives are non-competing

feasible $x^{po}$ is Pareto optimal if
$y\text{ feasible}, f_0(y) \preceq f_0(x^{po}) \Rightarrow f_0(x^{po}) = f_0(y)$
if there are multiple Pareto optimal values, there is a trade-off between the objectives.

Example : Regularized least-squares

\text{minimize }(\|Ax - b\|_2^2, \|x\|_2^2) \text{ (w.r.t }\R_+^2)

In the deep learning concept, this concept can be interpreted as a regularization.

Scalarization

Scalarization is a standard technique for finding Pareto optimal points for a vector optimization problem. To find Pareto optimal points: choose $\lambda \succ_{K^*}0$ and solve scalar problem

\text{minimize }\lambda^T f_0(x)

subject to

f_i(x) \le 0, \quad i = 1, \dots,m \\ h_i(x) = 0 \quad i = 1, \dots, p

One way to say that

f_0

K

-convex, if

\lambda^T f_0(x)

is convex a function function for all

\lambda \in K^*

Any optimal points in this problem is the Pareto optimal problem for the original problem.

for convex vector optimization problems, can find (almost) all Pareto optimal points by varying

\lambda \succ_{K^*}0

Scalarization for multicriterion problems

to find Pareto optimal points, minimize positive weighted sum

\lambda^T f_0(x) = \lambda_1 F_1(x) + \cdots + \lambda_qF_q(x)

Example

Regularized least-squares problem

Take $\lambda = (1, \gamma)$ with $\gamma > 0$

\text{minimize }\|Ax - b\|_2^2 + \gamma \|x\|_2^2

for fixed $\gamma$

In Deep learning perspective, this can be interpreted as a regularization term.