Open-Loop and Closed-Loop Solvabilities for Stochastic Linear Quadratic Optimal Control Problems

This paper is concerned with a stochastic linear quadratic (LQ, for short) optimal control problem. The notions of open-loop and closed-loop solvabilities are introduced. A simple example shows that these two solvabilities are different. Closed-loop solvability is established by means of solvability of the corresponding Riccati equation, which is implied by the uniform convexity of the quadratic cost functional. Conditions ensuring the convexity of the cost functional are discussed, including the issue that how negative the control weighting matrix-valued function R(s) can be. Finiteness of the LQ problem is characterized by the convergence of the solutions to a family of Riccati equations. Then, a minimizing sequence, whose convergence is equivalent to the open-loop solvability of the problem, is constructed. Finally, an illustrative example is presented.

Most recently, Sun-Yong [24] established that the existence of open-loop optimal controls is equivalent to the solvability of the corresponding optimality system which is a forward-backward stochastic differential equation (FBSDE, for short), and the existence of closed-loop optimal strategies is equivalent to the existence of a regular solution to the corresponding Riccati equation. From this point of view, this paper can be regarded as a continuation of [24], in a certain sense. Inspired by a result found in [29], we are able to cook up an example for which open-loop optimal controls exist, but the closed-loop optimal strategy does not exist. Because of this, it is necessary to distinguish open-loop and closed-loop solvabilities of Problem (SLQ). Next, having the equivalence between the solvability of the Riccati equation and the closed-loop solvability of Problem (SLQ), it is natural to seek conditions under which the Riccati equation is solvable, and the sought conditions are expected to be more general than (1.4) so that they could include some (although might not be all) cases that R(·) is allowed to be not positive semi-definite. One of our main results in this paper is to establish the equivalence between the strongly regular solvability of the Riccati equation (see below for a definition) and the uniform convexity of the cost functional. Note that uniform convexity condition is much weaker than (1.4), and is different from conditions imposed in [23].
Finiteness of Problem (SLQ) (meaning that the infimum of the cost functional is finite) is another important issue. The notion was introduced in [30] (see also [8,9]). Some investigations were carried out in [21]. In this paper, due to the perfect structure of Problem (SLQ) 0 , its finiteness will be characterized by the convergence of the solutions to a family of Riccati equations. As a by-product, we will construct minimizing sequences of Problem (SLQ) in a very natural way, and the convergence of the sequences will lead to the open-loop solvability of Problem (SLQ).
Among other things, we find several interesting facts which are listed below: Fact 2. If Problem (SLQ) 0 is finite at t, then it is finite at any t ′ > t.
Fact 3. For Problem (SLQ) with D(·) = 0, under the assumption that R(·) is uniformly positive definite, without assuming the non-negativity of Q(·) and G, the finiteness and the unique closed-loop solvability of Problem (SLQ) are equivalent, which are also equivalent to the uniform convexity of the cost functional. The rest of the paper is organized as follows. Section 2 collects some preliminary results. In Section 3, we study the cost functional from a Hilbert space viewpoint and represent it as a quadratic functional of u(·). Section 4 is devoted to the strongly regular solvability of the Riccati equation under the uniform convexity of the cost functional. In Section 5, we discuss the finiteness of Problem (SLQ) as well as the convexity of the cost functional. In Section 6, we characterize the open-loop solvability of Problem (SLQ) by means of the convergence of minimizing sequences. An example is presented in section 7 to illustrate some relevant results obtained.

Preliminaries
We recall that R n is the n-dimensional Euclidean space, R n×m is the space of all (n × m) matrices, endowed with the inner product M, N → tr [M ⊤ N ] and the norm |M | = tr [M ⊤ M ], S n ⊆ R n×n is the set of all (n × n) symmetric matrices, S n + ⊆ S n is the set of all (n × n) positive semi-definite matrices, and S n + ⊆ S n + is the set of all (n × n) positive-definite matrices. When there is no confusion, we shall use · , · for inner products in possibly different Hilbert spaces. Also, M † stands for the (Moore-Penrose) pseudo-inverse of the matrix M ( [22]), and R(M ) stands for the range of the matrix M . Next, let T > 0 be a fixed time horizon. For any t ∈ [0, T ) and Euclidean space H, let For an S n -valued function F (·) on [t, T ], we use the notation F (·) ≫ 0 to indicate that F (·) is uniformly positive definite on [t, T ], i.e., there exists a constant δ > 0 such that F (s) δI, a.e. s ∈ [t, T ].
The following standard assumptions will be in force throughout this paper.
(H1) The coefficients of the state equation satisfy the following: (H2) The weighting coefficients in the cost functional satisfy the following: By [24,Proposition 2.1], under (H1)-(H2), for any (t, x) ∈ [0, T ) × R n and u(·) ∈ U[t, T ], the state equation (1.1) admits a unique strong solution X(·) ≡ X(· ; t, x, u(·)), and the cost functional (1.2) is welldefined. Then Problem (SLQ) makes sense. It is worthy of pointing out that in (H2), we do not impose any positive-definiteness/non-negativeness conditions on the weighting matrix/matrix-valued functions G, Q(·) and R(·). We now introduce the following definition. (ii) An element u * (·) ∈ U[t, T ] is called an open-loop optimal control of Problem (SLQ) for the initial pair If an open-loop optimal control (uniquely) exists for (t, where X * (·) is the strong solution to the following closed-loop system: We emphasize that in the definition of closed-loop optimal strategy, (2.3) has to be true for all x ∈ R n . One sees that if (Θ * (·), v * (·)) is a closed-loop optimal strategy of problem (SLQ) on [t, T ], then the outcome u * (·) ≡ Θ * (·)X * (·) + v * (·) is an open-loop optimal control of Problem (SLQ) for the initial pair (t, X * (t)). Hence, the existence of closed-loop optimal strategies implies the existence of open-loop optimal controls. But, the existence of open-loop optimal controls does not necessarily imply the existence of a closed-loop optimal strategy. Here is such an example which is inspired by an example for deterministic LQ problems found in [29].
Then we have from (2.8) that the solution X * (·) of the following closed-loop system: Consequently, which is impossible since the above has to be true for all x ∈ R. Hence, the corresponding Problem (SLQ) 0 is not closed-loop solvable on any [t, 1] with t ∈ [0, 1), although the problem admits a continuous open-loop optimal control for any initial pair (t, x) ∈ [0, 1) × R n .
Due to the above indicated situation, unlike in [24], and in classical literature on LQ problems, we distinguish the notions of open-loop and closed-loop solvabilities for Problem (SLQ). We repeat here that for given initial time t ∈ [0, T ), an open-loop optimal control is allowed to depend on the initial state x, whereas, a closed-loop optimal strategy is required to be independent of the initial state x. Because of the nature of closed-loop strategies, we define the finiteness of Problem (SLQ) only in terms of open-loop controls.
The following lemma is concerned with the solution to a Lyapunov equation, whose proof can be found in [10] (see also [30,Chapter 6]).
Finally, we state the following extended Schur's lemma whose proof can be found in [3].
Lemma 2.6. Let Q ∈ S n , R ∈ S m and S ∈ R m×n . Then if and only if Note that the third condition in (2.13) is equivalent to the following: (2.14) S ⊤ (I − RR † ) = 0.

Representation of the Cost Functional
In this section, we will present a representation of the cost functional for Problem (SLQ), from which we will obtain some basic conditions ensuring the convexity of the cost functional. Convexity of the cost functional will play a crucial role in the study of finiteness and open-loop/closed-loop solvability of Problem (SLQ).
The following proposition is a summary of some relevant results found in [30].
Moreover, let (X 0 (·), Y (·), Z(·)) be the adapted solution of the following (decoupled) linear FBSDE: Then Let (X 0 (·),Ȳ (·),Z(·)) be the adapted solution to the following (decoupled) FBSDE: Then Also, let ( X 0 (·), Y (·), Z(·)) be the adapted solution to the following (decoupled) FBSDE: Then Finally, M 0 (·) solves the following Lyapunov equation: and it admits the following representation: where Φ(·) is the solution to the following SDE for R n×n -valued process: Proof. Let Φ(·) be the solution to (3.10). Then Φ(s) −1 exists for all s 0 and the following holds: By the variation of constants formula, the solution X(·) ≡ X(· ; t, x, u(·)) of the state equation (1.1) can be written as follows: and define the following operators: Clearly, for any t ∈ [0, T ), Then, for any t ∈ [0, T ) and (x, u(·)) ∈ R n × U[t, T ], the corresponding state process X(·) and its terminal value X(T ) are given by Hence, the cost functional can be written as In the above, · , · is used for inner products in possibly different spaces. Further, the adjoint operators are given by the following: with (Y 0 (·), Z 0 (·)) being the adapted solution to the following backward stochastic differential equation (BSDE, for short): with (Y 1 (·), Z 1 (·)) being the adapted solution to the following BSDE: Then with the well-defined adjoint operators, we can rewrite the cost functional as follows: T ] being bounded, and M 0 (t) ∈ S n ; y t ∈ R n , ν t (·) ∈ U[t, T ] and c t ∈ R. Note that ν t (·), y t , c t = 0 when b(·), σ(·), g, q(·), ρ(·) = 0. This gives (3.1). Further, if we let and (Y (·), Z(·)) is the adapted solution to the following BSDE: and (Ȳ (·),Z(·)) is the adapted solution to the following BSDE: Thus (3.5) follows. Likewise, if we let and ( Y (·), Z(·)) is the adapted solution to the following BSDE: Thus (3.7) follows. Finally, we know that Then (3.9) follows. Also, by Lemma 2.4, we see that M 0 (·) is the unique solution of Lyapunov equation (3.8).
From the representation of the cost functional, we have the following simple corollary.
Note that if u(·) happens to be an open-loop optimal control of Problem (SLQ), then the following stationarity condition holds: which brings a coupling into the FBSDE (3.18). We call (3.18), together with the stationarity condition (3.21), the optimality system for the open-loop optimal control of Problem (SLQ).
The following is concerned with the convexity of the cost functional, whose proof is straightforward, by making use of the representation (3.1) of the cost functional. Corollary 3.3. Let (H1)-(H2) hold and let t ∈ [0, T ) be given. Then the following are equivalent: Similar to the above, we have that u(·) → J(t, x; u(·)) is uniformly convex if and only if for some λ > 0. This is also equivalent to the following: for some λ > 0. Further, it is obvious that if the standard conditions (1.4) hold, then which means that the functional u(·) → J 0 (t, 0, u(·)) is convex. The following result tells us that under (1.4), one actually has the uniform convexity of the cost functional.
Note that in the above, the constant α does not have to be nonnegative.
Proof. First of all, by the uniform convexity of u(·) → J 0 (0, 0; u(·)), we may assume that for some λ > 0. Now, for any t ∈ [0, T ), and any u(·) ∈ U[t, T ], we define the zero-extension of u(·) as follows: Then , and due to the initial state being 0, the solution Hence, Thus, u(·) → J 0 (t, x; u(·)) is uniformly convex for any given (t, x) ∈ [0, T ) × R n . By Corollary 3.2, we have (4.5) Consequently, by a standard argument involving minimizing sequence and locally weak compactness of Hilbert spaces, we see that for any given initial pair (t, x) ∈ [0, T ) × R n , Problem (SLQ) admits a unique open-loop optimal control. Moreover, when b(·), σ(·), g, q(·), ρ(·) = 0, (4.5) implies that Note that the functions on the right-hand side of (4.6) are quadratic in x and continuous in t. (4.1) follows immediately. Now, we introduce the following Riccati equation associated with Problem (SLQ): A solution P (·) of (4.7) is said to be strongly regular if for some λ > 0. The Riccati equation (4.7) is said to be (strongly) regularly solvable, if it admits a (strongly) regular solution. Clearly, condition (4.11) implies (4.8)-(4.10). Thus, a strongly regular solution P (·) must be regular. Moreover, it was shown in [24] that if a regular solution of (4.7) exists, it must be unique.
In [1], it was showed that for Problem (SLQ) 0 , the existence of a continuous open-loop optimal control for any initial pair (t, x) ∈ [0, T ] × R n is equivalent to the solvability of the corresponding Riccati equation  It is easy to see that P (t) = t is the unique solution of (4.12), satisfying (4.8) and (4.10). Now, we claim that this problem does not admit an open-loop optimal control for initial pair (0, x) with x = 0. In fact, if for some x = 0, there exists an open-loop optimal control u * (·) ∈ U[0, T ], then by the maximum principle, the solution (X * (·), Y * (·)) of the following (decoupled) forward-backward differential equation: Observe that the solution (X * (·), Y * (·)) of (4.13) is given by Hence, a.e. s ∈ (0, 1]. Noting that u * (·) is square-integrable, we must have X * (1) = 0 and hence u * (·) = 0. Consequently, which is a contradiction.
From the above example, we see that the sufficiency part of the above Theorem 4.2 (a result from [1]) does not hold. We will see in Section 7 that the necessity part does not hold either.
Instead of Theorem 4.2, in [24], the following were proved, which establishes the equivalence between the closed-loop solvability of Problem (SLQ) and the regular solvability of the Riccati equation (4.7).
In this case, Problem (SLQ) is closed-loop solvable on any [t, T ], and the closed-loop optimal strategy (Θ * (·), v * (·)) admits the following representation: for some Π(·) ∈ L 2 (t, T ; R m×n ) and ν(·) ∈ L 2 F (t, T ; R m ), and the value function is given by Note that in Example 4.3, the solution P (t) = t to the Riccati equation (4.12) is not regular since Hence, by Theorem 4.4, the corresponding LQ problem does not have a closed-loop optimal strategy.
From the above theorem, we see that the existence of a strongly regular solution to the Riccati equation  [30,10]). To summarize up, we have the following diagram: where "RE" stands for the Riccati equation (4.7). It is clear that the uniform convexity of the map u(·) → J 0 (t, x; u(·)) does not imply the standard conditions (1.4), which will be even clearer by the results of Section 5 below. Therefore, it is a desire to establish the following: x; u(·)) uniformly convex ⇐⇒ RE strongly regularly solvable This is our next goal. To achieve this, we first present the following proposition, which plays a key technical role in this section.  where α ∈ R is the constant appears in (4.1).
Remark 4.7. From the first part of the proof of Theorem 4.6, we see that if (4.2) holds, then the strongly regular solution of (4.7) satisfies (4.11) with the same constant λ > 0.
Combining Theorem 4.4 and Theorem 4.6, we obtain the following corollary.

Finiteness of Problem (SLQ) and Convexity of Cost Functional
We have seen that the uniform convexity of the cost functional implies the open-loop and closed-loop solvabilities of Problem (SLQ). We expect that the finiteness of Problem (SLQ) should be closely related to the convexity of the cost functional. A main purpose of this section is to make this clear. Other relevant issues will also be discussed. First, we introduce the following: We have the following result. (ii) Problem (SLQ) 0 is finite at t.
(iii) There exists a P (t) ∈ S n such that (iv) The map u(·) → J(t, x; u(·)) is convex, for any x ∈ R n .
the following implications hold: Proof. (i) ⇒ (ii). By Proposition 3.1, for any x ∈ R n and u(·) ∈ U[t, T ], we have which implies (ii).
(ii) ⇒ (iii) can be shown by a simple adoption of the well-known result in the deterministic case (see [11,4]).
T ], and any P (·) ∈ AC(t, T ; S n ), one has Hence, if P[t, T ] = ∅, then by taking P (·) ∈ P[t, T ], one has This implies that the corresponding Problem (SLQ) 0 is finite at t.
It is worth to point out that the convexity of the map u(·) → J 0 (t, x; u(·)) is not sufficient for the finiteness of Problem (SLQ) 0 . We present the following example (see also [21] for an example of a quadratic functional in Hilbert space).
Also, by Remark 4.9 and Proposition 3.1, This leads to On the other hand, let Φ A (·) be the solution of (5.15) and set Hence, combining (5.18)-(5.19), we have Thus, Then using the same argument as in the previous paragraph, we can show that which implies the finiteness of Problem (SLQ) 0 . Moreover, let P : [0, T ] → S n such that (5.2) holds, then and (5.13) follows.
The following is another sufficient condition for the finiteness of Problem (SLQ) 0 , which is a corollary of Theorem 5.3 and Proposition 5.1, (v) ⇒ (ii).
Corollary 5.4. Let (H1)-(H2) hold. If there exists a ∆(·) ∈ L 1 (0, T ; S n + ) such that where P (·) is the solution of the following Lyapunov equation: We now return to the study of convexity of the map u(·) → J 0 (t, 0; u(·)). First, from the definition of M 2 (t) (see (3.13)), we see that M 2 (t) 0 if and only if with the right hand side being non-positive. Thus, unlike the well-known situation for the deterministic LQ problems (for which R(·) 0 is necessary for M 2 (t) 0 ([30])), R(·) does not have to be positive semi-definite. Actually, as shown by examples in [7,30], R(·) could even be negative definite in some extent. Let us now take a closer look at this issue.
Note that when u(·) → J 0 (0, 0; u(·)) is convex, for any ε > 0, the unique strongly regular solution P ε (·) to the Riccati equation (5.9) satisfies (5.10) and (5.19). Hence, or equivalently, This is another necessary condition for the finiteness of Problem (SLQ) 0 , which is easier to check. From (5.26), we see that if R(·) happens to be negative definite, then in order u(·) → J 0 (0, 0; u(·)) to be convex, it is necessary that D(·) is injective, and either G or Q(·) (or both) has to be positive enough to compensate. Note that D(s) was assumed to be invertible in [23]. Therefore, in some sense, our result justifies the assumption of [23].
The following gives a little improvement when more restrictive conditions are assumed.
Note that in the case (5.27), we have with Φ A (·) being the solution of (5.15). Hence, (5.26) can also be written as (5.29) As we pointed out earlier, Problem (SLQ) 0 may still be infinite at some initial pair (t, x) even if the cost functional is convex. In this case, by Theorem 5.3, the sequence {P ε (t)} ε>0 diverges. The following result is concerned with the divergence speed of {P ε (t)} ε>0 .
Note that although D(·) = 0, since C(·) is not necessarily zero, our state equation is still an SDE. For such a case, the above results can be restated as follows.
Theorem 5.8. Let (H1)-(H2) and (5.32) hold. Then the following statements are equivalent: (i) Problem (SLQ) is finite at t = 0; (ii) Problem (SLQ) 0 is finite at t = 0; (iii) The map u(·) → J 0 (0, 0; u(·)) is uniformly convex; (iv) The following Riccati equation (iii) ⇔ (iv). In the case of (5.32), the corresponding Riccati equation becomes (5.33). If P (·) ∈ C([0, T ]; S n ) is a solution of (5.33), then it is automatically strongly regular. Thus, by Theorem 4.6, we obtain the equivalence of (iii) and (iv). An interesting point of the above is that under condition (5.32), finiteness of Problem (SLQ) implies the closed-loop solvability of Problem (SLQ). In the deterministic case, such a fact was firstly revealed in [31] for two-person zero-sum differential games, and was proved in [29] for deterministic LQ problems by means of Fredholm operators.
Denote the above problem by Problem (SLQ) ε and the corresponding value function by V ε (· , ·). By Corollary 4.8, u ε (·) defined by (6.4) is the unique optimal control of Problem (SLQ) ε at (t, x) ∈ [0, T ) × R n . Note that The proof is completed.
Using the minimizing sequence constructed in Theorem 6.1, the open-loop solvability of Problem (SLQ) can be characterized as follows.
In this case, the weak (strong) limit of any weakly (strongly) convergent subsequence of {u ε (·)} ε>0 is an open-loop optimal control of Problem (SLQ) at (t, x).
(ii) The necessity is obvious. Now suppose θ n → θ and θ n → θ weakly. Then This completes the proof.

An example
In this section we re-exam Example 2.2 to illustrate some results we obtained. In this example, the stochastic LQ problem admits a continuous open-loop optimal control at all (t, x) ∈ [0, T ) × R n , hence it is open-loop solvable, while the value function is not continuous in t; the corresponding Riccati equation has a unique solution P (·), which does not satisfy the range condition (4.8) and therefore is not regular. Therefore, the problem is not closed-loop solvable on any [t, T ]. This example also tells us that the necessity part in Theorem 4.2 does not hold.