OPEN-LOOP AND CLOSED-LOOP SOLVABILITIES FOR STOCHASTIC LINEAR QUADRATIC OPTIMAL CONTROL PROBLEMS OF MARKOVIAN REGIME SWITCHING SYSTEM∗

This paper investigates the stochastic linear quadratic (LQ, for short) optimal control problem of Markovian regime switching system. The representation of the cost functional for the stochastic LQ optimal control problem of Markovian regime switching system is derived by the technique of Itô’s formula with jumps. For the stochastic LQ optimal control problem of Markovian regime switching system, we establish the equivalence between the open-loop (closed-loop, resp.) solvability and the existence of an adapted solution to the corresponding forward-backward stochastic differential equation with constraint. (i.e., the existence of a regular solution to Riccati equations). Also, we analyze the interrelationship between the strongly regular solvability of Riccati equations and the uniform convexity of the cost functional. Finally, we present an example which is open-loop solvable but not closed-loop solvable. Mathematics Subject Classification. 49N10, 93E20. Received March 22, 2020. Accepted June 12, 2021.


Introduction
Linear-quadratic (LQ) optimal control problem plays an important role in control theory. It is a classical yet fundamental problem in the fields of control theory. In the past few decades, both the deterministic and stochastic linear quadratic (LQ) control problems have been widely studied. Stochastic LQ optimal control problem was first carried out by Kushner [12] via dynamic programming method. Later, Wonham [27] studied the generalized version of the matrix Riccati equation arose in the problems of stochastic control and filtering. Using functional analysis techniques, Bismut [1] proved the existence of the Riccati equation and derived the existence of the optimal control in a random feedback form for stochastic LQ optimal control with random coefficients. Tang [24] discussed the existence and uniqueness of the associated stochastic Riccati equation for general stochastic LQ optimal control problems with random coefficients and state control dependent noise via the method of stochastic flow, which can be applied to Bismut and Peng's long-standing open problems. Moreover, Tang provided a rigorous derivation of the interrelationship between the Riccati equation and the stochastic Hamilton system as two different but equivalent tools for the stochastic LQ problem. For more details on the progress of the stochastic Riccati equation, interested readers may refer to [8-11, 21, 25].
Under some mild conditions on the weighting coefficients in the cost functional, such as positive definite quadratic weighting control martix, and so on, the stochastic LQ optimal control problems can be solved elegantly via the Riccati equation approach, see Chapter 6 of [30]. Chen et al. [2] was the first to start the pioneering work of stochastic LQ optimal control problems with indefinite quadratic weighting control matrix, which turns out to be useful in solving the continuous-time mean-variance portfolio selection problems. Since then, there has been an increasing interest in the so-called indefinite stochastic LQ optimal control, see, for example, Chen and Yong [3], Li and Zhou [14], Li et al. [15,16], and so on.
Another extension to stochastic LQ optimal control problems involves random jumps in the state systems, such as Poisson jumps or the regime switching jumps. Wu and Wang [28] considered the stochastic LQ optimal control problems with Poisson jumps and derived the existence and uniqueness of the deterministic Riccati equation. Using the technique of completing squares, Hu and Oksendal [4] discussed the stochastic LQ optimal control prblmlem with Poisson jumps and partial information. Existence and uniqueness of the stochastic Riccati equation with jumps and connections between the stochastic Riccati equation with jumps and the associated Hamilton systems of stochastic LQ optimal control problem were also presented. Yu [31] investigated a kind of infinite horizon backward stochastic LQ optimal control problems and differential game problems under the jump-diffusion model state system. Li et al. [13] solved the indefinite stochastic LQ optimal control problem with Poisson jumps.
The stochastic control problems involving regime switching jumps are of interest and of practical importance in various fields such as science, engineering, financial management and economics. The past few years have witnessed a dramatically increasing interest in financial market models whose key parameters, such as interest rate, stocks return and volatility, are modulated by Markov processes. In particular, one could face two market regimes in financial markets, one of which stands for a bull market with price increase, while the other for a bear market with price drop. One such formulation is the so-called regime switching model, where the market parameters depend on market modes that switch among a finite number of regimes. More recently, the regime switching models and related topics have been extensively studied in the areas of stochastic controls as well as their financial applications, see, for examples, [14-16, 19, 29, 35-39]. Ji and Chizeck [6,7] formulated a class of continuous-time LQ optimal controls with Markovian jumps. Zhang and Yin [34] developed hybrid controls of a class of LQ systems modulated by a finite-state Markov chain. Li and Zhou [14], Li et al. [15,16] introduced indefinite stochastic LQ optimal controls with regime switching jumps. Liu et al. [17] considered near-optimal controls of regime switching LQ problems with indefinite control weight costs.
Recently, Sun and Yong [22] investigated the two-person zero-sum stochastic LQ differential games. It was shown in Sun and Yong [22] that the open-loop solvability is equivalent to the existence of an adapted solution to a forward-backward stochastic differential equation (FBSDE, for short) with constraint and closed-loop solvability is equivalent to the existence of a regular solution to the Riccati equation. As a continuation work of Sun and Yong [22], Sun et al. [20] considered the open-loop and closed-loop solvabilities for stochastic LQ optimal control problems. Moreover, the equivalence between the strongly regular solvability of the Riccati equation and the uniform convexity of the cost functional was established. This naturally calls for us to study the open-loop and closed-loop solvabilities within the framework of regime switching jumps. Moreover, if there is only one regime jump state in our model and then it can be reduced to that model of Sun et al. [20]. The aim of this paper is to further develop the results in Sun et al. [20] to the case of stochastic LQ optimal control problems with regime switching jumps. Although the only difference between our model and their one is the regime switching jumps, it is not easy for one to obtain the similar results. The first difficulty we encounter is the uniformly convexity characterization of the cost functional J(t, x, i; u(·)). In Sun et al. [20], however, it is hard to understand the relationships of the uniformly convexity among J 0 (t, x; u(·)), J 0 (t, 0; u(·)) and J(t, x; u(·)), because they just briefly mentioned "uniformly convexity" and did not clearly prsent readers its fundamental definition. Thus, one cannot directly deal with the similar control works including the regime switching case. Therefore, in our work, we definitely and strictly introduce the definition of "uniformly convexity" from Zalinescu [32,33]. Then, using this definition, we prove that the uniformly convexity of the cost functional J(t, x, i; u(·)) is equivalent to the positive definiteness of the operator M 2 (t, i) (see (3.2) or (3.8)). From this equivalence, we can further obtain the relationships of the uniformly convexity among J 0 (t, x, i; u(·)), J 0 (t, 0, i; u(·)) and J(t, x; u(·)). The second difficulty in our model is to prove the equivalence between the closed-loop solvability and the existence of regular solution to Riccati equation. In the model without regime switching jumps, this equivalence was studied in Sun and Yong [22] and Sun et al. [20]. In order to prove the equivalence, Sun and Yong [22] constructed two matrix valued stochastic processes X(·) and Y(·) (see Eqs. (5.16) and (5.17) in [22]) and set P (s) := Y(s)X(s) −1 . Then the authors prove that P (s) satisfies the Riccati equation. In our case, note that solution P (s, i) of Riccati equation (5.2) depends on state α(s) = i. However, Y(s)X(s) −1 may not equal to P (s, α(s)) even though the evolution of the matrix valued processes Y(s) and X(s) depend on the Markov chain α. Thus, if we adopt the technique in Sun and Yong [22], we may not parallelly derive P (s, α(s)). Therefore, the method adopted in [22] no longer works in our model. Wang [26] provided a new method to avoid the procedures of defining P (s) := Y(s)X(s) −1 for the model without regime switching. The method adopted by [26] can also be applied for the model with regime switching. In this paper, we develop a different method to overcome the above mentioned difficulty of defining P (s, α(s)) in our model arisen in the proof of Theorem 5.2. We think that the definition of P (s, α(s)) provides additional insight to our proof. In fact, if we interpret the solution of (5.11) as the flow of the initial state (t, x, i) and define the matrix valued stochastic processes X(s; t, i) and Y (s; t, i) in (5.13), then we can prove that Y (s; t, i) = P (s, α(s))X(s; t, i) under the definition of T ], we finally derive the expression of P (s, α(s)) = Y (s; t, i)X(s; t, i) −1 . Furthermore, our method is more general and also can be applied to solving the problems in [20,22] with random coefficients.
It is worthy to mention that Lv et al. [18] established the characterization of closed-loop optimal control when the coefficients are adapted processes with respect to the filtration generated by the Brownian motion, and Huang and Yu [5] derived some equivalent conditions for closed-loop optimal control for deterministic case. If there is only one regime jump state in our model, our characterization of the closed-loop optimal control becomes a special case of [18]. Furthermore, the equivalence between the uniform convex and the strongly regular solution of Riccati equation (Thm. 6.3) implies that R is uniformly positive, which is the same as the statement (ii) of Theorem 4.6 of [5].
The first main contribution of our paper is to develop a new method for obtaining the representation of the cost functional for the stochastic LQ optimal control problem with regime switching jumps. In Sun et al. [20], the representation of the cost functional, which is the summary results of Yong and Zhou [30], is fundamental to prove the above equivalence. Unlike the techniques of analysis established in [20,30], our method for deriving the representation of the cost functional is mainly based on the technique of Itô's formula with jumps. The second main contribution of our paper is to adopt the stochastic flow theory for proving the equivalence between the closed-loop solvability and the existence of regular solution to the Riccati equation. Due to incorporating the regime switching jumps, the method applied in Sun et al. [20] no longer works for proving the equivalence between the closed-loop solvability and the existence of regular solution to the Riccati equation when one studies the stochastic LQ optimal control problem with regime switching jumps.
The rest of the paper is organized as follows. Section 2 introduces some useful notation, collect some preliminary results and state the stochastic LQ optimal control problem with regime switching jumps. Section 3 is devoted to deriving the representation of the cost functional using the technique of Itô's formula with jumps. In Sections 4 and 5, we prove the equivalence between the open-loop (closed-loop) solvability and the existence of an adapted solution to the corresponding FBSDE with constraint (i.e., the existence of a regular solution to the Riccati equation) for the stochastic LQ optimal control problem of Markovian regime switching system. The equivalence between the strongly regular solvability of the Riccati equation and the uniform convexity of the cost functional is established in Section 6. In the last section, we present an example which is open-loop solvable but not closed-loop solvable.

Preliminaries and model formulation
Let (Ω, F, F, P) be a complete filtered probability space on which a standard one-dimensional Brownian motion W = {W (t); 0 t < ∞} and a continuous time, finite-state, Markov chain α = {α(t); 0 t < ∞} are defined, where F = {F t } t 0 is the natural filtration of W and α augmented by all the P-null sets in F. In the rest of our paper, we use the following notation: For an S n -valued function F (·) on [t, T ], we use the notation F (·) 0 to indicate that F (·) is uniformly positive definite on [t, T ], i.e., there exists a constant δ > 0 such that a.e. s ∈ [t, T ]. Now we start to formulate our system. We identify the state space of the chain α with a finite set S := {1, 2 . . . , D}, where D ∈ N. Without loss of generality, we assume that the Markov chain is irreducible. To specify the statistical or probabilistic properties of the chain α, we define the generator λ(t) := [λ ij (t)] i,j=1,2,...,D of the chain under P. Here, for each i, j = 1, 2, . . . , D, λ ij (t) is the constant transition intensity of the chain from state i to state j at time t. Note that λ ij (t) ≥ 0, for i = j and D j=1 λ ij (t) = 0, so λ ii (t) ≤ 0. In what follows, for each i, j = 1, 2, . . . , D with i = j, we suppose that λ ij (t) > 0, so λ ii (t) < 0. For each fixed j = 1, 2, . . . , D, let N ij (t) be the number of jumps from state i into state j up to time t and set Then for each i, j = 1, 2, . . . , D, the term N ij (t) is an (F, P)-martingale. Consider the following controlled Markovian regime switching linear stochastic differential equation on the finite horizon [t, T ]: where A(·, ·), B(·, ·), C(·, ·), D(·, ·) are given deterministic matrix-valued functions of proper dimensions, and b(·), σ(·) are vector-valued F-progressively measurable processes. In the above, X u (· ; t, x, i), valued in R n , is the state process, and u(·), valued in R m , is the control process. u(·) is called an admissible control on [t, T ], if it belongs to the following Hilbert space: For any admissible control u(·), we consider the following general quadratic cost functional: x, i; u(·)) E G(T, α(T ))X u (T ; t, x, i) + 2g(T, α(T )), X u (T ; t, x, i) where G(T, i) is a symmetric matrix, Q(·, i), S(·, i), R(·, i), i = 1, 2, . . . , D are deterministic matrix-valued functions of proper dimensions with Q(·, i) = Q(·, i), R(·, i) = R(·, i); g(T, ·) is allowed to be an F T -measurable random variable and q(·, ·), ρ(·, ·) are allowed to be vector-valued F-progressively measurable processes. The following standard assumptions will be in force throughout this paper.
(H2) The weighting coefficients in the cost functional satisfy the following: for each i ∈ S G(T, i) ∈ S n , Q(·, i) ∈ L 1 (0, T ; S n ), S(·, i) ∈ L 2 (0, T ; R m×n ), R(·, i) ∈ L ∞ (0, T ; S m ), Now we state the stochastic LQ optimal control problem for the Markovian regime switching system as follows.
Similar to Sun et al. [20], we introduce the following definitions of open-loop (closed-loop) optimal control.
The set of all closed-loop strategies (Θ(·, where X * (·) is the strong solution to the following closed-loop system: Remark 2.4. (i) The inequality (2.5) in defining the closed-loop optimal strategy can be replaced by the following inequality: In fact, using the similar method of Sun and Yong [23] (see Prop. 2.1.5), we can prove the equivalence of (2.5) and (2.7). Therefore, we can take (2.7) for defining the closed-loop optimal strategy.
(ii) We emphasize that in the definition of closed-loop optimal strategy, (2.5) must be true for all (x, i) ∈ R n × S. One sees that if (Θ * (·, α(·)), v * (·)) is a closed-loop optimal strategy of problem (M-SLQ) on [t, T ], then the outcome u * (·) ≡ Θ * (·, α(·))X * (·) + v * (·) is an open-loop optimal control of Problem (M-SLQ) for the initial pair (t, X * (t), α(t)). Hence, the existence of closed-loop optimal strategies implies the existence of open-loop optimal controls. However, the existence of open-loop optimal controls does not necessarily imply the existence of a closed-loop optimal strategy, see an example in section 7 for more details.
To simplify the notation of our further analysis, we introduce the following forward-backward stochastic differential equation (FBSDE for short) on a finite horizon [t, T ]: The solution of the above FBSDE system is denoted by ( x, i)) denoting the solution of the above FBSDE. If b(·) = σ(·) = q(·, ·) = g(·, ·) = 0, the solution of the above FBSDE system is denoted by

Representation of the cost functional
In this section, we present a representation of the cost functional for Problem (M-SLQ), which plays a crucial role in the study of open-loop/closed-loop solvability of Problem (M-SLQ). Unlike the method adopted in Yong and Zhou [30], we derive the representation of the cost functional using the technique of Itô's formula with jumps.
Thus we complete the proof.
From the representation of the cost functional, we have the following simple corollary.

Open-loop solvabilities
We first present the equivalence between the open-loop solvability and the corresponding forward-backward differential equation system. where (X u (· ; t, x, i), Y u (· ; t, x, i), Z u (· ; t, x, i), Γ u (· ; t, x, i)) is an adapted solution to FBSDE (2.8).
Proof. By definition, u(·) is an open-loop optimal control if and only if the following inequality holds: From Corollary 3.4, we have Note that if u(·) happens to be an open-loop optimal control of Problem (M-SLQ), then the stationarity condition (4.1) holds. We call (2.8), together with the stationarity condition (4.1), the optimality system for the open-loop optimal control of Problem (M-SLQ).
Next we investigate the relationships between open-loop solvability and uniform convexity of the cost functional. We first introduce the definition of uniform convexity, which is from Zalinescu [33] or [32].  for some > 0.
Proof. From Proposition 3.1, we see that for any u(·), v(·) ∈ U[t, T ] and ∈ [0, 1], Thus, it follows from the definition of uniformly convex that the cost functional J(t, x, i; u(·)) is uniformly convex if and only if there exists h : R + → R + with h(t) > 0 for t > 0 and h(0) = 0 such that which is equivalent to M 2 (t, i) I for some > 0. From Proposition 3.1, we have J 0 (t, 0, i; u(·)) = M 2 (t, i)u, u .
Observing the definition of ρ(s) and substituting the above equation into (5.28), we get the desired result of equation (5.5).

Uniform convexity of the cost functional and the strongly regular solution of the Riccati equation
We first present some properties for the solution to Lyapunov equation, which play a crucial role on establishing the equivalence between uniform convexity of the cost functional and the strongly regular solution of the Riccati equation.
In particular, by taking u(·) = 0 in the above, we obtain and the second inequality therefore follows.
Now we further prove the equivalence between the uniform convexity of the cost functional and the strongly regular solution of Riccati equation. Proof. (i) ⇒ (ii). Let P 0 (·, ·) be the solution of Applying Proposition 6.2 with Θ(·) = 0, we obtain that R(s, i) λI, P 0 (s, i) γI, a.e. s ∈ [t, T ].

Example
In this section, we present an example of M-SLQ problem which is open-loop solvable but not closed-loop solvable.
Noting that Q(·, i) = 0, R(·, i) = 0, D(·, i) = 0 for every i ∈ S and 0 † = 0, it is easy to see that Riccati equation ( Clearly, the unique solution of the above ordinary differential equation system is