Reduction of lower semicontinuous solutions of Hamilton-Jacobi-Bellman equations

This article is devoted to the study of lower semicontinuous solutions of Hamilton-Jacobi equations with convex Hamiltonians in a gradient variable. Such Hamiltonians appear in the optimal control theory. We present a necessary and sufficient condition for a reduction of a Hamiltonian satisfying optimality conditions to the case when the Hamiltonian is positively homogeneous and also satisfies optimality conditions. It allows us to reduce some uniqueness problems of lower semicontinuous solutions to Barron-Jensen and Frankowska theorems. For Hamiltonians, which cannot be reduced in that way, we prove the new existence and uniqueness theorems.


INTRODUCTION
The Cauchy problem for the Hamilton-Jacobi equation with a convex Hamiltonian H in the gradient variable can be studied with connection to a calculus of variations problem. Let H * be the Legendre-Fenchel conjugate of H in its gradient variable (in our case H * is an extended real-valued function): Here v, p denotes the inner product of v and p. We will use the notation dom H * (t, x, ·) for the effective domain of H * (t, x, ·), which is a set of all v such that H * (t, x, v) = ±∞. The value function of a calculus of variations problem is defined by the formula where A [t 0 , T ], R N denotes the space of all absolutely continuous functions from [t 0 , T ] into R N . If the value function is real-valued and differentiable, it is well-known that it satisfies (1.1) in the classical sense. However, in many situations the value function is extended real-valued and merely lower semicontinuous. Then the solution of (1.1) must be defined in a nonsmooth sense in such a way that under quite general assumptions on H and g, V is the unique solution of (1.1).
We consider the following optimality conditions: (H1) H : [0, T ] × R N × R N → R is continuous with respect to all variables; (H2) H(t, x, p) is convex with respect to p for every t ∈ [0, T ] and x ∈ R N ; (H3) For any R 0 there exists C R 0 such that for all t ∈ [0, T ], x ∈ IB R and p, q ∈ R N one has |H(t, where IB R denotes a closed ball in R N of center 0 and radius R 0 and | · | denotes the Euclidean norm on R N . For a nonempty subset W of R N we define W := sup ξ∈W |ξ|. Barron-Jensen [5] and Frankowska [13] studied extended viscosity solutions to semicontinuous functions for Hamiltonians that are convex with respect to the last variable. Frankowska [13] called these solutions lower semicontinuous solutions. for all x ∈ R N , and for any (t, x) ∈ dom U, for all (p t , p x ) ∈ ∂U(t, x), one has (1.4) where ∂U(t, x) denotes subdifferential of U at (t, x).
Frankowska [13] proved that the value function V is the unique lower semicontinuous solution of (1.1) if the Hamiltonian H satisfies (H1)-(H5) and it is positively homogeneous in p, i.e. ∀ r 0 H(t, x, rp) = rH(t, x, p). Whereas g is a lower semicontinuous extended real-valued function which does not take on the value −∞. Earlier, Barron-Jensen [5,6] using quite different methods obtained similar results to those of Frankowska assuming slightly stronger conditions on the Hamiltonian H. The paper by Barles [4] provides some extensions and an informal discussion of Barron and Jensen's ideas.
From the above theorem it follows that boundedness of the sets H * (t, x, dom H * (t, x, ·)) and dom H * (t, x, ·) by an appropriately regular function λ is a condition not only sufficient but also necessary for the reduction described above. We prove Theorem 1.2 using recently derived results [22,23] concerning representations of Hamilton-Jacobi equations in the optimal control theory; see Section 3. In papers [12,16,20,26,27] one investigates existence and uniqueness of lower semicontinuous solutions of (1.1), assuming a weak growth and a weak Lipschitz continuity for the Hamiltonian H. These nonrestrictive conditions imply unboundedness of the sets dom H * (t, x, ·) and H * (t, x, dom H * (t, x, ·)). Thus, such Hamiltonian does not satisfy the condition (A). The conditions (H3) and (H4) imposed on the Hamiltonian H imply some type of boundedness of the set dom H * (t, x, ·). However, there exists a large class of Hamiltonians H, that satisfy (H1)-(H5), but do not have bounded sets H * (t, x, dom H * (t, x, ·)); see Section 2. This kind of Hamiltonians derive from the optimal control problems with unbounded control set; see Example 2.10. We also give an example of Hamiltonian H with the bounded sets dom H * (t, x, ·) and H * (t, x, dom H * (t, x, ·)) such that an appropriately regular function λ bounding them does not exist; see Example 2.7. Thus, we see that there exists a large class of Hamiltonians, that satisfy conditions (H1)-(H5), but do not fulfill the condition (A). For such kind of Hamiltonians we prove the following theorem. Theorem 1.3. Let g be a lower semicontinuous extended real-valued function which does not take on the value −∞. Assume that H satisfies (H1)-(H5). If V is the value function associated with H * and g, then V is a lower semicontinuous solution of (1.1). Moreover, if U is a lower semicontinuous solution of (1.1), then U = V on [0, T ] × R N .
We prove Theorem 1.3 using methods of the viability theory similarly to Frankowska [13]; see Sections 4-5. The difference is that we consider set-valued maps with unbounded values. Whereas in paper [13] set-valued maps with compact values are considered. These differences cause new difficulties, but we are able to deal with them. Thus, we obtain result that do not need the positively homogeneous assumption on the Hamiltonian in p. In papers [3,7,9,11,18,19,24,25] one can find similar results to Theorem 1.3. However, these results usually require that the solution is continuous and bounded and that the Hamiltonian satisfies some kind of uniform continuity conditions with constant functions c(·) and k R (·). Galbraith [16] also obtained similar results to Theorem 1.3. However, his methods need a strong Lipschitz-type assumption on the Hamiltonian with respect to the time variable. In the literature related to such results one usually assumes that the Hamiltonian is continuous in the time variable [13] or only measurable [15]. Recently, there has been a paper [8] assuming that the Hamiltonian is discontinuous with respect to the time variable in the following sense: it has everywhere left and right limits and is continuous on a set of the full measure. Finally, it is worth considering whether the positively homogeneous assumption on the Hamiltonian in p can be removed in papers [8,14,15] using the methods of this paper. At the moment we do not know if it is possible.
The outline of the paper is as follows. Section 2 contains a preliminary material and examples. In Section 3 we prove Theorem 1.2. Section 4 contains new viability and invariance theorems along with their proofs. In Section 5 using viability and invariance theorems from Section 4 we prove Theorem 1.3.

PRELIMINARY MATERIAL AND EXAMPLES
are called the effective domain, the graph and the epigraph of ϕ, respectively. We say that ϕ is proper if it never takes the value −∞ and it is not identically equal to +∞. Using properties of the Legendre-Fenchel conjugate from [30] we can prove the following proposition.
Let us define the function γ : [0, 1] × R → [0, +∞) by the formula The function γ is continuous. Moreover, for any R 0 there exists an integrable function where γ is defined as in Example 2.8. This Hamiltonian satisfies conditions (H1)-(H5) with the unbounded functions k R (·) and c(·). Indeed, let us fix t ∈ [0, 1] and x, y ∈ R. We observe that Therefore, in view of Example 2.8, we obtain our assertion. We notice that L(t, x, · ) = H * (t, x, · ) has the form The function v → L(t, x, v) is not bounded on dom L(t, x, ·) = (−γ(t, x), γ(t, x)) for γ(t, x) = 0. So this Hamiltonian does not satisfy the condition (A).
We observe that if f and l satisfy (1)-(5), then the Hamiltonian H given by satisfies (H1)-(H5). Moreover, the HamiltonianĤ given by (2.1) withf ,l is the same as in Example 2.6 and the HamiltonianȞ given by (2.1) withf ,ľ is the same as in Example 2.9.
Using the results from [28,Sect. 4] and [29,Sect. 2] one can show that if f and l satisfy (1)-(6), and g is proper and lower semicontinuous, and H is given by (2.1), then where S f (t 0 , x 0 ) denotes a set of all trajectory-measurable pairs of the control system Remark 2.11. We cannot apply Theorem 1.3 to H : [0, 1] × R → R given by the formula Because this Hamiltonian satisfies the conditions (H1)-(H2) and (H4)-(H5), but it does not satisfy the condition (H3).

REDUCTION THEOREM
Let us define the Hamiltonian H : R × R → R by the formula This Hamiltonian satisfies conditions (H1)-(H5). Moreover, L(x, · ) = H * (x, · ) has the form Let λ(x) = |x| + 1 for every x ∈ R. Then λ is Lipschitz continuous with a sublinear growth.
Barron-Jensen in [5,Prop. 3.7] proposed the following construction ofH: Observe thatH satisfies (1.5), but the function x →H(x, r, p, q) is not continuous for all (r, p, q) ∈ R × R × (0, ∞). ThereforeH does not satisfy (H1) and (H5). It means that the construction ofH proposed by Barron-Jensen is not appropriate in this case.
Our construction of the HamiltonianH is based on representations of H. The triple (A, f, l) is called a representation of H if it satisfies the following equality We observe that the triple A = [−1, 1], f (x, a) = a|x|, l(x, a) = |a| is a representation of the Hamiltonian from the example above. Moreover, the following Hamiltonian satisfies (1.5) and (H1)-(H5). Obviously, our construction of the HamiltonianH makes sense, provided that we can find an appropriately regular representation of H. In papers [22,23] one proved that this kind representations always exist.
Now we explain the reason that our construction of HamiltonianH gives the expected results in contrast to the construction of Barron-Jensen. We know that Hamiltonian H from the above example satisfies (H1)-(H5). Therefore, by Theorem 2.3 Lagrangian L satisfies (L6). It means that L is lower-Lipschitz continuous. On the other hand, our example shows that L is not upper-Lipschitz continuous and, what is more, it is not upper semicontinuous. Indeed, L is not upper semicontinuous, because lim sup i→∞ L (1/i, 1/i) = 1 0 = L(0, 0). It is not difficult to see that the main reason for ineffectiveness of the construction ofH proposed by Barron-Jensen is the lack of upper-Lipschitz continuity of L. If the triple (A, f, l) is a faithful representation of H, then the functions f and l are Lipschitz continuous. In particular, the function l is lower/upper-Lipschitz continuous. Due to that, our construction of HamiltonianH gives the expected result.
In the proof of Theorem 1.2 we need slightly modified Theorems 3.1 and 3.4 from [22]. We use the first theorem to prove (B) ⇒ (A) and the second theorem to prove (A) ⇒ (B). Proof. Assume that H satisfies (H1)-(H5) with c(·), k R (·). Let λ be as in the condition (A) with ϑ(·), ζ R (·). We define e :

Theorem 3.1. Assume that H satisfies the condition (A) from Theorem 1.2. Then there exists a representation (IB, f, l) of H such that IB is a closed unit ball in
in the Steiner selection and ω(t, x) := 2λ(t, x) + 1. By [22,Section 5] the function e(·, ·, ·) is well defined and continuous. Moreover, e(t, · , a) is Next, we define the functions f and l as components of the function e, i.e., e = ( f, l). Then, by (3.1) and [22,Prop. 5.7], the triple (IB, f, l) is a representation of H. It is not difficult to prove that the functions f and l satisfy (R1)-(R3). Proof. Let (IB, f, l) be a representation of H and f, l satisfy (R1)-(R3) with C(·), K R (·). Using (R1)-(R3) one can show that H satisfies (H1)-(H5). We define a simplex in R N+1 by Obviously, the set ∆ is compact. Moreover, we define the set by := IB N+1 × ∆. We notice that the set is compact. The functions , Ð are defined for every t ∈ [0, T ], x ∈ R N and = (a 0 , . . . , a N , α 0 , . . . , α N ) ∈ IB N+1 × ∆ = by the formulas: x, a n ).

VIABILITY AND INVARIANCE THEOREMS
In this section we present viability and invariance theorems with unbounded differential inclusions. Working with unbounded differential inclusions we encounter new problems which we solve in this section. Let π K (·) be a projection of R N onto a nonempty closed convex subset K of R N . We denote by χ K (·) an indicator function of a subset K of R N .

Invariance Theorem.
We start by formulating the invariance theorem: . Let U be a proper and lower semicontinuous function satisfying the following condition: Then for every t 0 ∈ [0, T ) and every absolutely continuous function (x, u) We show that the standard methods fail when used in trying to prove the Theorem 4.1.
In the first method we takeQ(s, We notice thatẏ(t) ∈Q(y(t)) for a.e. t ∈ [t 0 , T ] and ϕ is absolutely continuous with ϕ(T ) = 0. The latter and (4.1) implyφ(t) k ϕ(t) for a.e. t ∈ [t 0 , T ], provided thatQ is k-Lipschitz with respect to all variables; see [26,Thm. 4.2]. Hence, using Gronwall's lemma, we have We cannot apply this method to prove Theorem 4.1, because we do not assume that Q is Lipschitz continuous with respect to the time variable.
The second method is similar to the first one, but does not require the assumption that Q is Lipschitz continuous with respect to the time variable. LetQ(t, epi U(t, ·)) for all t ∈ [t 0 , T ]. Our assumptions imply that ϕ(·) is lower semicontinuous with ϕ(T ) = 0. One can prove, due to (4.1), that dϕ(t)(1) k(t) ϕ(t) for a.e. t ∈ [t 0 , T ]; see [1, p. 394].
The latter inequality yields ϕ ≡ 0, provided that dϕ(t)(1) < ∞ for every t ∈ [t 0 , T ]; see [17, p. 42]. Unfortunately, without additional assumptions, we cannot say whether the latter condition is true. So, we cannot apply this method to prove Theorem 4.1.
The third method is based on the reduction set-valued map with unbounded values to the case of set-valued map with compact values. This kind of method was used by author in the work [20].
As we mentioned earlier, Frankowska proved the invariance theorem for set-valued maps with compact values; see [13,Thm. 3.3]. In the proof of this theorem, she used the parametrization Θ of the set-valued mapQ with the parameter set IB. Indeed, in view of [2, Thm. 9.6.2] there exists a continuous single-valued Then, by Filippov Theorem, there exists a measurable function a : a(t)) for a.e. t ∈ [t 0 , T ]. We notice that a(·) is also integrable, since it is bounded. Then we can choose a sequence of continuous functions a i : [t 0 , T ] → IB converging to a(·) in L 1 -spaces. Let f i (t, x, u) = Θ(t, x, u, a i (t)) for all t ∈ [t 0 , T ], x ∈ R N , u ∈ R. We observe that f i is continuous and f i (t, ·, ·) is k R (t)-Lipschitz for a.e. t ∈ [t 0 , T ]. By Picard-Lindelöf Theorem, there exist unique solutions (ẋ i ,u i )(t) = f i (t, x i (t), u i (t)) with (x i , u i )(T ) = (x(T ), u(T )). In view of Viability Theorem, we have (x i , u i )(t) ∈ epi U(t, ·) for all t ∈ [t 0 , T ]. Passing to the limit as i → ∞, we obtain (x, u)(t) ∈ epi U(t, ·) for all t ∈ [t 0 , T ]. Obviously, we cannot use this method to prove Theorem 4.1, because parametrization of the set-valued map Q with the compact parameter set does not exist. However, there is a parametrization of the set-valued map Q with the unbounded parameter set: Additionally, if Q satisfies (Q6), then we have It turns out that the lack of compactness of the parameter set causes a serious problem.
x ∈ R N , then by Filippov Theorem there exists a measurable function a : [t 0 , T ] → R N+1 such that (ẋ,u)(t) = Θ(t, x(t), a(t)) for a.e. t ∈ [t 0 , T ]. Obviously, the measurable function a(·) may not be integrable. Therefore, we cannot approximate a(·) by continuous functions in L 1 -spaces, which prevents the continuation of the Frankowska method. It turns out that the above approximation problem can be solved using an extra-property. Property (P2) in Theorem 4.2 is called the extra-property. We discovered this property by researching the regularities of the value functions in [23]. We observe that Parametryzation Theorem 9.6.2 in the monograph of Aubin-Frankowska [2] does not contain the property of type (P2). Now, we show how this extra-property can be used to solve the approximation problem.
is an absolutely continuous function, (ẋ,u)(·) is an integrable function. Thus, a(·) is also integrable. Therefore, a(·) can be approximated by continuous functions in L 1 -spaces. Summarizing, the method of Frankowska can be applied to the unbounded case, provided that we have appropriately regular parametrization of the set-valued map. We define the functionΘ : x, U(t, x)) for all (t, x, u) ∈ epi U. Therefore, in view of (4.1), ∀ a ∈ R N+1 , n t + (n x , n u ),Θ(t, x, u, a) 0.

Viability Theorem.
We start by formulating the viability theorem:  x, U(t, x)), there exist (t k , x k ) → (t, x) and α k → 0, exists v k ∈ dom L(t k , x k , ·) such that n t + v k , n x − n u L(t k , x k , v k ) α k for all k ∈ N.
In the proof of Theorem 4.3 we construct a viable trajectory using Euler's broken lines and methods from the monography of Cesari [10], similarly to Plaskacz-Quincampoix in the proof of [26,Theorem 3.19]. There are two significant differences between proofs of holds for each h satisfying Proof. First, we show that (4.11) is well-defined. Indeed, if f = y − w, then Next, we show that (4.10) holds. By multiplying both sides of (4.11) by | f − (y − w)| 2 and moving the expressions from right to left we obtain We transform the above inequality as follows By multiplying both sides of the above inequality by h, we get By adding |y − w| 2 to both sides of the above inequality, we have We observe that the above inequality implies (4.10).  4.6. In Theorem 4.5 the condition (L5) is not required. Moreover, the conditions (4.6) and (4.12)

Theorem 4.5 (Local Viability Theorem). Assume that L satisfies (L1)-(L4). Let U be a proper and lower semicontinuous function satisfying the following condition:
C R for every t, u ∈ R, x ∈ R N . Since functions U and L are proper and lower semicontinuous, there exists a constant STEP 1. Definition and properties of an ε ε ε-approximate solution. Let ε and t ε satisfy We say that a family x 0 , u 0 ). We say that a triple [t 0 , t ε ), y(·), ∑ is an ε-approximate solution if the following inequality holds and for all j ∈ J there exist f j , w j ,w j such that We show that an ε-approximate solution satisfies the following inequality Indeed, suppose that y(t j ) ∈ epi U. Then by (iii), (i) we get w j = y(t j ), |w j −y(t)| = (t −t j )| f j |. The latter, together with (iv), implies (4.16). Now, suppose that y(t j ) / ∈ epi U. Then by (iii), (4.15) we get |w j − y(t j )| ε. The latter, together with (v), implies (4.16).

HAMILTON-JACOBI-BELLMAN THEORY
We define the functional Γ[ · ] for any x(·) ∈ A [t 0 , T ], R N by the formula Then the value function for any (t 0 , x 0 ) ∈ [0, T ] × R N is given by Theorem 5.1. Assume that L satisfies (L1)-(L5) and g is a proper, lower semicontinuous function. Let V be the value function associated with L and g. Then we have the following.
(a) V is a proper, lower semicontinuous function.
Since L is proper and lower semicontinuous (due to (L1)-(L2)), there exists a constant k R > 0 such that x ∈ IB R , v ∈ R N . It means that the condition (L6) is not necessary in the above theorem.
In the Subsection 5.1 we show that the value function V, associated with H * and g, is a lower semicontinuous solution of (1.1), provided that H satisfies (H1)-(H4) and g is proper and lower semicontinuous. We will see that the condition (H5) is not required to prove the above fact. In the Subsection 5.2, assuming additionally that H satisfies (H5), we show that the value function V is a unique lower semicontinuous solution of (1.1). It turns out that the condition (H5) is necessary in some sense to prove the uniqueness result. More precisely, one can propose two different lower semicontinuous solutions of (1.1) with the Hamiltonian H satisfying (H1)-(H4); see [21].
Theorem 5.6 (Existence). Assume that H satisfies (H1)-(H4) and g is proper and lower semicontinuous. If V is the value function associated with H * and g, then V is a lower semicontinuous solution of (1.1).

Uniqueness of Lower Semicontinuous Solutions.
Proposition 5.7. Let U be a proper and lower semicontinuous function. Assume that H satisfies (H1)-(H2). If for every (t, x) ∈ dom U ∩ (0, T ] × R N and every (p t , p x ) ∈ ∂U(t, x) the inequality (1.4) holds, then the condition (4.1) also holds.
Due to relation between normal cones and subdifferentials, from Subsection 2.1, we get (n t /|n u |, n x /|n u |) ∈ ∂U(t, x).
By multiplying both sides of the above inequality by −|n u |, we get n t + n x , v − n u L(t, x, v) 0, which completes the proof of (4.1) in this case. Let (n t , n x , 0) ∈ N epi U (t, x, U(t, x)). In view of Lemma 2.5 there exist (t k , x k ) → (t, x) and (n t k , n x k , n u k ) → (n t , n x , 0) satisfying n u k < 0 and (n t k , n x k , n u k ) ∈ N epi U (t k , x k , U(t k , x k )) for all k ∈ N. Then, by the definition of a normal cone, for all k ∈ N we have n t k /|n u k |, n x k /|n u k |, −1 ∈ N epi U (t k , x k , U(t k , x k )). Due to relation between normal cones and subdifferentials, from Subsection 2.1, we get (n t k /|n u k |, n x k /|n u k |) ∈ ∂U(t k , x k ), for all k ∈ N. The latter, together with (1.4), implies that, for all large k ∈ N, −n t k /|n u k | + H(t k , x k , −n x k /|n u k |) 0. Since H(t, x, · ) = L * (t, x, · ), we have, for all v ∈ dom L(t k , x k , ·) and all large k ∈ N, −n t k /|n u k | + −n x k /|n u k |, v − L(t k , x k , v) 0. By multiplying both sides of the above inequality by −|n u k |, we get (5.10) n t k + n x k , v − n u k L(t k , x k , v) 0, for all v ∈ dom L(t k , x k , ·) and all large k ∈ N. Let us fixv ∈ dom L(t, x, ·). By (L3) there exists v k →v such that L(t k , x k , v k ) → L(t, x,v). Hence v k ∈ dom L(t k , x k , ·) for all large k ∈ N. If we set v := v k in the inequality (5.10) and pass to the limit as k → ∞, then, n t + n x ,v 0, which completes the proof of (4.1).