Relationship Between Maximum Principle and Dynamic Programming in presence of Intermediate and Final State Constraints

In this paper, we consider a class of optimal control problems governed by a diﬀerential system. We analyze the sensitivity relations satisﬁed by the co-state arc of the Pontryagin maximum principle and the value function that associates the optimal value of the control problem to the initial time and state. Such a relationship has been already investigated for state-constrained problems under some controllability assumptions to guarantee Lipschitz regularity property of the value function. Here, we consider the case with intermediate and ﬁnal state constraints, without any controllability assumption on the system, and without Lipschitz regularity of the value function. Because of this lack of regularity, the sensitivity relations cannot be expressed with the sub-diﬀerentials of the value function. This work shows that the constrained problem can be reformulated with an auxiliary value function which is more regular and suitable to express the sensitivity of the adjoint arc of the original state-constrained control problem along an optimal trajectory. Furthermore, our analysis covers the case of normal optimal solutions, and abnormal solutions as well.


Introduction
The aim of this paper is to study the relationship between the Pontryagin maximum principle and dynamic programming for control problems in presence of final and/or intermediate state constraints. Let T > 0 be a fixed final time horizon, let t 0 = 0 < t 1 < · · · < t m = T be real numbers (with a fixed integer m ≥ 1), U be a compact subset of R r (with r ≥ 1). Our control problem is formulated as follows: Minimize ϕ(x(T )) + T t (s, x(s), u(s)) ds subject toẋ(s) = f (s, x(s), u(s)), a.e. s ∈ (t, T ), (1.1a) x(t) = x, (1.1b) u(s) ∈ U, a.e. s ∈ (t, T ), (1.1c) g i (x(t i )) ≤ 0, i = 1, . . . , m (1.1d) where the dynamics f : [0, T ] × R d × U → R d (with d ≥ 1), the distributed cost : [0, T ] × R d × U → R, the final cost ϕ : R d → R, the constraints g i : R d → R are smooth functions (the precise nature of these functions is unimportant at this juncture, the assumptions will be made precise in the next section). An admissible control function is a measurable function u : [0, T ] → R m that satisfies (1.1c). An admissible process is a pair of functions (x, u) where u is an admissible control function and x is an absolutely continuous solution of (1.1a)-(1.1b) that satisfies the constraints (1.1d).
The study of unconstrained continuous non-linear control problems (i.e. when g i ≡ 0) has provided the foundation of numerous advances in control theory over many decades. In that setting it is very well known by now (see [8,22,25] and the references therein) that any optimal process (x, u) satisfies the so-called Pontryagin maximum principle (PMP) stating the existence of a dual arc p satisfying the following relations: where H(s, x, u, p) := −f (s, x, u) · p − (s, x, u), for every (s, x, u, p) ∈ [0, T ] × R d × U × R d . The PMP provides a necessary condition for any optimal process. On the other hand, the dynamic programming approach provides valuable information on the value function V that associates to every initial condition (t, x), the optimal value V (t, x) of the control problem. Under standing assumptions, the value function is guaranteed to be Lipschitz continuous everywhere, see for instance [3,25]. It is also well known that the value function is characterized as the unique viscosity solution of a corresponding partial differential equation (PDE), called Hamilton-Jacobi-Bellman (HJB) equation: where ∂ t V and D x V represent respectively the derivative with respect to time and to space (in the viscosity sense), and where H(t, x, p) := max u∈U H(t, x, u, p) for every (t, x, p) ∈ [0, T ] × R d × R d . Semi-smooth analysis and viscosity theory provide adequate tools to analyse the properties of solutions for HJB equations, we refer to [3,25] and the references therein. The connection between the dynamic programming approach and the Pontryagin maximum principle is known since the early work of Bellman and is described in introductory works of optimal control theory with the following relation: emphasizing the fact that the adjoint arc is a measure of the sensitivity of the optimal cost with respect to perturbations around the optimal trajectory. Combining (1.2a) and (1.3), we obtain: (H(t, x(t), p(t)), p(t)) = ∇ t,x V (t, x(t)). (1.4) When the value function is C 1 , and all the data are smooth enough, the proof of the relations (1.3)-(1.4) is quite straightforward. However, the assumption on the regularity of the value function does not hold in general even for simple cases [4,13,22].
When the value function is only Lipschitz continuous, and in situations where the costate is unique, relations (1.3)-(1.4) can be extended by using the framework of the super-differential and the sub-differential [26] (we refer also to [3] where different proofs for relations (1.3)-(1.4) are provided).
If the functions f , and ϕ are not supposed continuously differentiable in x-variable, the co-state satisfying the PMP may not be unique. In these situations, a natural question arises regarding the existence of some co-state arc that satisfies an extension of the sensitivity relations (1.3)- (1.4).
In [9], Clarke and Vinter proved that, for a very large class of non-smooth free endpoint problems, it is possible to choose a co-state function p such that p(t) ∈ ∂ x V (t, x(t)) a.e. on (0, T ), p(0) ∈ ∂ x V (0, x(0)), and p(T ) ∈ ∂ x V (T, x(T )). (1.5) These inclusions hold for control problems with a Lipschitz continuous value function, and extend the relation (1.3) by using the generalized gradient of the value function (which is well defined for Lipschitz continuous functions). In [24], Vinter established also a generalization of (1.4) stating the existence of a co-state satisfying: (H(t, x(t), p(t)), p(t)) ∈ ∂ t,x V (t, x(t)) for all t ∈ [0, T ]. (1.6) The link (1.6) involves the generalized gradient of V in both variables and asserts a property which holds everywhere on the time interval [0, T ]. Relations (1.5) and (1.6) do not entail each other. Moreover, examples are given in [9,25] showing that, in some cases, there are a number of possible choices of co-state arcs associated with the same optimal control problem, but not all of them satisfy both sensitivity relations (1.5) or (1.6) (some of the co-state arcs do not satisfy any of the two relations, while some other co-state satisfy one relation and not the other). The proof for (1.5) (in [9]) and the one for (1.6) (in [24]) are both based on the construction of a new optimal control problem, related to the original one, but involving additional control variables, and for which the original minimizer, is still a minimizer of the new problem. Necessary conditions for optimality are then applied to the new optimal control problem which has, by construction, a richer class of variations. New information emerge from these necessary conditions and lead to the generalized sensitivity relation. It should be noticed also that the original proof of (1.5) (in [9]) is more complex than that of (1.6) and requires the approximation of the dynamic equation by a new 'impulsive' system.
In [5,14], the generalization of sensitivity relations are analyzed for a general non-smooth differential inclusion setting. In particular in [5], the authors establish the existence of a co-state that satisfies the two sensitivity relations (1.5)-(1.6). Note that in [5] the problem is in presence of state constraints, but some controllability assumptions are considered so that the value function is locally Lipschitz continuous. Moreover, the proof of (1.6) given in [5] is based on the approximation of the dynamic equation by an impulsive system.
The introduction of state constraints into non-linear control problems fundamentally impacts their analysis. Indeed, unless some strong controllability assumptions are satisfied, the value function is no longer guaranteed to be Lipschitz continuous. The connection between the value function of a given control problem and corresponding Hamilton-Jacobi-Bellman equation, in the presence of state constraints, is a central issue in control theory, which has been attacked with methods coming from viscosity solutions theory as well as from non-smooth analysis and viability theory. It is well known that, unless some compatibility condition between constraints and dynamics does hold, the value function has not enough regularity, or can fail to be the unique constrained viscosity solution of a HJB equation. Several works have been devoted to the analysis of the regularity of the value function when the control problem is in presence of state constraints, we refer for instance to [14][15][16][17][18]23] and the references therein.
The link between the dynamic programming principle and Pontryagin maximum principle has been established for pointwise state constrained control problems in [5,14] under some controllability assumptions under which the value function is Lipschitz continuous. The case with final state constraints has been also investigated in [9,24] under the assumption of Lipschitz regularity of the value function in a "tube" around the trajectory.
In the present work, we are interested in the case when the control problem is in presence of final and/or intermediate constraints. Such problems have already been considered in the literature and optimality conditions have been derived in the form of Pontryagin maximum principle, see for instance [10][11][12]. Moreover, in this setting, and without controllability assumption, the value function is in general merely lower semicontinuous, it may have infinite values and might even take finite values only on a set having empty interior. So, it is not clear how to extend the results of [5,9,24] (as mentioned in [9], such extension must involve generalized gradients of possibly non-Lipschitz continuous, extended valued functions).
Instead of assuming regularity assumptions on the value function, we follow an idea introduced in [1] and consider an auxiliary control problem without state constraints. The value function of this auxiliary control problem is an interesting tool that can be used to characterize the epigraph of the function ϑ. Moreover, it turns out that the value function of the auxiliary control problem can be used to establish the link between the PMP and the HJB approach along an optimal path, without assuming any controllability assumption and for general optimal solutions (including the abnormal optimal trajectories).
The contributions of this paper are threefold. First, we derive new sensitivity relations between the PMP and the dynamic programming principle for a class of control problems with endpoint and intermediate state constraints without assuming any controllability assumptions. The sensitivity relations hold for normal optimal trajectories and abnormal trajectories as well. Furthermore, we simplify the arguments introduced in [9,24] to establish the sensitivity relations and we show that both relations can be obtained by considering a same perturbed control problem (without adding complex approximations of the dynamic equation by impulse systems).
The paper is organized as follows. Section 2 introduces the control problem with endpoint and intermediate constraints, and recall some known results in non-smooth analysis. The main results of the paper are given in Section 3. Section 4 is devoted to the particular case of endpoint constrained problem. In Section 5, we derive the sensitivity relations for a control problem with intermediate costs but without any constraint. This result is also a contribution of the paper and constitutes an important step for the proof of the main results of the paper. Finally, Section 6 presents the proof of the main results.

Statement of the problem. Hypotheses
We recall in this section some useful notions in non-smooth analysis. Then we formulate the control problem and lay out the assumptions that will be used in the paper.

Notations and some preliminaries
In all this paper, R denotes the set of real numbers, ·, · and · denote the Euclidean inner product and norm on R N (for any N ≥ 1), B N is the unit closed ball {x ∈ R N : x ≤ 1} (also denoted B if there is no ambiguity) and B(x; r) = x + rB. For any set S ⊆ R N , • S, S, ∂S, coS denote its interior, closure, boundary, and convex envelope, respectively. For any a, b ∈ R, we define Similarly, for a 1 , . . . , a m ∈ R, the notation m i=1 a i is defined as: m i=1 a i := max(a 1 , . . . , a m ).
Let us recall some definitions and properties from sub-differential calculus that will be used through the paper. We refer to ([8], Chap. 10) or ( [25], Chap. 5) for more details on non-smooth analysis.
Let φ : R N → R (with N ≥ 1) be a locally Lipschitz function in a neighborhood of a point x ∈ R N . The function φ is not necessary differentiable at x, but we can define a generalized directional derivative at x, for any direction v ∈ R N , as follows: With this notion of directional derivative, we can define the Clarke's sub-differential of φ at x by: The generalized directional derivative satisfies also: 3) It turns out that the sub-differential of φ at x is a compact convex set of R N , and if L > 0 is the Lipschitz constant of φ in a neighborhood of x then ∂φ(x) ⊂ B(0, L). The sub-differential satisfies some properties reminiscent of the classical differential calculus; for instance for any α ∈ R, we have: ∂(αφ(x)) = α∂φ(x). Furthermore, we have: ∂(φ + ψ)(x) ⊂ ∂φ(x) + ∂ψ(x) for φ and ψ Lipschitz functions in a neighborhood of x, and when ψ is continuously differentiable (in a neighborhood of x) then Some other properties satisfied by the sub-differential have no smooth counterpart: consider φ 1 , . . . , φ n continuously differentiable functions on an open neighborhood Ω of x and φ the pointwise maximum given by φ(y) := max(φ 1 (y), . . . , φ n (y)) for every y ∈ R N , is a Lipschitz continuous function. Moreover, if we denote Then the sub-differential of φ at x is given by: In the sequel, for a function φ : R N → R, we will also use the lower Dini directional derivative, in the direction v ∈ R N , defined as: and when φ is Lipschitz in a neighborhood of x then we have the following more simple expression: Some useful properties, that will be used later in the paper, are given in the next Lemma: Lemma 2.1. Let φ : R N → R be a Lipschitz continuous function on a neighborhood of x ∈ R N , and let u, v ∈ R N . Then Proof. The following decomposition holds: Therefore, taking the lim inf h↓0 in (2.8), we obtain the inequality (2.6).
As direct consequence of Lemma 2.1, for u, v, w ∈ R N , the following inequality holds:

Formulation of the problem
Let T > 0 be a finite time horizon and consider the following dynamical system (for 0 ≤ t < T ) : x(s) = f (s, x(s), u(s)), a.e. s ∈ (t, T ), (2.10) where the control input u : [0, T ] −→ R r (r ≥ 1) is a measurable function such that u(s) ∈ U for almost every s ∈ [0, T ]. Throughout this paper, we assume that We denote by U the set of all admissible controls: U := {u : [0, T ] −→ R r measurable, and u(s) ∈ U a.e.}.
given functions satisfying the following assumptions: x, u) is locally Lipschitz continuous in the following sense: and is locally Lipschitz continuous in the following sense: In the light of the standard theory of differential equations, assumption (H 1a ) guarantees, for every u ∈ U and (t, x) ∈ [0, T ] × R d , the existence of an absolutely continuous curve x : [t, T ] → R N which satisfies (2.10) and the initial data x(t) = x. Gronwall's Lemma leads also to (2.11) In the sequel, we shall denote the set of all admissible pairs control-and-trajectories starting at time t from the position x as: Let t 0 = 0 < t 1 < · · · < t m = T be given real numbers in [0, T ] (for some given integer m ≥ 1), and consider the control problem with the standard convention that inf ∅ = +∞. The final cost ϕ : R d → R and the constraint functions g i are given functions satisfying: (H 2 ) ϕ : R d → R is locally Lipschitz continuous and there exists a constant c ϕ ≥ 0 such that |ϕ(y)| ≤ c ϕ (1 + y ).
(H 3 ) For every i ∈ {1, . . . , m}, the constraint function g i : R d → R is locally Lipschitz continuous. There exists a constant c g ≥ 0 such that, for all i: Remark 2.2. To simplify the presentation, we assume here that every intermediate constraint is scalar. In the case where there are multiple constraints at an intermediate time t i : it suffices to consider the function g i := min(g 1 i , · · · , g qi i ) and replace the multiple constraints by a single one: We consider the value function ϑ : [0, T ] × R d −→ R, associated to this control problem, as follows: It is known that when the control problem does not include any state constraint (for instance if g i ≡ 0, ∀i), under assumptions (H 0 )-(H 3 ), the value function ϑ is locally Lipschitz continuous and can be characterized as unique solution of a Hamilton-Jacobi equation, see Chapter III of [3] and [7,19,20]. Furthermore, a relation between the Pontryagin's Maximum Principle and the Hamilton-Jacobi approach for problem (2.12) has been established in [24]. This relation gives a link between an adjoint state of (2.12) and the sub-differential of the value function ϑ along an optimal trajectory stating from x 0 at t = 0.
In this work, we are interested in the case when the control problem is in presence of final or intermediate state constraints (m ≥ 1). In this case the value function fails to be continuous unless some controllability assumptions are satisfied (see [3], Chap. V). Actually, in general the value function is discontinuous and may have infinite values. So, it is not clear how to define the sub-differential of the value function along an optimal trajectory. A direct extension of the work [24] requires some additional assumptions on the Lipschitz regularity of ϑ in at least a tube around an optimal trajectory, see [9] for more details. Instead of assuming such regularity assumptions, we follow an idea introduced in [1] and consider an auxiliary control problem without state constraints. It turns out that the value function of this auxiliary control problem is an interesting tool that can be used to obtain the values of the function ϑ and also to establish the link between the PMP and the HJB approach along an optimal path. Let us first consider the augmented dynamical system: (2.14b) This augmented dynamical system is usually used as a classical technique in control theory to recast the Bolza problem in the Mayer form: In what follows, we will assume that the augmented dynamics satisfies the following assumption: consider the set of all trajectories satisfying (2.14) on the time interval [a, b], for a control input u ∈ U, and starting from a position (x, z) at time a: Under assumptions (H 0 ), (H 1a ), (H 1b ) and (H 4 ), and by Filippov's Theorem, the multi-valued application Now, we define an auxiliary control problem and its value function w : We are interested in characterizing the function w as a solution to a HJB equation and also in expressing the link between the HJB equation and the Pontryagin's principle for problem (2.12).
In the sequel, we will use the following Hamiltonian function H : (2.16a) We will also use the maximized Hamiltonian function H :

Main results
The optimal control problem (2.12) might admit no solution if there is no trajectory that can realize all the intermediate constraints. However, under assumptions (H 0 )-(H 4 ), the auxiliary control problem does always achieve a minimum (x, z) ∈ S 0,T ] (x, z). Moreover, every optimal solution of the original control problem (2.12) is related to an optimal solution of the auxiliary control problem (2.15). More precisely, we have the following result. (i) For every (t, x, z) ∈ [0, T ] × R d+1 , the auxiliary control problem (2.15) admits an optimal solution.
x) < ∞, then the optimal control problem (2.13) admits a solution. Moreover, the function ϑ : (iii) For every x ∈ R d and t ∈ [0, T ], we have: is an optimal solution of (2.13), then (x, z, u) is also an optimal solution of the auxiliary problem (2.15), where z(s) : Conversely, any optimal solution of the auxiliary control problem (2.15) with z = ϑ(t, x) < ∞ is also an optimal solution of the original problem (2.13).

Proof. Under assumptions (H
T ]) endowed with the C 0 -topology. This means that there exists a minimizing sequence , Thm. 0.3.4 and Thm. 1.4.1). Moreover the sequence of derivatives (ẋ n ,ẏ n ) converges weakly in L 1 (0, T ) to (ẋ,ż). On the other hand, for any t ∈ [0, T ] and (x, z) ∈ R d × R, the functional: is a continuous function from C 0 ([t, T ]; R d+1 ) to R. Therefore, problem (2.15) achieves a minimum, which proves assertion (i).
Assertion (ii) is also a classical result whose proof is based again on the compactness of the set of trajectories, see Proposition 2.1 of [17]. To Hence, there exists an admissible control u ∈ U corresponding to the trajectory (x, z) such that (x, u) ∈ X [t,T ] (x) and Therefore, by definition of ϑ, we conclude that ϑ(t, x) ≤ z. Furthermore, if there is no z such that w(t, x, z) ≤ 0, then it means that there is no trajectory x satisfying the constraints g j (x(t j )) ≤ 0 (∀j ≥ 1 s.t. t j ≥ t), hence ϑ(t, x) = +∞. Conversely, if z t,x := ϑ(t, x) < +∞, then by assertion (ii), there exists an admissible pair (x, u) ∈ X [t,T ] (x) such that g j (x(t j )) ≤ 0 for every j such that t j ≥ t, and such that 0 = ϕ(x(T )) − z(T ), where z(T ) = z t,x − T t (s, x(s), u(s)) ds. This implies that w(t, x,z) ≤ 0. If ϑ(t, x) = +∞, this means there is no admissible trajectory x satisfying the constraints g j (x(t j )) ≤ 0 (∀j ≥ 1 s.t. t j ≥ t), so w(t, x, z) > 0 for every z ∈ R, and inf{z ∈ R, w(t, x, z) ≤ 0} = +∞. This concludes the proof of assertion (iii).
Finally assertion (iv) can be proved by using the same arguments as for (iii).
Proposition 3.1 states that the auxiliary problem admits an optimal solution even when the original problem has no solution. It is the sign of the optimal values w(t, x, ·) that indicates if the original problem has a solution or not.
The next theorem gives preliminary results on the auxiliary value function w, and a characterization of w as a solution of an HJB equation (assumption (H 4 ) is not needed in this theorem).
In particular, I 1 ∪ I 2 ∪ · · · ∪ I m is a partition of [0, T ], and for any t ∈ [0, T ] there is a unique j ∈ {1, . . . , m} such that t ∈ I j . In the sequel, we denote by the application from [0, T ] into {1, ·, m} that associates to every t ∈ [0, T ] the index j such that t ∈ I j : (t) = j ⇐⇒ t ∈ I j .
. Moreover, w is the unique L 1 -viscosity solution of the following HJB system (with H defined in (2.16b)): For the precise notion of L 1 -viscosity solution, we refer to [6,7,20,21]. Notice that if the dynamics f and the distributed cost are Lipschitz continuous with respect to the time variable, then w is a viscosity solution of the HJB system in the classical viscosity sense (see [3], Chap. III).
Proof. The result of this Theorem is quite straightforward and uses known results in viscosity theory. Indeed, for t ∈]t m−1 , t m ] =]t m−1 , T ], the value function w is given by: As a value function of a Mayer control problem, with assumptions (H 2 )-(H 3 ), the function w is locally Lipschitz continuous on ]t m−1 , T ] × R d × R and is the unique L 1 -viscosity solution of the equation (see [6]): Now, by using the Dynamic Programming principle, for i = 1, . . . , m − 1 and for every t ∈ I i , we have for any h > 0 such that t i < t + h ≤ t i+1 : In particular, for t = t − i , we deduce the relation (3.2b). Again by classical arguments, for any i = 1, . . . , m − 1, the function w is locally Lipschitz continuous on I i × R d × R and is the unique L 1 -viscosity solution of: More precisely, there is a unique locally Lipschitz continuous L 1 -viscosity solution of (3.3a) with the given terminal data at ). Let us denote by w i this solution. Then w(t, x, z) = w i (t, x, z) for any t i−1 < t ≤ t i . While w i is Lipschitz regular up to the left boundary t = t i−1 of [t i−1 , t i ], the function w may differ from w i at this boundary because of the jump condition (3.2b). Uniqueness then follows by recursion on i = m − 1, . . . , 1. This concludes the proof.
So far, Propositions 3.1 and 3.2 establish the link between the original control problem and the auxiliary one. One main feature of the function w consists in the fact that it is more regular than ϑ, it takes finite values everywhere, and it is characterized as the viscosity solution of a Hamilton-Jacobi equation (which is convenient for numerical approximations). Now, we are interested in linking the optimality conditions for an optimal solution of problem (2.12) with the auxiliary value function w. This link is the main result of this paper and it is stated precisely in the following Theorem. Then, there exist multipliers (λ i ) 0≤i≤m and a co-statep(·), absolutely continuous on each sub-interval (t j−1 , t j ): p (tj−1,tj ) ∈ W 1,1 ((t j−1 , t j ), R d ), such that the following assertions hold: (i) non-triviality of the multipliers: m j=0 λ j = 1, λ j ≥ 0 for j ≥ 0, and λ j g j (x(t j )) = 0 ∀j ≥ 1; (iv) general sensitivity relation: H t,x(t),p(t), −λ 0 ,p(t), −λ 0 ∈ q(t)∂ t,x,z w(t,x(t),z(t)) ∀t ∈ I j , j = 1, . . . , m (3.7) where the function q(t) is given by: λ j , with(·) defined as in (3.1); (v) partial sensitivity relations: Remark 3.5. In the case when λ 0 = 0, the co-statep and the multipliers (λ i ) 0≤i≤m can be rescaled , and the statement (i) can be replaced by: In the case of a Mayer problem (i.e., when there is no distributed cost, ≡ 0), the previous theorem holds withz(t) = const = z 0 . Furthermore, in this case, we can establish the sensitivity relations in a slightly different way. The precise statement is given in the next theorem.
The proof of Theorem 3.4 in the case when m = 1 (no intermediate state constraints) will be first given in Section 4. The proof of Theorem 3.4 (as well as Thm. 3.6), for the general case m ≥ 1, will be the focus of Sections 5 and 6.
Comments. First of all, let us notice that when there are no state constraints (for instance when m = 1 and g 1 (y) ≡ C, where C is a sufficiently large negative constant), then ϑ(t, x) − z = w(t, x, z).
This result is exactly the one already established in [24] (see also [9]). Theorems 3.4 & 3.6 establish the relation between the PMP and the HJB approach by using the auxiliary value function w and not the value function ϑ. Let us stress on the fact that because the function ϑ may be discontinuous, a relation between the PMP and the HJB approach cannot be expressed by using only ϑ. Indeed, if the value function is not locally Lipschitz then it is not even clear which sub-differential should be used, see [9].
Another feature of Theorems 3.4 & 3.6 is that no qualification constraint is required and so the extremal (x,ū,p) may be abnormal (i.e., λ 0 = 0). In all cases, the multiplier λ 0 is related to the sensitivity of the auxiliary value function w w.r.t the variable z.

The case of Mayer problem with final state constraints
In this section, we prove most of the assertions of Theorem 3.4 in the particular case where the control problem is a Mayer problem with only one end-point constraint (case m = 1, t m = t 1 = T , ≡ 0), and we furthermore denote g = g 1 throughout this section. The problem in this simplified setting is : minimize ϕ(x(T )) subject to g(x(T )) ≤ 0 and (x, u) ∈ X [0,T ] (x) (4.1) where g and ϕ are given locally Lipschitz continuous functions.
In this setting the auxiliary optimal control problem becomes ϕ(x(T )) − z g(x(T )). (4.2) This problem corresponds to an augmented dynamics where the dynamics for the z variable is zero. We recall that if z 0 = inf{z ∈ R, w(0, x, z) ≤ 0} is finite, then any optimal trajectory (x, u) for the auxiliary OCP associated with the value w(0, x, z 0 ) is also an optimal solution of the initial problem (2.13), and z 0 = ϑ(t, x).
As it can be seen in the proof of Theorem 4.1, p z (t) = const = −λ 0 . Also p z represents a sensitivity of the cost function w with respect to the z variable. The case when λ 0 = 0 means that there is no sensitivity with respect to the z variable and the cost only depends on the value of g(x(T )).

Sensitivity relations for a control problem with intermediate costs
In order to prove Theorem 3.4, we need to proceed in several steps. The first step aims to establish sensitivity relations for a general control problem where the cost function is in the form of a maximum of intermediate costs at given intermediate times 0 < t 1 < t 2 < · · · < t m = T . In this section we consider functions (Φ j ) 1≤j≤m : R N → R (with N ≥ 1) verifying the following regularity assumption: For each j = 1, . . . , m, Φ j : R N → R is locally Lipschitz continuous.
Consider the dynamics F : [0, T ] × R N × U satisfying the following assumption: x, u) is locally Lipschitz continuous in the following sense: ∀R > 0, ∃k R ≥ 0, ∀(x, y) ∈ (B N (0; R)) 2 , for a.e. t ∈ [0, T ] and for every u ∈ U : (iv) ∃c F > 0 such that for a.e. t ∈ [0, T ] and for every (x, u) ∈ R N × U : The control problem considered here is the following: y(s) = F (s, y(s), u(s)), a.e. on [0, T ], Note that the above control problem has no state constraint. Assumption (H F ) guarantees that for every u ∈ U and (t, x) ∈ [0, T ] × R N , there exists an absolutely continuous curve y : [t, T ] → R N which satisfies the dynamic equation and the initial data y(t) = y. We define the set of all admissible pairs control-and-trajectories starting at time t from the position y as: The value function associated to the control problem (5.1) is defined as follows. For every x ∈ R N , for every t ∈ [0, T ] and j ∈ {1, . . . , m} such that t ∈ I j : V(t, y) := min m k=j Φ j (y(t k )), (y, u) ∈ X F [t,T ] (y) .
By using similar arguments as in the proof of Proposition 3.2, we get that the value function V is locally Lipschitz continuous on each I j × R N (∀j = 1, . . . , m). Moreover, V is a L 1 -viscosity solution of the following system of HJB equations (see [7,20,21] for the precise definition of L 1 -viscosity notion): Furthermore, if we assume that all the functions Φ j have linear growth (i.e., ∃c Φ > 0 such that |Φ j (y)| ≤ c Φ (1 + y ) ∀y ∈ R N and j = 1, . . . , m), then V has also a linear growth on [0, T ] × R N and V is the unique L 1 -viscosity solution of (5.2).
Optimality conditions for the problem (5.1) can be deduced as particular cases of the setting considered in [11] or in [12]. The main result of this section aims at deriving the relation between the co-state associated to an optimal solution and the generalized gradient of the value function along that optimal solution.
It is worth noticing that the sensitivity relations cannot be deduced directly from the existing literature, however the proof is inspired by the arguments in [5,9,24]. We give here the complete proof for two main reasons: first, we show how the arguments can be adapted in case of intermediate costs and discontinuities of the value function at times t i , for i = 1, . . . , m − 1. Secondly, we give a simplified proof and show that all the sensitivity relations can be obtained by a simple perturbed control problem without using an approximation of the dynamic equation by impulse systems (as required in [5,9]). We introduce the Hamiltonian corresponding to problem (5.1): (5.4c) (iv) General sensitivity relation: for every t ∈ [0, T ]\{t 1 , . . . , t m−1 }, (v) Partial sensitivity relations: for a.e. t ∈ [0, T ], (5.4e) p(0) ∈ ∂ y V(0,ȳ(0)), Remark 5.2. Analogously to (5.4g), the following relation holds: but since q(T ) = ν m and V(T, x) = Φ m (x) this leads to the last relation already given in (5.4b).
Remark 5.3. The cost function being a maximum of intermediate costs, the sensitivity relation that gives a link between the adjoint arc and the value function holds actually along the optimal trajectory until the final time t m0 that corresponds to the time where the maximal intermediate cost is reached.
Proof. Let (ȳ,ū) be a local minimizer of problem (5.1). The proof of the optimality conditions and sensitivity relations will be split into three steps.
The proof of Lemma 5.4 is based on the following result whose proof comes directly from the dynamic programming principle of the value function V j (a sketch of the proof is given below for sake of completeness).
Let ε > 0. By assumption (H F ) on the dynamics F , and by standard arguments, we can show that, for h sufficiently small and for any (s , y ) close enough to (s, y), we have: y u s ,y (s + h) − (y + hF (s, y, u)) ≤ hε.
Moreover, since V j is locally Lipschitz continuous on [t j−1 , t j ] × R N , there exists L > 0 such that: V j (s + h, y + hF (s, y, u)) ≥ V j (s + h, y u s ,y (s + h)) − Lhε.
This inequality with the dynamic programming principle leads to: V j (s + h, y + hF (s, y, u)) ≥ V j (s + h, y u s ,y (s + h)) − Lhε ≥ V j (s , y ) − Lhε.