# Symmetry breaking operators for the restriction of representations of indefinite orthogonal groups O(p, q).

1. Branching problem. Suppose G [contains] G' are reductive groups and [pi] is an irreducible representation of G. The restriction of [pi] to the subgroup G' is no more irreducible in general as a representation of G'. If G is compact, then any irreducible [pi] is finite-dimensional and splits into a finite direct sum

[mathematical expression not reproducible]

of irreducibles [pi]' of G' with multiplicities m([pi], [pi]'). These multiplicities have been studied by various techniques including combinatorial algorithms.

However, for noncompact G' and for infinite-dimensional [pi], the restriction [pi][|.sub.G'] is not always a direct sum of irreducible representations, see [5, 6] for details. In order to define the "multiplicity" in this generality, we recall that, associated to a continuous representation [pi] of a Lie group on a Banach space H, a continuous representation [[pi].sup.[infinity]] is defined on the Frechet space [H.sup.[infinity]] of [C.sup.[infinity]]-vectors of H. Given another representation [pi]' of a subgroup G', we consider the space of continuous G'-intertwining operators (symmetry breaking operators)

(1.1) [Home.sub.G']([[pi].sup.[infinity]][|.sub.G'], [([pi]').sup.[infinity]]).

If both [pi] and [pi]' are admissible representations of finite length of reductive Lie groups G and G', respectively, then the dimension of the space (1.1) is determined by the underlying (g, K)-module [[pi].sub.K] of [pi] and the (g', K')-module [[pi]'.sub.K'] of [pi]', and is independent of the choice of Banach globalizations by the Casselman-Wallach theory [17, Chap. 11]. We denote by m([pi], [pi]') the dimension of (1.1), and call it the multiplicity of [pi]' in the restriction [pi][|.sub.G'].

The above definition of the multiplicity m([pi], [pi]') makes sense for nonunitary representations, too.

In general, m([pi], [pi]') may be infinite, even when G' is a maximal reductive subgroup of G (e.g. symmetric pairs). By using the theory of real spherical spaces [14], the geometric criterion for finite multiplicities was proved in [7] and [14] as follows.

Fact 1.1. Let (G, G') be a pair of real reductive Lie groups with complexification ([G.sub.C], [G'.sub.C]).

(1) The multiplicity m([pi], [pi]') is finite for all irreducible representations [pi] of G and all irreducible representations [pi]' of G' if and only if a minimal parabolic subgroup of G' has an open orbit on the real flag variety of G.

(2) The multiplicity m([pi], [pi]') is uniformly bounded if and only if a Borel subgroup of [G'.sub.C] has an open orbit on the complex flag variety of [G.sub.C].

The complete classification of symmetric pairs (G, G') satisfying the above geometric criteria was accomplished in Kobayashi-Matsuki [11].

On the other hand, switching the order in (1.1), we may also consider another space

[mathematical expression not reproducible].

The study of these objects is closely related to the theory of discretely decomposable restrictions [5, 6].

Notation. We adopt the same convention as in [16] for the following notation. N := {0, 1, 2, ...}. [(x).sub.j] := x(x + 1) ... (x + j - 1). For two subsets A and B of a set, we write A - B :={a [member of] A : a [not member of] B} rather than the usual notation A \ B. The symbols [??], [??], [parallel], and [??] are defined to be subsets of [C.sup.2], and are not binary relations.

2. ABC program for branching. In [8] the first author suggested a program for studying the restriction of representations of reductive groups, which may be summarized as follows:

(A) Abstract features of the restriction;

(B) Branching law of [pi][|.sub.G'];

(C) Construction of symmetry breaking operators.

Program A aims for establishing the general theory of the restrictions [pi][|.sub.G'] (e.g. spectrum, multiplicity), which would single out the good triples (G, G', [pi]). In turn, we could expect concrete and detailed study of those restrictions [pi][|.sub.G'] in Programs B and C.

The current work concerns Program C for certain standard representations with focus on symmetry breaking operators (SBOs for short) as follows:

(C1) Construct SBOs explicitly;

(C2) Classify all SBOs;

(C3) Find residue formulae for SBOs;

(C4) Study functional equations among SBOs;

(C5) Determine the images of subquotients by SBOs.

The subprogram (C1)-(C5) was proposed by Kobayashi Speh in their book [16] with a complete answer for the pair (G, G') = (O(n + 1, 1), O(n, 1)) of real rank one groups.

In this note we treat degenerate spherical principal series representations [pi] = I([lambda]) of G and [pi]' = J(v) of G' for the pair of higher real rank groups

(2.1) (G, G') = (O(p + 1, q + 1), O(p, q + 1)),

and give an answer to (C1)-(C4). The subprogram (C5) will be discussed in a separate paper.

Concerning Program A, Fact 1.1 assures the following a priori estimate:

m([pi], [pi]') is uniformly bounded

if the pair of Lie algebras (g, g') is a real form of (sl(n + 1, C), gl(n, C)) or (o(n + 1, C), o(n, C)), in particular, if (G, G') is of the form (2.1).

3. Settings. Let G = O(p + 1, q + 1) be the automorphism group of the quadratic form on [R.sup.p+q+2] of signature (p + 1, q + 1) defined by

[mathematical expression not reproducible].

A degenerate spherical principal series representation I([lambda]) := [Ind.sup.G.sub.P]([C.sub.[lambda]]) with parameter [lambda] [member of] C of G is induced from a character [C.sub.[lambda]] of a maximal parabolic subgroup P = [MAN.sub.+] with Levi part MA [equivalent] O(p, q) x {[+ or -]1} x R. We realize I([lambda]) on the space of [C.sup.[infinity]] sections of the G-equivariant line bundle

[L.sub.[lambda]] = G [x.sub.P] [C.sub.[lambda]] [right arrow] G/P

so that I([lambda]) itself is the smooth Frechet globalization of moderate growth. Our parametrization is chosen in a way that I([lambda]) contains a finite-dimensional submodule if -[lambda] [member of] 2N and a finite-dimensional quotient if [lambda] - (p + q) [member of] 2N (cf. [3]).

Let G' = O(p, q + 1) be the stabilizer of the basis element [e.sub.p]. Similarly to I([lambda]), we denote by J(v) := [Ind.sup.G'.sub.P']([C.sub.v]) the representation of G' induced from a character [C.sub.v] of a maximal parabolic subgroup P' of G' with Levi part O(p - 1, q) x {[+ or -] 1} x R.

The representation I([lambda]) arises from conformal geometry as follows. We endow the direct product manifold [S.sup.p] x [S.sup.q] with the pseudo-Riemannian structure [mathematical expression not reproducible] of signature (p, q). Then the group G = O(p + 1, q + 1) acts as conformal diffeomorphisms on [S.sup.p] x [S.sup.q], and also on its quotient space X = ([S.sup.p] x [S.sup.q])/[Z.sub.2] by identifying the direct product of antipodal points. By the general theory of conformal groups, one has a natural family of representations [[??].sub.[lambda]] on [C.sup.[infinity]](X) with parameter [lambda] [member of] C [12, Sect. 2]. Then X identifies with G/P, and [[??].sub.[lambda]] identifies with I([lambda]). Thus the branching problem in our setting arises from the conformal construction of representations for the pair

(X, Y) = (([S.sup.p] x [S.sup.q])/[Z.sub.2], ([S.sup.p-1] x [S.sup.q])/[Z.sub.2]).

4. Multiplicity formulae. In this section we determine explicitly the multiplicity

m(I([lambda]), J(v)) = dim [Home.sub.G'](I([lambda])[|.sub.G'], J(v)).

We shall find m(I([lambda]), J(v)) > 0 for all [lambda], v [member of] C. Following [16], we define four subsets of [C.sup.2] as below:

[mathematical expression not reproducible],

and two subsets of [Z.sup.2] by

[mathematical expression not reproducible].

Theorem 4.1. Let (G, G') be as in (2.1) with p, q [greater than or equal to] 1. Then

m(I([lambda]), J(v)) [member of] {1, 2}

for all [lambda], v [member of] C. Furthermore, m(I([lambda]), J(v)) = 2 if and only if one of the following conditions holds:

Case 1. p > 1. ([lambda], v) [member of] A.

Case 2. p = 1 and q is odd. ([lambda], v) [member of] A [union] X.

Case 3. p = 1 and q is even. ([lambda], v) [member of] A [union] X - X [intersection] [??].

We shall construct explicitly all the symmetry breaking operators in Section 6.

5. Double coset space P'\G/P. In general, as is seen in Fact 1.1 (and Fact 6.2 below), the double coset space P'\G/P plays a fundamental role in analyzing symmetry breaking operators

[Ind.sup.G.sub.P]([sigma]) [right arrow] [Ind.sup.G'.sub.P']([tau]),

where [sigma] is a representation of a parabolic subgroup P of G and [tau] is that of a parabolic subgroup P' of G'. The description of the double coset space P'\G/P is nothing but the Bruhat decomposition if G' = G; the Iwasawa decomposition if G' is a maximal compact subgroup K of G where P' automatically equals K.

In this section we give a description of P'\G/P together with its closure relation in the setting where (G, G', P, P') is given as in Section 3. Then the natural action of G = O(p + 1, q + 1) on [R.sup.p+q+2] preserves the isotropic cone

[mathematical expression not reproducible],

inducing the G-action on its quotient space

X := [XI]/[R.sup.x] [equivalent] ([S.sup.p] x [S.sup.q])/[Z.sub.2].

We define the subvarieties of X by

Y := {[x] [member of] X | [x.sub.p] = 0},

C := {[x] [member of] X | [x.sub.0] = [x.sub.p+q+1]}.

Let P be the stabilizer of the point

o := [1 : 0 : ... : 0 : 1] [member of] X [equivalent] [XI]/[R.sup.x],

and P' := P [intersection] G'. Then X and Y are identified with the real flag varieties G/P and G'/P', respectively.

Theorem 5.1 (description of P'\G/P). Suppose p, q [greater than or equal to] 1. The left P'-invariant closed subsets of G/P are described in the following Hasse diagram. Here [mathematical expression not reproducible] means that A [contains] B and that the subvariety B is of codimension m in A.

[formula not reproducible]

6. Construction of SBOs. Let n := p + q. The slice of [XI] by the hyperplane [x.sub.0] + [x.sub.p+q+1] = 2 defines the coordinates ([x.sub.1], ..., [x.sub.n]) [member of] [R.sup.n] of the open Bruhat cell U of G/P, and induces the N-picture of the representation I([lambda]), [[iota].sup.*.sub.[lambda]] : I([lambda]) [??] [C.sup.[infinity]]([R.sup.n]) via the trivialization [L.sub.[lambda]][|.sub.U] [equivalent] [R.sup.n] x C. Likewise, x' = ([x.sub.1], ..., [[??].sub.p], ..., [x.sub.n]) [member of] [R.sup.n-1] give the coordinates of the Bruhat cell of G'/P', and we have the N-picture [[iota].sup.*.sub.v] : J(v) [??] [C.sup.[infinity]]([R.sup.n-1]).

We shall realize a symmetry breaking operator T in the N-pictures of I([lambda]) and J(v), and find a distribution [K.sub.T] [member of] D'([R.sup.n]) such that for all f [member of] I([lambda])

[mathematical expression not reproducible].

In order to analyze the distribution kernels [K.sub.T] of symmetry breaking operators T, we begin with:

Definition 6.1. We let O(p - 1, q) act on [R.sup.n] (n = p + q) by leaving [x.sub.p] invariant. We define Sol([R.sup.p,q]; [lambda], v) to be the space of distributions F [member of] D'([R.sup.n]) satisfying the following three conditions:

(1) F is O(p - 1, q)-invariant and F(x) = F(-x);

(2) F is homogeneous of degree [lambda] - v - n;

(3) F is invariant by [N'.sub.+] : = [N.sub.+] [intersection] G'.

Applying the general results proven in [16, Chap. 3] to our particular setting, we get the following.

Fact 6.2 ([16, Thm. 3.16]). Recall n = p + q(p, q [greater than or equal to] 1). Then the following diagram commutes:

[formula not reproducible]

For T [member of] [Hom.sub.G'](I([lambda])[|.sub.G'], J(v)), a closed P'- invariant subset Supp(T) in X = G/P is defined to be the support of the distribution kernel [K.sub.T] [member of] [(D'(G/P, [L.sub.n-[lambda]]) [cross product] [C.sub.v]).sup.P']. By [15, Lem. 2.22], T is a differential symmetry breaking operator if and only if Supp(T) is a singleton.

Conversely, for each P'-invariant closed subset S = {o}, C, Y or X itself, we define a subset [D.sub.S] of [C.sup.2] which is either the whole [C.sup.2] or a countable union of one-dimensional complex affine spaces, and construct a family of SBOs, [R.sup.S.sub.[lambda],v]: I([lambda]) [right arrow] J(v), such that

* [R.sup.S.sub.[lambda],v] depends holomorphically on ([lambda], v) [member of] [D.sub.S];

* Supp([R.sup.S.sub.[lambda],v]) [subset] S for every ([lambda], v) [member of] [D.sub.S], and the equality holds for generic points in [D.sub.S].

The distribution kernels [K.sup.S.sub.[lambda],v] of the operators [R.sup.S.sub.[lambda],v] will be given explicitly in Theorems 6.3 6.6 and Fact 6.7. The relations among them are discussed in Section 8 as "residue formulae". The space of SBOs is generated by these operators, as we shall see the classification results in Theorem 6.9.

Here is a summary of the symmetry breaking operators that we construct below.
```[R.sup.S.sub.[lambda],v] =            [D.sub.S]
Op ([K.sup.S.sub.[lambda],v])

[R.sup.X.sub.[lambda],v] =            [C.sup.2]      Theorem 6.3
Op ([K.sup.X.sub.[lambda],v])

[[??].sup.X.sub.[lambda],v] =         [??]           Theorem 6.4
Op ([[??].sup.X.sub.[lambda],v])

[R.sup.Y.sub.[lambda],v] =            [??]           Theorem 6.5
Op ([K.sup.Y.sub.[lambda],v])

[R.sup.C.sub.[lambda],v] =            [parallel]     Theorem 6.6
Op ([K.sup.C.sub.[lambda],v])

[R.sup.{o}.sub.[lambda],v] =          [??]           Fact 6.7
Op ([K.sup.{o}.sub.[lambda],v])
```

Theorem 6.3 (regular symmetry breaking operator). Suppose n = p + q with p, q [greater than or equal to] 1.

(1) There exists a family of symmetry breaking operators [R.sup.S.sub.[lambda],v] [member of] [Hom.sub.G'](I([lambda])[|.sub.G'], J(v)) that depends holomorphically on ([lambda], v) in the entire [C.sup.2] with the distribution kernel [K.sup.X.sub.[lambda],v](x) given by

[mathematical expression not reproducible].

(2) [R.sup.X.sub.[lambda],v] vanishes if and only if ([lambda], v) belongs to the discrete set A for p > 1, A [union] X for p = 1, q odd and A [union] X - X [intersection] [??] for p = 1, q even.

(3) Supp([R.sup.X.sub.[lambda],v]) [subset] Y, C or {o} if ([lambda], v) [member of] [??], [parallel] or [??], respectively, and Supp([R.sup.X.sub.[lambda],v]) = X otherwise.

The above normalization of [R.sup.X.sub.[lambda],v] is optimal in the sense that the zeros of [R.sup.X.sub.[lambda],v] form a subset of codimension two in [C.sup.2]. Next, we renormalize [R.sup.X.sub.[lambda],v] in the places where [R.sup.X.sub.[lambda],v] vanishes. For this, we observe that [GAMMA]([[lambda]-v]/2) is holomorphic in [C.sup.2] - [??], and therefore

[mathematical expression not reproducible]

makes sense if ([lambda], v) [member of] [C.sup.2] - [??]. Moreover, in light of the fact that [K.sup.X.sub.[lambda],v] vanishes on A = [??] [intersection] [??], we obtain its analytic continuation on [??] as follows.

Theorem 6.4 (renormalized operator [[??].sup.X.sub.[lambda],v]).

(1) The renormalized symmetry breaking operator

[mathematical expression not reproducible]

is defined for ([lambda], v) [member of] [??] that depends holomorphically on A in the entire C for each fixed v.

(2) [[??].sup.X.sub.[lambda],v] vanishes if and only if p = 1, q even and ([lambda], v) [member of] X - [??].

Let N:R [right arrow] Z be a discontinuous function defined by N(x) := x if x [member of] N; = 0 otherwise.

Associated to closed subsets Y and C in P'\G/P we introduce families of singular SBOs. For later purpose, we discuss only the case p = 1.

Theorem 6.5 (singular symmetry breaking operators [R.sup.Y.sub.[lambda],v]). Suppose p = 1 and q [greater than or equal to] 1. For ([lambda], v) [member of] [??], we fix k := 1/2 (q - [lambda] - v) [member of] N. Then there exists a family of symmetry breaking operators [R.sup.Y.sub.[lambda],v] that depends holomorphically on v in the entire plane C with the distribution kernel [K.sup.Y.sub.[lambda],v] given by

[mathematical expression not reproducible].

Theorem 6.6 (singular symmetry breaking operators [R.sup.C.sub.[lambda],v]). Suppose p = 1 and q [greater than or equal to] 1. For ([lambda], v) [member of] [parallel], we fix m := 1/2 (v - 1) [member of] N. Then there exists a family of symmetry breaking operators [R.sup.C.sub.[lambda],v] that depends holomorphically on [lambda] in the entire plane C with the distribution kernel [K.sup.C.sub.[lambda],v] given by

[mathematical expression not reproducible].

The differential symmetry breaking operators [R.sup.{o}.sub.[lambda],v]: [C.sup.[infinity]]([R.sup.n]) [right arrow] [C.sup.[infinity]]([R.sup.n-1]) were previously found in [4, Thms. 5.1.1 and 5.2.1] for q = 0 and in [13, Thm. 4.3] for general p, q by a different approach. See also [9, 10] for further generalization.

Fact 6.7. Suppose ([lambda], v) [member of] [??]. We set l := 1/2(v - [lambda]) [member of] N. We define [R.sup.{o}.sub.[lambda],v] by

[mathematical expression not reproducible]

where [a.sub.j]([lambda], v) is given by

[mathematical expression not reproducible].

Then [R.sup.{o}.sub.[lambda],v] [member of] [Hom.sub.G'](I([lambda])[|.sub.G'], J(v)). The coefficients [a.sub.j]([lambda], v) give rise to a Gegenbauer polynomial

[mathematical expression not reproducible]

renormalized as [mathematical expression not reproducible].

Its distribution kernel is given by

[mathematical expression not reproducible].

Remark 6.8. The operators [R.sup.Y.sub.[lambda],v], [R.sup.C.sub.[lambda],v] and [R.sup.{o}.sub.[lambda],v] do not vanish.

The SBOs are not always linearly independent, but exhaust all SBOs. We provide explicit basis for [Hom.sub.G'](I([lambda])[|.sub.G'], J(v)) for every ([lambda], v) [member of] [C.sup.2]:

Theorem 6.9 (classification of SBOs). The vector space [Hom.sub.G'](I([lambda])[|.sub.G'], J(v)) is spanned by the operators as below.

(1) Suppose p = 1 and q [greater than or equal to] 1.

[mathematical expression not reproducible].

(2) Suppose p [greater than or equal to] [member of] and q [greater than or equal to] 1.

[mathematical expression not reproducible].

7. Spectrum of SBOs. The representation I([lambda]) of G contains a one-dimensional subspace of spherical vectors (i.e. K-fixed vectors), and likewise J(v) of G'. Let [1.sub.[lambda]] [member of] I([lambda]), [1.sub.v] [member of] J(v) be the spherical vectors normalized by [1.sub.[lambda]](e) = [1.sub.v](e) = 1. With this normalization, we have:

Theorem 7.1 (spectrum for spherical vectors). Let n = p + q (p, q [greater than or equal to] 1) as before.

[mathematical expression not reproducible].

Remark 7.2. Theorem 7.1 was known in Bernstein Reznikov [1] for p = q = 1 and in [16, Prop. 7.4] for q = 0. Another generalization was given in [2, Thm. 1.1] for higher dimensional cases.

8. Residue formulae of SBOs. The regular symmetry breaking operators [R.sup.X.sub.[lambda],v] have two complex parameters ([lambda], v) [member of] [C.sup.2], whereas the singular operators [R.sup.Y.sub.[lambda],v], [R.sup.C.sub.[lambda],v], and [R.sup.{o}.sub.[lambda],v] are defined for ([lambda], v) [member of] [??], [parallel] and [??], respectively. We find the relationship among these operators as explicit residue formulae.

Proposition 8.1. Suppose p = 1.

(1) For ([lambda], v) [member of] [??], we set k = 1/2 (q - [lambda] - v) [member of] N.

Then

[mathematical expression not reproducible].

(2) For ([lambda], v) [member of] [parallel], we set m := 1/2 (v - 1) [member of] N. Then

[mathematical expression not reproducible].

Theorem 8.2 (residue formula). Let n = p + q (p, q [greater than or equal to] 1). For ([lambda], v) [member of] [??], we set l := 1/2 (v - [lambda]) [member of] N. Then we have for ([lambda], v) [member of] [??]

[mathematical expression not reproducible].

Proposition 8.1 treats easier cases as the subvarieties Y and C are of codimension one in X (see Theorem 5.1), whereas Theorem 8.2 is more involved.

Remark 8.3. The residue formula in the case q = 0 was given in [16, Thm. 12.2].

9. Functional identities among SBOs. Let n := p + q as before. We recall that there exist nonzero Knapp Stein intertwining operators

[[??].sup.G.sub.[lambda]] : I([lambda]) [right arrow] I(n - [lambda])

with holomorphic parameter [lambda] [member of] C by the distribution kernel in the N-picture normalized as follows:

[mathematical expression not reproducible]

Similarly, we write [[??].sup.G'.sub.v] : J(v) [right arrow] J(n - 1 - v) for the Knapp Stein intertwining operator for G'.

Theorem 9.1 (functional identities).

[mathematical expression not reproducible],

for any [lambda], v [member of] C, where

[mathematical expression not reproducible].

Remark 9.2. The functional identities in the case q = 0 were proven in [8, Thm. 12.6].

We have given all the constants in this note as multiplicative formula so that we can tell the zeros explicitly. Their representation-theoretic interpretation serves as a clue in the subprogram (C5).

A detailed proof will appear elsewhere.

doi: 10.3792/pjaa.93.86

Acknowledgement. The first author was partially supported by the Grant-in-Aid for Scientific Research (A) 25247006.

References

[1] J. Bernstein and A. Reznikov, Estimates of automorphic functions, Mosc. Math. J. 4 (2004), no. 1, 19-37.

[2] J.-L. Clerc, T. Kobayashi, B. Orsted, and M. Pevzner, Generalized Bernstein-Reznikov integrals, Math. Ann. 349 (2011), no. 2, 395-431.

[3] R. E. Howe and E.-C. Tan, Homogeneous functions on light cones: the infinitesimal structure of some degenerate principal series representations, Bull. Amer. Math. Soc. (N.S.) 28 (1993), no. 1, 1-74.

[4] A. Juhl, Families of conformally covariant differential operators, Q-curvature and holography, Progress in Mathematics, 275, Birkhauser Verlag, Basel, 2009.

[5] T. Kobayashi, Discrete decomposability of the restriction of [A.sub.q]([lambda]) with respect to reductive subgroups. II. Micro-local analysis and asymptotic K-support, Ann. of Math. (2) 147 (1998), no. 3, 709-729.

[6] T. Kobayashi, Discrete decomposability of the restriction of [A.sub.q]([lambda]) with respect to reductive subgroups. III. Restriction of Harish-Chandra modules and associated varieties, Invent. Math. 131 (1998), no. 2, 229-256.

[7] T. Kobayashi, Shintani functions, real spherical manifolds, and symmetry breaking operators, in Developments and retrospectives in Lie theory, Dev. Math., 37, Springer, Cham, 2014, pp. 127-159.

[8] T. Kobayashi, A program for branching problems in the representation theory of real reductive groups, in Representations of reductive groups: in honor of the 60th birthday of D. Vogan (MIT, 2014), 277-322, Progr. Math., 312, Birkhauser/ Springer, Cham, 2015.

[9] T. Kobayashi, T. Kubo and M. Pevzner, Conformal symmetry breaking operators for differential forms on spheres, Lecture Notes in Mathematics, 2170, Springer, Singapore, 2016.

[10] T. Kobayashi, T. Kubo and M. Pevzner, Conformal symmetry breaking operators for antide Sitter spaces, arXiv:1610.09475. (to appear in Trends Math.).

[11] T. Kobayashi and T. Matsuki, Classification of finite-multiplicity symmetric pairs, Transform. Groups 19 (2014), no. 2, 457 493. (In special issue in honour of Professor Dynkin for his 90th birthday).

[12] T. Kobayashi and B. Orsted, Analysis on the minimal representation of O(p, q). I. Realization via conformal geometry, Adv. Math. 180 (2003), no. 2, 486-512.

[13] T. Kobayashi, B. Orsted, P. Somberg and V. Soucek, Branching laws for Verma modules and applications in parabolic geometry. I, Adv. Math. 285 (2015), 1796-1852.

[14] T. Kobayashi and T. Oshima, Finite multiplicity theorems for induction and restriction, Adv. Math. 248 (2013), 921-944.

[15] T. Kobayashi and M. Pevzner, Differential symmetry breaking operators: I. General theory and F-method, Selecta Math. (N.S.) 22 (2016), no. 2, 801-845.

[16] T. Kobayashi and B. Speh, Symmetry breaking for representations of rank one orthogonal groups, Mem. Amer. Math. Soc. 238 (2015), no. 1126, v+110 pp.

[17] N. R. Wallach, Real reductive groups. II, Pure and Applied Mathematics, 132-II, Academic Press, Inc., Boston, MA, 1992.

By Toshiyuki KOBAYASHI (*, **) and Alex LEONTIEV (*)

(Communicated by Masaki KASHIWARA, M.J.A., Sept. 12, 2017)

(*) Graduate School of Mathematical Sciences, The University of Tokyo, 3-8-1 Komaba, Meguro-ku, Tokyo 153-8914, Japan.

(**) Kavli Institute for the Physics and Mathematics of the Universe, The University of Tokyo, 5-1-5 Kashiwanoha, Kashiwa, Chiba 277-8583, Japan.