Tohoku Mathematical Journal 2001 September SECOND SERIES VOL. 53, NO. 3

 Tohoku Math. J. 53 (2001), 395-442

Title THE CLOSURE ORDERING OF ADJOINT NILPOTENT ORBITS IN $\mathfrak{so}(p, q)$

Author Dragomir Ž. Đoković, Nicole Lemire and Jiro Sekiguchi

(Received August 31, 2001, revised June 12, 2000)
Abstract. Let $\mathcal{O}$ be a nilpotent orbit in $\mathfrak{so} (p,q)$ under the adjoint action of the full orthogonal group $\mathrm{O} (p,q)$. Then the closure of $\mathcal{O}$ (with respect to the Euclidean topology) is a union of $\mathcal{O}$ and some nilpotent $\mathrm{O} (p,q)$-orbits of smaller dimensions. In an earlier work, the first author has determined which nilpotent $\mathrm{O} (p,q)$-orbits belong to this closure. The same problem for the action of the identity component $\mathrm{SO} (p,q)^0$ of $\mathrm{O} (p,q)$ on $\mathfrak{so} (p,q)$ is much harder and we propose a conjecture describing the closures of the nilpotent $\mathrm{SO} (p,q)^0$-orbits. The conjecture is proved when $\min(p,q) \le 7$.
Our method is indirect because we use the Kostant-Sekiguchi correspondence to translate the problem to that of describing the closures of the unstable orbits for the action of the complex group $\mathrm{SO}_p (\boldsymbol{C}) \times\mathrm{SO}_q (\boldsymbol{C})$ on the space $M_{p,q}$ of complex $p \times q$ matrices with the action given by $(a,b) \cdot x=axb^{-1}$. The fact that the Kostant--Sekiguchi correspondence preserves the closure relation has been proved recently by Barbasch and Sepanski.

2000 Mathematics Subject Classification. Primary 17B45; Secondary 17B20, 17B40.

Key words and phrases. Nilpotent adjoint orbits, standard triples, Kostant-Sekiguchi correspondence.