Gramian Based Interaction Measures
\( \newcommand{\states}{\mathcal{S}} \newcommand{\actions}{\mathcal{A}} \newcommand{\observations}{\mathcal{O}} \newcommand{\rewards}{\mathcal{R}} \newcommand{\traces}{\mathbf{e}} \newcommand{\transition}{P} \newcommand{\reals}{\mathbb{R}} \newcommand{\naturals}{\mathbb{N}} \newcommand{\complexs}{\mathbb{C}} \newcommand{\field}{\mathbb{F}} \newcommand{\numfield}{\mathbb{F}} \newcommand{\expected}{\mathbb{E}} \newcommand{\var}{\mathbb{V}} \newcommand{\by}{\times} \newcommand{\partialderiv}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\defineq}{\stackrel{{\tiny\mbox{def}}}{=}} \newcommand{\defeq}{\stackrel{{\tiny\mbox{def}}}{=}} \newcommand{\eye}{\Imat} \newcommand{\hadamard}{\odot} \newcommand{\trans}{\top} \newcommand{\inv}{{-1}} \newcommand{\argmax}{\operatorname{argmax}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\avec}{\mathbf{a}} \newcommand{\bvec}{\mathbf{b}} \newcommand{\cvec}{\mathbf{c}} \newcommand{\dvec}{\mathbf{d}} \newcommand{\evec}{\mathbf{e}} \newcommand{\fvec}{\mathbf{f}} \newcommand{\gvec}{\mathbf{g}} \newcommand{\hvec}{\mathbf{h}} \newcommand{\ivec}{\mathbf{i}} \newcommand{\jvec}{\mathbf{j}} \newcommand{\kvec}{\mathbf{k}} \newcommand{\lvec}{\mathbf{l}} \newcommand{\mvec}{\mathbf{m}} \newcommand{\nvec}{\mathbf{n}} \newcommand{\ovec}{\mathbf{o}} \newcommand{\pvec}{\mathbf{p}} \newcommand{\qvec}{\mathbf{q}} \newcommand{\rvec}{\mathbf{r}} \newcommand{\svec}{\mathbf{s}} \newcommand{\tvec}{\mathbf{t}} \newcommand{\uvec}{\mathbf{u}} \newcommand{\vvec}{\mathbf{v}} \newcommand{\wvec}{\mathbf{w}} \newcommand{\xvec}{\mathbf{x}} \newcommand{\yvec}{\mathbf{y}} \newcommand{\zvec}{\mathbf{z}} \newcommand{\Amat}{\mathbf{A}} \newcommand{\Bmat}{\mathbf{B}} \newcommand{\Cmat}{\mathbf{C}} \newcommand{\Dmat}{\mathbf{D}} \newcommand{\Emat}{\mathbf{E}} \newcommand{\Fmat}{\mathbf{F}} \newcommand{\Gmat}{\mathbf{G}} \newcommand{\Hmat}{\mathbf{H}} \newcommand{\Imat}{\mathbf{I}} \newcommand{\Jmat}{\mathbf{J}} \newcommand{\Kmat}{\mathbf{K}} \newcommand{\Lmat}{\mathbf{L}} \newcommand{\Mmat}{\mathbf{M}} \newcommand{\Nmat}{\mathbf{N}} \newcommand{\Omat}{\mathbf{O}} \newcommand{\Pmat}{\mathbf{P}} \newcommand{\Qmat}{\mathbf{Q}} \newcommand{\Rmat}{\mathbf{R}} \newcommand{\Smat}{\mathbf{S}} \newcommand{\Tmat}{\mathbf{T}} \newcommand{\Umat}{\mathbf{U}} \newcommand{\Vmat}{\mathbf{V}} \newcommand{\Wmat}{\mathbf{W}} \newcommand{\Xmat}{\mathbf{X}} \newcommand{\Ymat}{\mathbf{Y}} \newcommand{\Zmat}{\mathbf{Z}} \newcommand{\Sigmamat}{\boldsymbol{\Sigma}} \newcommand{\identity}{\Imat} \newcommand{\epsilonvec}{\boldsymbol{\epsilon}} \newcommand{\thetavec}{\boldsymbol{\theta}} \newcommand{\phivec}{\boldsymbol{\phi}} \newcommand{\muvec}{\boldsymbol{\mu}} \newcommand{\sigmavec}{\boldsymbol{\sigma}} \newcommand{\jacobian}{\mathbf{J}} \newcommand{\ind}{\perp!!!!\perp} \newcommand{\bigoh}{\text{O}} \)
So far, we have described the agent design assuming we know the underlying system’s structure. Instead, there are several well studied objects which can be used to understand the interactions between actions and their outputs. For simplicity, we will focus on \(N \times N\) linear time-invariant MIMO systems (i.e., one degree of freedom for each output)
\begin{align*} \dot{\xvec}(t) &= \Amat \xvec(t) + \Bmat \uvec(t) \\ \yvec(t) &= \Cmat \xvec(t), \end{align*}
where \(\xvec\) is known as the state of they system, and \(\yvec\) is the output or observation of the system.
The Graiman matrices in control are used to study different aspects of a control problem including the observability—the ability to measure the state from the observed variables—and the controllability—the ability to move from an initial state to a desired state through manipulating the input \todocite{textbook}. If the underlying system dynamics are known and stable, the interaction from inputs to outputs through the state of the system is represented by the controllability and observability Gramian matrices by solving the following
\begin{align} \Amat \Wmat_C + \Wmat_C \Amat^\top &= - \Bmat \Bmat^\top \\ \Amat \Wmat_O + \Wmat_O \Amat^\top &= - \Cmat^\top \Cmat \end{align}
where \(\Wmat_C\) and \(\Wmat_O\) are the controllability and observability Gramians respectively. From the controllability and observability Gramians, we can derive what is known as the participation matrix
\begin{align*} [\Psi]_{ij} = \frac{tr({\Wmat_C}_j {\Wmat_O}_i)}{tr(\Wmat_C \Wmat_O)}, \end{align*}
which is a standard well used object to decentralize feedback controllers in linear time-invariant systems.
Although defined using the system model, for non-linear and unknown systems the Gramians, and in turn the participation matrix, can be estimated empirically \citep{himpe2018emgr}. This is done through a set of controlled perturbations to the system. We define the sets of perturbations and their strengths as,
\begin{align} E_u &= \{\epsilon^j \in \Reals^M: m=1\ldots M, \epsilon^j_i = \delta_{im}\} \\ S_u &= \{c_k \in \Reals: k=1\ldots K, c_k \neq 0\} \\ E_x &= \{\epsilon^j \in \Reals^N: j=1\ldots N, \epsilon^j_i = \delta_{ij}\} \\ S_x &= \{d_l \in \Reals: l=1\ldots L, d_l \neq 0\} \end{align}
The set \(E_{u|x}\) are a set of orthogonal vectors, one for each basis in the inputs space or state space. The key is we use these directions to generate perturbations to simulate trajectories. The gramians can then be estimated by,
\begin{align*} \hat{W}_c &= \frac{1}{K} \sum_{k=1}^{K} \sum_{m=1}^M \frac{1}{c_k} \int_0^\infty \left(x^{km}(t) - \bar{x}^{km} \right) \left( x^{km}(t) - \bar{x}^{km} \right)^\top dt \\ \hat{W}_o &= \frac{1}{L} \sum_{l=1}^{L} \frac{1}{d_l^2} \int_0^\infty \left(y^{li}(t) - \bar{y}^{li}\right)^\top \left(y^{lj} - \bar{y}^{lj}\right) dt \end{align*}
The mean \(\bar{x}\) for a function of time \(x(t)\) \[ \bar{x} = \lim_{t\rightarrow \infty} \frac{1}{t} \int_0^t x(\tau) d\tau. \] or other average over time. There are several types of ways to center the perturbations including using a nominal trajectory average over time \citep{himpe2018emgr}. A trajectory \(x^{km}(t)\) is the state of the system corresponding to the impulsive input \(u(t)= c_k e_m \delta(t) + \bar{u}\) where \(\delta(t)\) is the Kronecker-delta and \(\bar{u}\) is the current operating point. For the simulation, the Kronecker-delta is estimated by
\begin{equation*} \delta(t) = \begin{cases} \frac{1}{dt} & \text{if } t = 0 \\ 0 & \text{o.w.} \end{cases} \end{equation*}
A trajectory \(y^{il}(t)\) is the output of the system with initial condition \(x^{il}(0) = d_l e_i + x_0 \) where \(x_0\) is the nominal state as \(t\rightarrow \infty\) for an operating point \(u\). The empirical Gramians can be used in place of the Gramian matrices for calculating interaction measures.