Derivative of an eigenvalue dependent on a parameter

In numerical applications it is often the case that we have to differentiate a eigenvalue which is a function of some parameter. Let $H(\alpha )$$H(\alpha)$ be a non-singular matrix dependent on a real parameter $\alpha$$\alpha$. Let the eigenvalues be ${\lambda }_i(\alpha )$$\lambda_i(\alpha)$ and let the right and left eigenvectors be $v_i(\alpha )$$v_{i}(\alpha)$ and $u_i(\alpha )$$u_{i}(\alpha)$, respectively. Let us further assume that in expressions the eigenvectors are already normalized. Now suppose we want to calculate $\frac{\partial {\lambda }_i(\alpha )}{\partial \alpha }$$\frac{\partial \lambda_i(\alpha)}{\partial \alpha}$. The naive way to do this is via finite differences:

but this requires doing two eigendecompositions. Instead, we can appeal to the definition of the $i$$i$th eigenvalue:

Differentiate both sides w.r.t $\alpha$$\alpha$:

Now using the orthogonality relations between the right and left eigenvectors, i.e. $u_i^T{v}_j={\delta }_{ij}$$u^T_i v_j = \delta_{ij}$ we can see that $\frac{\partial u_i^T}{\partial \alpha }{v}_i=u_i^T\frac{\partial v_i}{\partial \alpha }=0$$\frac{\partial u^T_i}{\partial \alpha} v_i =u^T_i \frac{\partial v_i}{\partial\alpha} = 0$ because the derivatives are orthogonal to the vectors by normalization (the eigenvectors live on a $d$$d$-dimensional unit hypersphere, where $d$$d$ is the dimension of $H$$H$, and we can just throw away any changes that would bring us off the sphere). So the formula is

If nothing else you can get the derivative of $H$$H$ by automatic differentiation.

Degenerate Eigenvalues

What if ${\lambda }_i$$\lambda_i$ is degenerate with another eigenvalue ${\lambda }_j$$\lambda_j$ for some range of the parameter $\alpha$$\alpha$ ? Then we are no longer guaranteed the orthogonality from before since we have a two dimensional subspace in which we are free rotate our eigenvectors. Suppose we reorthogonalize $u_i$$u_i$ and $u_j$$u_j$ inside the degenerate subspace. Reviewing the steps, we find that the derivative $\frac{\partial u_i^T}{\partial \alpha }$$\frac{\partial u^{T}_{i}}{\partial \alpha}$ may rotate inside the degenerate subspace, acquiring components in the direction $u_j$$u_j$. Then we get

where $g(\alpha )$$g(\alpha)$ and $g^{\prime }(\alpha )$$g'(\alpha)$ are functions that measure the magnitude of the derivative in the direction of $u_j$$u_j$ and $v_j$$v_j$, respectively. Then what we need to do is clear: we need to construct $u_j$$u_j$ and $u_i$$u_i$ such that they are orthogonal to $v_i$$v_i$ and $v_j$$v_j$, respectively.

Eigenvectors

To get the derivatives of the eigenvectors of a Hermitian matrix with distinct eigenvalue ${\lambda }_i$$\lambda_i$ and normalized eigenvector $u_i$$u_i$, we differentiate

noting that the inverse is justified because the nullspace of $H-{\lambda }_{i}I$$H - \lambda_i I$ is spanned by $u_i$$u_i$, which is the direction that is subtracted from $\frac{\partial H}{\partial \alpha }$$\frac{\partial H}{\partial \alpha}$. Hence the action of $(H-{\lambda }_{i}I{)}^{-1}$$(H - \lambda_i I)^{-1}$ is unambiguous. We may use the Moore-Penrose pseudo inverse to write: