Model-Based Covariance Estimation for Regression M- and GM-Estimators

Tobias Schoch

1 Introduction

The population regression model is given by

where the population $U$$U$ is of size $N$$N$; the parameters $\mathit{\theta }$$\boldsymbol{\theta}$ and $\sigma$$\sigma$ are unknown; the ${\mathbf{x}}_i$$\bold x_i$'s are known values (possibly containing outliers), ${\mathbf{x}}_i\in {\mathbb{R}}^p$$\bold x_i \in \R^p$, $1\le p<N$$1 \leq p < N$; the $v_i$$v_i$'s are known positive (heteroscedasticity) constants; the errors $E_i$$E_i$ are independent and identically distributed (i.i.d.) random variables with zero expectation and unit variance; it is assumed that $\sum _{i\in U}{\mathbf{x}}_i{\mathbf{x}}_i^T/v_i$$\sum_{i \in U} \bold x_i \bold x_i^T / v_i$ is a non-singular $(p\times p)$$(p \times p)$ matrix.

It is assumed that a sample $s$$s$ is drawn from $U$$U$ with sampling design $p(s)$$p(s)$ such that the independence structure of model $\xi$$\xi$ is maintained. The sample regression $GM$$GM$-estimator of $\mathit{\theta }$$\boldsymbol{\theta}$ is defined as the root to the estimating equation ${\hat{\mathbf{\Psi }}}_n(\mathit{\theta },\sigma )=\mathbf{0}$$\widehat{\boldsymbol{\Psi}}_n(\boldsymbol{\theta}, \sigma) = \bold 0$ (for all $\sigma >0$$\sigma > 0$), where

where the function $\eta :\mathbb{R}\times {\mathbb{R}}^p\to \mathbb{R}$$\eta: \R \times \R^p \rightarrow \R$ parametrizes the following estimators

where $\psi :\mathbb{R}\to \mathbb{R}$$\psi:\R \rightarrow \R$ is a continuous, bounded, and odd (possibly redescending) function, and $h:{\mathbb{R}}^p\to {\mathbb{R}}_{+}$$h: \R^p \rightarrow \R_+$ is a weight function.

2 Covariance estimation

The model-based covariance matrix of $\mathit{\theta }$$\boldsymbol{\theta}$ is (Hampel et al., 1986. Chap. 6.3)

where

and ${\mathrm{E}}_{\xi }$$\mathrm{E}_{\xi}$ denotes expectation with respect to model $\xi$$\xi$. For the sample regression $GM$$GM$-estimator ${\hat{\mathit{\theta }}}_n$$\widehat{\boldsymbol{\theta}}_n$, the matrices $\mathbf{M}$$\bold M$ and $\mathbf{Q}$$\bold Q$ must be estimated. In place of $\mathbf{M}$$\bold M$ and $\mathbf{Q}$$\bold Q$ in ($\text{1}$$\ref{eq:cov}$), we have

where

and

Remarks

• The $i$$i$-th diagonal element of ${\mathbf{S}}_1$$\bold S_1$ and ${\mathbf{S}}_2$$\bold S_2$ depends on $h({\mathbf{x}}_i)$$h(\bold x_i)$, but the summation is over $j\in s$$j \in s$; see also Marazzi (1987, Chap. 6).
• When $\mathbf{W}$$\bold W$ is equal to the identity matrix $\mathbf{I}$$\bold I$, the asymptotic covariance of ${\hat{\mathit{\theta }}}_M$$\widehat{\boldsymbol{\theta}}_M$ is equal to the expression in Huber (1981, Eq. 6.5), which is implemented in the R packages MASS Venables and Ripley (2002) and robeth Marazzi (2020).
• For the Mallows and Schweppe type $GM$$GM$-estimators and given that $\mathbf{W}=\mathbf{I}$$\bold W = \bold I$, the asymptotic covariance coincides with the one implemented in package/ library robeth for the option averaged; see Marazzi (1993, Chap. 4) and Marazzi (1987, Chap. 2.6) on the earlier ROBETH-85 implementation.

3 Implementation

The main function—which is only a wrapper function— is cov_reg_model . The following display shows pseudo code of the main function.

cov_reg_model()
{
   get_psi_function()         // get psi function (function pointer)
   get_psi_prime_function()   // get psi-prime function (function pointer)
   switch(type) {
      case 0: cov_m_est()              // M-estimator
      case 1: cov_mallows_gm_est()     // Mallows GM-estimator
      case 2: cov_schweppe_gm_est()    // Schweppe GM-estimator
   }
   robsurvey_error()       // terminate with error in case of failure
}

The functions cov_m_est(), cov_mallows_gm_est(), and cov_schweppe_gm_est() implement the covariance estimators; see below. All functions are based on the subroutines in BLAS (Anderson et al., 1999) and LAPACK (Blackford et al., 2002).

To fix notation, denote the Hadamard product of the matrices $\mathbf{A}$$\bold A$ and $\mathbf{B}$$\bold B$ by $\mathbf{A}\circ \mathbf{B}$$\bold A\circ \bold B$ and suppose that $\sqrt{\cdot }$$\sqrt{\cdot}$ is applied element by element.

M-estimator (cov_m_est). The covariance matrix is (up to a scalar) equal to

and is computed as follows:

• Compute the factorization $\sqrt{\mathbf{w}}\circ \mathbf{X}:=\mathbf{Q}\mathbf{R}$$\sqrt{\bold w} \circ \bold X := \bold Q \bold R$ (LAPACK: dgeqrf).
• Invert the upper triangular matrix $\mathbf{R}$$\bold R$ by backward substitution to get ${\mathbf{R}}^{-1}$$\bold R^{-1}$ (LAPACK: dtrtri).
• Compute ${\mathbf{R}}^{-1}{\mathbf{R}}^{-T}$$\bold R^{-1} \bold R^{-T}$, which is equal to ($\text{2}$$\ref{eq:cov_m}$); taking advantage of the triangular shape of ${\mathbf{R}}^{-1}$$\bold R^{-1}$ and ${\mathbf{R}}^{-T}$$\bold R^{-T}$ (LAPACK: dtrmm).

Mallows GM-estimator (cov_mallows_gm_est). The covariance matrix is (up to a scalar) equal to

and is computed as follows:

• Compute the QR factorization: $\sqrt{\mathbf{w}\cdot \mathbf{h}}\circ \mathbf{X}:=\mathbf{Q}\mathbf{R}$$\sqrt{\bold w \cdot \bold h} \circ \bold X := \bold Q \bold R$ (LAPACK: dgeqrf).
• Invert the upper triangular matrix $\mathbf{R}$$\bold R$ by backward substitution to get ${\mathbf{R}}^{-1}$$\bold R^{-1}$ (LAPACK: dtrtri).
• Define a new matrix: $\mathbf{A}\leftarrow \sqrt{\mathbf{h}}\circ \mathbf{Q}$$\bold A \leftarrow \sqrt{\bold h} \circ \bold Q$ (extraction of $\mathbf{Q}$$\bold Q$ matrix with LAPACK: dorgqr).
• Update the matrix: $\mathbf{A}\leftarrow \mathbf{A}{\mathbf{R}}^{-T}$$\bold A \leftarrow \bold A \bold R^{-T}$ (taking advantage of the triangular shape of ${\mathbf{R}}^{-1}$$\bold R^{-1}$; LAPACK: dtrmm).
• Compute $\mathbf{A}{\mathbf{A}}^T$$\bold A \bold A^T$, which corresponds to the expression in ($\text{3}$$\ref{eq:cov_mallows}$); (LAPACK: dgemm).

Schweppe GM-estimator (cov_schweppe_gm_est). The covariance matrix is (up to a scalar) equal to

Put ${\mathbf{s}}_1=\mathrm{diag}({\mathbf{S}}_1)$$\bold s_1 = \mathrm{diag}(\bold S_1)$, ${\mathbf{s}}_2=\mathrm{diag}({\mathbf{S}}_2)$$\bold s_2 = \mathrm{diag}(\bold S_2)$, and let $\cdot /\cdot$$\cdot / \cdot$ denote elemental division (i.e., the inverse of the Hadamard product). The covariance matrix in ($\text{4}$$\ref{eq:cov_schweppe}$) is computed as follows

• Compute the factorization $\sqrt{\mathbf{w}\circ {\mathbf{s}}_1}\circ \mathbf{X}:=\mathbf{Q}\mathbf{R}$$\sqrt{\bold w \circ \bold s_1} \circ \bold X := \bold Q \bold R$ (LAPACK: dgeqrf).
• Invert the upper triangular matrix $\mathbf{R}$$\bold R$ by backward substitution to get ${\mathbf{R}}^{-1}$$\bold R^{-1}$ (LAPACK: dtrtri).
• Define a new matrix: $\mathbf{A}\leftarrow \sqrt{{\mathbf{s}}_2/{\mathbf{s}}_1}\circ \mathbf{Q}$$\bold A \leftarrow \sqrt{\bold s_2 / \bold s_1 } \circ \bold Q$ (extraction of $\mathbf{Q}$$\bold Q$ matrix with LAPACK: dorgqr).
• Update the matrix: $\mathbf{A}\leftarrow \mathbf{A}{\mathbf{R}}^{-T}$$\bold A \leftarrow \bold A \bold R^{-T}$ (taking advantage of the triangular shape of ${\mathbf{R}}^{-1}$$\bold R^{-1}$; LAPACK: dtrmm).
• Compute $\mathbf{A}{\mathbf{A}}^T$$\bold A \bold A^T$, which corresponds to the expression in ($\text{4}$$\ref{eq:cov_schweppe}$); (LAPACK: dgemm).

Note. Marazzi (1987) uses the Cholesky factorization (see subroutines RTASKV and RTASKW) which is computationally a bit cheaper than our QR factorization.

Literature

ANDERSON, E., BAI, Z., BISCHOF, C., BLACKFORD, L. S. , DEMMEL, J., DONGARRA, J., CROZ, J. D. , GREENHAUM, A., HAMMARLING, S., MCKENNEY, A. AND SORENSEN, D. (1999). LAPACK Users’ Guide. Philadelphia: Society for Industrial and Applied Mathematics (SIAM), 3rd edition. DOI: 10.1137/1.9780898719604

BLACKFORD, L. S., PETITET, A., POZO, R., REMINGTON, K., WHALEY, R. C., DEMMEL, J., DONGARRA, J., DUFF, I., HAMMARLING, S., HENRY, G., HEROUX, M., KAUFMAN, L. AND LUMSDAINE, A. (2002). An Updated Set of Basic Linear Algebra Subprograms (BLAS). ACM Transactions on Mathematical Software 28, 135–151. DOI: 10.1145/567806.567807

HAMPEL, F. R., RONCHETTI, E. M., ROUSSEEUW, P. J. AND STAHEL, W. A. (1986). Robust Statistics: The Approach Based on Influence Functions, New York: John Wiley and Sons. DOI: 10.1002/9781118186435

HUBER, P. J. (1981). Robust Statistics, New York: John Wiley and Sons. DOI: 10.1002/ 0471725250

MARAZZI, A. (1987). Subroutines for robust and bounded influence regression in ROBETH, Cahiers de Recherches et de Documentation, 3 ROBETH 2, Division de Statistique et Informatique, Institut Universitaire de Médecine Sociale et Préventive, Lausanne, ROBETH-85 Document No. 2, August 1985, revised April 1987.

MARAZZI, A. (1993). Algorithms, Routines, and S Functions for Robust Statistics: The FORTRAN Library ROBETH with an interface to S-PLUS, New York: Chapman and Hall.

MARAZZI, A. (2020). robeth: R Functions for Robust Statistics, R package version 2.7-6.

VENABLES, W. N. AND RIPLEY, B. D. (2002). Modern Applied Statistics with S, New York: Springer, 4th edition. DOI: 10.1007/978-0-387-21706-2