\documentclass[12pt]{article} 
%\usepackage{amsbsy} % for \boldsymbol and \pmb 
\usepackage{graphicx} % To include pdf files!
\usepackage{amsmath}
\usepackage{amsbsy}
\usepackage{amsfonts}
\usepackage[colorlinks=true, pdfstartview=FitV, linkcolor=blue, citecolor=blue, urlcolor=blue]{hyperref} % For links
\usepackage{fullpage}
%\pagestyle{empty} % No page numbers

% Search instrumental and fix up.

\begin{document}

%\enlargethispage*{1000 pt} 



\begin{center}   
{\Large \textbf{STA 431s17 Assignment Eight}}\footnote{This assignment was prepared by  \href{http://www.utstat.toronto.edu/~brunner}{Jerry Brunner},
Department of Statistical Sciences, University of Toronto. It is licensed under a 
\href{http://creativecommons.org/licenses/by-sa/3.0/deed.en_US}
     {Creative Commons Attribution - ShareAlike 3.0 Unported License}. Use any part of it as you like and share the result freely. The \LaTeX~source code is available from the course website:
\href{http://www.utstat.toronto.edu/~brunner/oldclass/431s17} {\small\texttt{http://www.utstat.toronto.edu/$^\sim$brunner/oldclass/431s17}}}
\vspace{1 mm}
\end{center}


\noindent
The first three questions of this assignment are about how instrumental variables can help with measurement error and omitted variables at the same time; see Lecture slide set 13. When instrumental variables are not available, sometimes extra response variables allow us to purchase identifiability. Questions~\ref{1extra} through~\ref{SASIQ} explore this possibility.

The non-computer questions on this assignment are for practice, and will not be handed in.  For the SAS part of this assignment (Question~\ref{SASIQ}) please bring hard copy of your log file and your results file to the quiz. There may be one or more questions about them, and you may be asked to hand the printouts in with the quiz. 

\begin{enumerate} 

\item In the model pictured below, the explanatory variable $X$ is measured with error as well as being correlated with omitted variables. $Z$ is an instrumental variable.
\begin{center}
\includegraphics[width=2.2in]{2017A8path1}
\end{center}
        \begin{enumerate}
            \item Give the model equations in centered form. The centering can be invisible. 
            \item Guided by the symbols on the path diagram, provide notation for the variances and covariances of the error terms and exogenous variables. 
            \item Let $\boldsymbol{\theta}$ denote the vector of parameters you have witten down so far. These are the parameters that will appear in the covariance matrix of the observable data. What is $\boldsymbol{\theta}$?
            \item Does this model pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers. % 7 parameters, 6 equations. It would be just identifiable with kappa=0
(Notice that we are only trying to identify the parameters in $\boldsymbol{\theta}$, which is a function of the full parameter vector. The full parameter vector has intercepts and unknown probability distributions.)
            \item Calculate the covariance matrix $\boldsymbol{\Sigma}$ of $\mathbf{D_i}$, a single observable data vector. 
            \item Is the parameter $\beta$ identifiable provided $\phi_{12}\neq 0$? Answer Yes or No. If the answer is Yes, prove it. If the answer is No, give a simple numerical example of two parameter vectors with different $\beta$ values, yielding the same covariance matrix $\boldsymbol{\Sigma}$. 
            
            \item Why is is reasonable to assume $\phi_{12}\neq 0$?
            \item Now let's make the model more realistic and scary. The response variable is measured with error, so $V = Y + e_2$. Furthermore, because of omitted variables, all the error terms might be correlated with one another and with $X$. 
                \begin{enumerate}
                    \item Do your best to make a path diagram of the new model. You need not write symbols on the curved double-headed arrows you have added.
                    \item Show that $\beta$ is still identifiable.
                \end{enumerate} 
        \end{enumerate}
        
\item Here is a model with two explanatory variables and two instrumental variables. The path diagram looks busy, but it has features that make sense once you think about them. The instrumental and explanatory variables have covariance matrix $\boldsymbol{\Phi} = [\phi_{ij}]$, so that for example $Var(X_1) = \phi_{33}$. No doubt there are omitted explanatory variables that are correlated with $X_1$ and $X_2$, and affect $Y$. That is the source of $c_1=Cov(X_1,\epsilon)$ and  $c_2=Cov(X_2,\epsilon)$. The variables in the latent regression model are all measured (once) with error. Because of omitted variables in the measurement process, the measurement errors are correlated, with $3 \times 3$ covariance matrix $\boldsymbol{\Omega} = [\omega_{ij}]$. 
\begin{center}
\includegraphics[width=4in]{2017A8path2} % Replace with DoubleInstrumental1 when this goes in the text.
\end{center}
        \begin{enumerate}
            \item Give the model equations in centered form. The centering can be invisible. 
            \item How many parameters appear in the covariance matrix of the observable data? Scanning from the bottom, I get 10+2+2+6=20.
            \item Does this model pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers.
            \item The next step would be to calculate the covariance matrix $\boldsymbol{\Sigma}$ of the observable data vector, but that's a big job. To save work and also to reveal the essential features of the problem, please just calculate $cov\left( (Z_1,Z_2)^\top,(W_1,W_2,V)^\top \right)$.
            \item Are the parameters $\beta_1$ and $\beta_2$ identifiable? Answer Yes or No. If the answer is Yes, prove it. You don't have to finish solving for $\beta_1$ and $\beta_2$. You can stop once you have two linear equations in two unknowns, where the coefficients are $\sigma_{ij}$ quantities.  Presumably it's possible to solve two linear equations in two unknowns. To prove identifiability, you don't have to actually recover the parameters from the covariance matrix. All you have to do is show it can be done. In $\boldsymbol{\Sigma}$, please maintain the order  $Z_1,Z_2,W_1,W_2,V$ so we will have the same answer.
        \end{enumerate}

\item Here is a matrix version of instrumental variables. Independently for $i = 1, \ldots, n$, the centered model equations are 
\begin{eqnarray*} 
    \mathbf{Y}_i & = &  \boldsymbol{\beta} \mathbf{X}_i + \boldsymbol{\epsilon}_i \\
    \mathbf{W}_i & = &  \mathbf{X}_i + \mathbf{e}_{i,1}              \\
    \mathbf{V}_i & = &  \mathbf{Y}_i + \mathbf{e}_{i,2}. 
\end{eqnarray*}
The random vectors $\mathbf{X}_i$ and $\mathbf{Y}_i$ are latent, while $\mathbf{W}_i$ and $\mathbf{V}_i$ are observable. In addition, there is a vector of observable instrumental variables $\mathbf{Z}_i$. The random vectors $\mathbf{X}_i$ and $\mathbf{Z}_i$ are $p \times 1$, while $\mathbf{Y}_i$ is $q \times 1$. This determines the sizes of all the matrices. The variances and covariances are as follows:
$cov(\mathbf{X}_i)=\boldsymbol{\Phi}_x$, $cov(\mathbf{Z}_i)=\boldsymbol{\Phi}_z$, $cov(\mathbf{Z}_i,\mathbf{X}_i)=\boldsymbol{\Phi}_{zx}$, 
$cov(\boldsymbol{\epsilon}_i)=\boldsymbol{\Psi}$, 
$cov(\mathbf{X}_i, \boldsymbol{\epsilon}_i)=\mathbf{C}$,
and $cov\left( \begin{array}{c} \mathbf{e}_{i,1} \\ \mathbf{e}_{i,2}  \end{array} \right) = \boldsymbol{\Omega}$. All variance-covariance matrices are positive definite (why not), and in addition, the $p \times p$ matrix of covariances $\boldsymbol{\Phi}_{zx}$ has an inverse. Covariances that are not specified are zero; in particular, the instrumental variables  have zero covariance with the error terms.

Collecting $\mathbf{Z}_i$, $\mathbf{W}_i$, $\mathbf{V}_i$ into a single long data vector $\mathbf{D}_i$, we write its variance-covariance matrix as a partitioned matrix:
\begin{displaymath} 
    \boldsymbol{\Sigma} = 
    \left( \begin{array}{c|c|c}
        \boldsymbol{\Sigma}_{11} & \boldsymbol{\Sigma}_{12} & 
        \boldsymbol{\Sigma}_{13}  \\ \hline
                                 & \boldsymbol{\Sigma}_{22} &     
        \boldsymbol{\Sigma}_{23} \\ \hline
             &  &  \boldsymbol{\Sigma}_{33} 
    \end{array} \right), 
\end{displaymath}   
where $cov(\mathbf{Z}_i, \mathbf{W}_i) = \boldsymbol{\Sigma}_{12}$, and so on.

    \begin{enumerate}
        \item Give the dimensions (number of rows and number of columns) of the following matrices: $\boldsymbol{\beta}$, $\boldsymbol{\Psi}$, $\boldsymbol{\Omega}$, $\boldsymbol{\Sigma}_{23}$.
        % qxp,              qxq,                 (p+q)x(p+q)              pxq
         
        \item This problem fails the test of the Parameter Count Rule, though you are not required to show it. Fortunately, all we care about is $\boldsymbol{\beta}$. Doing as little work as possible, prove that $\boldsymbol{\beta}$ is identifiable by showing how it can be recovered from the $\boldsymbol{\Sigma}_{ij}$ matrices.  
        \item Give the formula for an estimator of $\boldsymbol{\beta}$ and show that it is consistent.
    \end{enumerate}

% q*p + p*(p+1)/2 + p*(p+1)/2 + p^2 + q*(q+1)/2 +p*q + (p+q)*(p+q+1)/2 - (2+p+q)*(2*p+q+1)/2
% 3/2*p^2 + 3/2*p*q + 1/2*q^2 - p - 1/2*q - 1

% \newpage

        
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\item \label{1extra} When instrumental variables are not available, sometimes identifiability can be obtained by adding more response variables to the model.  Independently for $i=1, \ldots, n$, let
\begin{eqnarray*} 
   W_{i\mbox{~}} & = &   X_i + e_i            \\  
   Y_{i,1}          & = &  \beta_1 X_i + \epsilon_{i,1}  \\  
   Y_{i,2}          & = & \beta_2 X_i + \epsilon_{i,2}  
\end{eqnarray*}
where $X_i$, $e_i$,  $\epsilon_{i,1}$ and $\epsilon_{i,2}$ are all independent,
$Var(X_i)=\phi$, $Var(e_i)=\omega$, $Var(\epsilon_{i,1})=\psi_1$, $Var(\epsilon_{i,2})=\psi_2$, and all the expected values are zero. The explanatory variable $X_i$ is latent, while $W_i$, $Y_{i,1}$ and $Y_{i,2}$ are observable  

        \begin{enumerate}
            \item Make a path diagram for this model 
            \item What are the unknown parameters in this model? 
            \item Let $\boldsymbol{\theta}$ denote the vector of Greek-letter unknowns that apear in the covariance matrix of the observable data. What is $\boldsymbol{\theta}$? 
            \item Does this model pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers.
            \item Calculate the variance-covariance matrix of the observable variables. Show your work.
            \item The parameter of primary interest is $\beta_1$. Is $\beta_1$ identifiable at points in the parameter space where $\beta_1=0$? Why or why not?
            \item Is $\omega$ identifiable where $\beta_1=0$? 
            \item Give a simple numerical example to show that $\beta_1$ is not identifiable at points in the parameter space where $\beta_1 \neq 0$ and $\beta_2=0$.
            \item Is $\beta_1$ identifiable at points in the parameter space where $\beta_2 \neq 0$? Answer Yes or No and prove your answer. 
            \item Show that the entire parameter vector is identifiable at points in the parameter space where $\beta_1 \neq 0$ and $\beta_2 \neq 0$. 
            \item Propose a Method of Moments estimator of $\beta_1$. 
            \item How do you know that your estimator cannot be consistent in a technical sense? 
            \item For what points in the parameter space will your estimator converge in probability to $\beta_1$? 
            \item How do you know that your Method of Moments estimator is also the Maximum Likelihood estimator (assuming normality)? 
            \item Explain why the likelihood rato test of $H_0: \beta_1=0$ will fail. Hint: What will happen when you try to locate $\widehat{\theta}_0$?
%            \item Recall that an \emph{instrumental variable} is an observable variable that has non-zero covariance with the explanatory variable and zero covariance with the error term in the regression, which in this case is $\epsilon_{i,1}$. Under what condition is $Y_2$ an instrumental variable for $X$?
            \item Since the parameter of primary interest is $\beta_1$, it's important to be able to test $H_0: \beta_1=0$. So at points in the parameter space where  $\beta_2 \neq 0$, what \emph{two} equality constraints on the elements of $\boldsymbol{\Sigma}$ are implied by $H_0: \beta_1=0$? Why is this unexpected? % See discussion in the text. It would be worth simulating a large data set to illustrate this. 
            \item Assuming $\beta_1 \neq 0$ and $\beta_2 \neq 0$, you can use the model to deduce more than one testable \emph{inequality} constraints on the variances and covariances. Give at least one example.
        \end{enumerate}
% The main lesson here is that extra response variables can yield identifiability, but also (a) parameters can be identifiable in some regions and not others, and (b) identifiability when H0 is true needs to be checked separately. 

% This is directly an example from the text. I've re-checked it carefully and it's very good. In 2017, the students don't have it yet.
\item In the model of Question~\ref{1extra}, suppose that $X$ and the extra response variable $Y_2$ are influenced by common omitted variables, so that there is non-zero covariance between $X$ and $\epsilon_2$. Here is a path diagram.
\begin{center}
\includegraphics[width=4in]{extra1bex} % In the text
\end{center}
    \begin{enumerate}
        \item How do you know that the full set of parameters (that is, the ones that appear in $\boldsymbol{\Sigma}$) cannot possibly be identifiable in most of the parameter space?
        \item Calculate the variance-covariance matrix of the observable variables. How does your answer compare to the one in Question~\ref{1extra}?
        \item Primary interest is still in $\beta_1$. Propose a Method of Moments estimator of $\beta_1$. Is it the same as the one in Question~\ref{1extra}, or different?
        \item For what set of points in the current parameter space is $\beta_1$ identifiable? % beta2 and kappa cannot both be zero.
        \item \label{checkid} If you had data in hand, what null hypothesis could you test about the $\sigma_{ij}$ quantities to verify the identifiability of $\beta_1$? % sigma13=sigma23=0
        \item Suppose you want to test $H_0:\beta_1=0$, which is likely the main objective.
        \begin{enumerate}
            \item If you rejected the null hypothesis in Question~\ref{checkid}, what null hypothesis would you test about the $\sigma_{ij}$ quantities to test $H_0:\beta_1=0$?
            \item If you failed to reject the null hypothesis in Question~\ref{checkid}, could you still test $H_0:\beta_1=0$? What is the test on $\sigma_{ij}$ quantities in this case?
            \item If you rejected $H_0:\beta_1=0$, naturally you would want to state whether $\beta_1$ is positive or negative. Is this possible? % Yes, look at the sign of sigma12 and/or sigma23/sigma13.  
        \end{enumerate}
    \end{enumerate}

% But if Cov(epsilon1,epsilon2) ne 0 we are just dead.

% \newpage

    \item \label{IQtheory} Question \ref{SASIQ} will use the \emph{Longitudinal IQ Data}. IQ is short for ``Intelligence Quotient," and IQ tests are attempts to measure intelligence. A score of 100 is considered average.  Most IQ tests have many sub-parts, including vocabulary tests, math tests, logical puzzles, tests of spatial reasoning, and so on. What the better tests probably succeed in doing is to measure one \emph{kind} of intelligence --- potential for doing well in school. Of course, they measure it with error.

In the Longitudinal IQ Data, the IQs of adopted children were measured at ages 2, 4, 8 and 13. The birth mother's IQ was assessed at the time of adoption, and the adoptive mother's education (in years) was also recorded. The variables are
        \begin{itemize}
            \item Adoptive mother's education
            \item Birth mother's IQ
            \item IQ at age 2
            \item IQ at age 4
            \item IQ at age 8
            \item IQ at age 13
        \end{itemize}
Here is a path diagram for just the IQ part of the data.         
\begin{center}
\includegraphics[width=4in]{2017A8IQpath} 
\end{center}   
$X$ is birth mother's true academic ability, $W$ is birth mother's measured IQ, $Y_1$ is child's measured IQ at age 2, and so on. Of course child's ability is a latent variable too, but we've re-parameterized, making the response variables appear to be observable. This model is far from perfect, but we'll go with it for now. 

        \begin{enumerate}
            \item Write the model equations in scalar form.
            \item Let $\boldsymbol{\theta}$ denote the vector of Greek-letter unknowns that apear in the covariance matrix of the observable data. What is $\boldsymbol{\theta}$? 
            \item Does this model pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers.
            \item Calculate the variance-covariance matrix of the observable variables. Show your work.
            \item The parameter vector is identifiable except on a set of volume zero in the parameter space. Show how $\beta_2$ can be recovered from $\sigma_{ij}$ quantities. 
            \item If $\beta_1=0$, is it identifiable? 
            \item How many $\beta$s need to be non-zero for all the parameters to be identifiable? 
            \item \label{fitdf} How many degrees of freedom will there be in the likelihood ratio test for model fit? The answer is a number.
            \item \label{eqdf} Suppose you want to test whether all four regression coefficients are equal, using a likelihood ratio test. What are the degrees of freedom for this test?
        \end{enumerate}
% \end{enumerate}


    \item \label{SASIQ}
The longitudinal IQ data described in Question~\ref{IQtheory} are given in the file
\href{http://www.utstat.toronto.edu/~brunner/data/illegal/origIQ.data.txt}
     {\texttt{origIQ.data.txt}}.
These data are taken from \emph{The Statistical Sleuth} by F. Ramsey and D. Schafer, and are used without permission. 

    \begin{enumerate}
        \item Start by reading the data. There are $n=62$ cases, so please verify that you are reading the correct number of cases. Now run \texttt{proc~corr} to produce a correlation matrix of all the variables (with tests), including adoptive mother's education.
        \item Do the test on the correlations support omitting adoptive mother's education from the model? 
        \item How are the \texttt{proc~corr} results helpful in justifying your identifiability condition from the Question~\ref{IQtheory}?
        \item Please fit the model of Question~\ref{IQtheory}. We'll call this the \emph{full model}. 
        \item Sticking strictly to the $\alpha=0.05$ significance level, does the full model fit the data adequately? Answer Yes or No\footnote{My $p$-value of $0.0707$ is too close to $0.05$ for comfort. This model is hard to defend. It says the only reason that kids' scores at different ages are correlated is the mother's academic potential as an adult. Just for starters, these children do technically have fathers. Also, the model fails to distinguish between genotype and phenotype, and it has other weaknesses.}, and give a value of $G^2$, the degrees of freedom and the $p$-value. These numbers are all directly on your printout. Do the degrees of freedom agree with your answer to Question~\ref{fitdf}?
        \item Now fit the reduced model in which all the regression coefficients are equal. Using a calculator (or \texttt{proc~IML} if you want to), calculate the likelihood ratio test comparing the full and reduced models. Obtain $G^2$, a number. 
        \item What are the degrees of freedom for this test? Compare your answer to Question~\ref{eqdf}.
        \item Using this table of critical values, do you reject $H_0$ at $\alpha=0.05$? Answer Yes or No. Does birth mother's IQ seem to affect her child's IQ to the same degree at different ages?
\begin{verbatim}
> df = 1:8
> CriticalValue = qchisq(0.95,df)
> round(rbind(df,CriticalValue),3)
               [,1]  [,2]  [,3]  [,4]  [,5]   [,6]   [,7]   [,8]
df            1.000 2.000 3.000 4.000  5.00  6.000  7.000  8.000
CriticalValue 3.841 5.991 7.815 9.488 11.07 12.592 14.067 15.507

\end{verbatim}
        \item Just one more thing. Please go back to the \texttt{proc calis} run where you fit the full model, and use the \texttt{simtests} statement to do a Wald test of $H_0: \beta_1 = \beta_2 = \beta_3 = \beta_4$. Do you reach the same conclusion? SAS makes it easy to do Wald tests and hard to do likelihood ratio tests, but likelihood ratio tests perform better for relatively small samples like this, so we'll believe the likelihood ratio test.

    \end{enumerate}

\end{enumerate}



\vspace{70mm}

\noindent Bring your log file and your results file to the quiz. You may be asked for numbers from your printouts, and you may be asked to hand them in. There are lots of \textbf{There must be no error messages, and no notes or warnings about invalid data on your log file.} If you see a reference to the Moore-Penrose inverse, you've made a coding error.


\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


% More LongIQ for next time:

        \item That was interesting, but now let's bring in adoptive mother's education. We wonder whether, controlling for birth mother's true IQ, adoptive mother's true education is related to the child's true IQ. Write the model equations. There are now \emph{two} regression coefficients for each response variable. Don't bother with the intercepts.
        \item Calculate the covariance matrix in terms of the model parameters. Does this model pass the test of the parameter count rule?
        \item There is still hope. Your model has a term $\phi_{12}$, representing the covariance between birth mother's true IQ and adoptive mother's true education. But unless the adoption agency acted in a very peculiar way, there is no reason these variables should be related. Furthermore, it's testable without actually fitting the model under consideration. Locate the test on your printout (it's there) and give the $p$-value.
        \item This is exploratory data analysis, so let's tentatively accept the (null) hypothesis $\phi_{12}=0$. Under this assumption, your covariance matrix simplifies quite a bit. Either re-write it, or else circle the terms with $\phi_{12}$, to remind yourself that they equal zero.
        \item The regression coefficients linking adoptive mother's education to child's IQ at various ages are now identifiable, meaning they are identifiable at points in the parameter space where $\phi_{12}=0$. Recover one of them from the covariance matrix just to show you can do it. % In 2017 I think I can do this, but it's not as easy as the question implies.
        \item Under the null hypothesis that adoptive mother's true education has no effect on child's IQ at any age, four covariances in $\boldsymbol{\Sigma}$ should be zero (assuming $\phi_{12}=0$, of course). Which ones are they?
        \item Give the $p$-values, numbers from your printout. What do you conclude? Do these data support a link between adoptive mother's education and child's IQ?

 
This story could be continued a bit more, but even as it stands, it's pretty good. The lesson is that valid inference about a latent variable model may be possible even when the model parameters cannot be estimated.




% A big question I cut out

    \item \label{twovars} Independently for $i=1, \ldots, n$, let
\begin{eqnarray*}
    W_{i,1} &=& X_{i,1} + e_{i,1}   \\ 
    W_{i,2} &=& X_{i,2} + e_{i,2}   \\ 
    Y_{i,1} &=& \beta_1 X_{i,1} + \epsilon_{i,1}   \\
    Y_{i,2} &=& \beta_2 X_{i,2} + \epsilon_{i,2}   \\
    Y_{i,3} &=& \beta_3 X_{i,1} + \beta_4 X_{i,2} + \epsilon_{i,3}   
\end{eqnarray*}
where 
\begin{itemize}
     \item The $X_{i,j}$ variables are latent, while the $W_{i,j}$ and $Y_{i,j}$ variables are observable. 
     \item $e_{i,1}\sim N(0,\omega_1)$ and $e_{i,2}\sim N(0,\omega_2)$.
     \item $\epsilon_{i,j} \sim N(0,\psi_j)$ for $j=1,2,3$.
     \item $e_{i,j}$ and $\epsilon_{i,j}$ are independent of each other and of $X_{i,j}$.
     \item $X_{i,j}$ have expected value zero and
\end{itemize}
\begin{displaymath}
V\left(
\begin{array}{c} X_{i,1} \\ X_{i,2} \end{array}
 \right) = \left(
\begin{array}{c c} 
\phi_{11} & \phi_{12} \\
\phi_{12} & \phi_{22}
\end{array} \right).
\end{displaymath} 
Denote the vector of observable data by $\mathbf{D}_i = (W_{i,1}, W_{i,2}, Y_{i,1}, Y_{i,2}, Y_{i,3})^\top$, with $V(\mathbf{D}_i) = \boldsymbol{\Sigma} = [\sigma_{ij}]$. 

Among other things, this question illustrates how the search for identifiability can be supported by exploratory data analysis. Hypotheses about \emph{single} covariances, like $H_0: \sigma_{ij}=0$ can be tested without effort by looking at tests of the corresponding correlations. These tests are produced automatically by \texttt{proc~corr}.

\vspace{5mm}

       \begin{enumerate}
            \item What is the parameter vector $\boldsymbol{\theta}$ for this model? 
            \item \label{countingruleQ} Does this problem pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers. % 12 parameters, 5(5+1)/2 = 15 moments.
            \item Calculate the variance-covariance matrix of the observable variables. Show your work.
            \item The parameter $\phi_{12}$ is identifiable. How?
            \item Suppose $\beta_1=0$. Why is the parameter $\beta_1$ identifiable? Of course the same applies to $\beta_2$.
            \item But the idea here is that $Y_1$ and $Y_2$ are instrumental variables, so that $\beta_1 \neq 0$ and $\beta_2 \neq 0$. What hypotheses about \emph{single} covariances would you test to verify this?
            \item From this point on, suppose we have verified $\beta_1 \neq 0$ and $\beta_2 \neq 0$. Under what circumstances (that is, where in the parameter space) can the parameters $\beta_1$ and $\beta_2$ be easily identified?
            \item What hypotheses about \emph{single} covariances would you test to persuade yourself that this is okay?
            \item  Assuming the last step worked out well, give a formula for $\beta_1$ in term of $\sigma_{ij}$ values. 
            \item \label{beta2hat} Suppose you were sure $\phi_{12} \neq 0$, but you were not so sure about normality so you were uncomfortable with maximum likelihood estimation. Suggest a nice estimator of $\beta_2$. Why are you sure it is consistent? Note that even if you were interested in the MLE, this estimate would be an excellent starting value.
            \item Suppose your test for $\phi_{12} = 0$ did not reject the null hypothesis, so  dividing by $\sigma_{12}$ makes you uncomfortable. Show that even if $\phi_{12} = 0$, 
% the parameters $\beta_1$ and $\beta_2$ are still identifiable except on a set of volume zero in the parameter space. 
%           \item What covariance would you test to verify that the true parameter vector is \emph{not} in that volume zero set where $\beta_1$ 
there is another way to identify $\beta_1$. What assumption to you have to make (that is, where in the parameter space does the true parameter vector have to be) for this to work? How would you test it? % \beta1 = sigma35/sigma15, beta2 = sigma45/sigma25
            \item How could you identify $\beta_2$ if $\phi_{12} = 0$? 
            \item In question \ref{beta2hat}, you gave an estimator for $\beta_2$ that is consistent in most of the parameter space. Based on your answer to the preceding question, give a second estimator for $\beta_2$ that is consistent in most of the parameter space. It should be geometrically obvious that except for a set of volume zero in the parameter space, \emph{both} estimators are consistent. % Bringing up the possibility of different asymptotic efficiency depending on there in the parameter space the true parameter is located. Lovely. 
            \item Assuming $\beta_1$ and $\beta_2$ are identifiable one way or the other, now we seek to identify $\phi_{11}$ and $\phi_{22}$. How can this be done? Give the formulas. Also, give a consistent estimator of $\phi_{22}$ that is not the MLE. Why are you sure it's consistent?
            \item Since  $Y_1$ and $Y_2$ are instrumental variables, primary interest is in $\beta_3$ and $\beta_4$, the coefficients linking $Y_3$ to $X_1$ and $X_2$. If our efforts so far have been successful (which they are, except on a set of volume zero in the parameter space), then $\beta_3$ and $\beta_4$ can be identified as the solution to two linear equations in two unknowns. Write these equations \emph{in matrix form}. 
            \item What condition on the $\phi_{ij}$ values ensure a unique solution to the two equations in two unknowns? Is this a lot to ask? 
            \item Now let's back up, and admit that the identification of $\beta_3$ and $\beta_4$ is really the whole point, since they are the parameters of interest. We have seen that $\phi_{12}$ is always identifiable. If $\phi_{12} \neq 0$, it can be used to identify $\beta_1$ and $\beta_2$, and they can be used to identify $\phi_{11}$ and $\phi_{22}$. Then $\beta_3$ and $\beta_4$ can be identified by solving the two equations in two unknowns. Now suppose that $\phi_{12}=0$. In this case $\beta_3$ and $\beta_4$ can be identified without knowing the values of $\phi_{11}$ and $\phi_{22}$, provided $\beta_1$ and $\beta_2$ are non-zero. Show how this can be done. % \beta_3=\sigma_{3,5}/\sigma_{13}, \beta_4=\sigma_{4,5}/\sigma_{2,4}
            \item Assuming that the parameters appearing in the covariances of $\boldsymbol{\Sigma}$ are identifiable, the additional 5 parameters (whch appear only in the variances) may be identified by subtraction. So we see that except on a set of volume zero in the parameter space, all the parameters are identifiable. In that region, how many equality constraints should the model impose on the covariance matrix? Use your answer to Question~\ref{countingruleQ}.
            \item To see what the equality constraints are, note that earlier parts of this question point to two ways of identifying $\beta_1$ and two ways of identifying $\beta_2$. There are also two simple ways to identify $\phi_{12}$. So write down the three constraints. Multiply through by the denominators.
            \item Now you have three equalities involving products of $\sigma_{ij}$ terms. For each one, use your covariance matrix to write both sides in terms of the model parameters. For each equality, does it hold everywhere in the parameter space, or are there some points in the parameter space where it does not hold? If there are points in the parameter space where an equality does not hold, state the set explicitly. 
            \item The idea here is that the three degrees of freedom in the likelihood ratio test of model fit correspond to three equalities involving the covariances, and those equalities are directly testable without the normality assumption\footnote{It's true that I have not told you how to do this yet, but it's not hard.} required by the likelihood ratio test. State the null hypothesis (there's just one) in terms of the $\sigma_{ij}$ quantities.
            \item If the null hypothesis were rejected, what would you conclude about the model?
            \item In ordinary multivariate regression (which has more than one response variable), it is standard to assume the error terms for the response variable may have non-zero covariance. Suppose, then, that $Cov(\epsilon_{i,1},\epsilon_{i,2}) = \psi_{12}$. How would this change the covariance matrix?
            \item Always remembering that $\beta_1$ and $\beta_2$ are non-zero, suppose that $\phi_{12}=0$. Is $\psi_{12}$ identifiable? What if $\phi_{12}\neq 0$?
            \item Well, what if there were non-zero covariances $\psi_{13}$ and $\psi_{23}$ as well? What does the parameter count rule tell you? 
            \item  Again by the parameter count rule, $\phi_{12}\neq 0$ is absolutely necessary to identify the entire parameter if all three $\psi_{ij}$ are added to the model. Why? In this case, are $\psi_{13}$ and $\psi_{23}$ identifiable? Why or why not?
        \end{enumerate}

% Need to work out whether correlation between the EXPLANATORY variables and \epsilon_{i,1},\epsilon_{i,2} would make beta3 and beta4 non-identifiable. 
% In the textbook, discuss why H_0 beta3=beta4=0 is still a 2df test even though it produces 4 constraints on the sigmas, as long as the whole parameter is identifiable.

% Here are more that were cut out in 2017.


\item  Independently for $i=1, \ldots, n$, let
\begin{eqnarray*} 
   Y_{i,1} & = & \alpha_1 + \beta_1 X_{i,1} + \epsilon_{i,1}    \\ 
   Y_{i,2} & = & \alpha_2 + \beta_2 X_{i,2} + \epsilon_{i,2}    \\
   W_{i,1} & = & \nu_1 +  X_{i,1} + e_{i,1}                     \\  
   W_{i,2} & = & \nu_2 +  X_{i,2} + e_{i,2}                     \\ 
   V_{i,1} & = & \nu_3 +  Y_{i,1} + e_{i,3}                     \\  
   V_{i,2} & = & \nu_4 +  Y_{i,2} + e_{i,4},               
\end{eqnarray*}
where $E(X_{i,j})=\mu_j$, $e_{i,j}$ and  $\epsilon_{i,j}$ are independent of one another and of $X_{i,j}$, $Var(e_{i,j})=\omega_j$, $Var(\epsilon_{i,j})=\psi_j$, and
\begin{displaymath}
V\left(
\begin{array}{c} X_{i,1} \\ X_{i,2} \end{array}
 \right) = \left(
\begin{array}{c c} 
\phi_{11} & \phi_{12} \\
\phi_{12} & \phi_{22}
\end{array} \right).
\end{displaymath} 
In this model, $X_{i,1}$, $X_{i,2}$, $Y_{i,1}$ and $Y_{i,2}$ are latent variables, while $W_{i,1}$, $W_{i,2}$, $V_{i,1}$ and $V_{i,2}$ are observable. 
  \begin{enumerate}
        \item Make a path diagram for this model.
        \item Does this model fit the double measurement design?
        \item Calculate the variance-covariance matrix of the observable variables. Show your work.
            \item Does this problem pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers.
        \item Show that $\beta_1$ and $\beta_2$ are identifiable provided $\phi_{12} \neq 0$.
        \item Are there any other points in the parameter space where $\beta_1$ is identifiable? $\beta_2$?
        \item Give one testable equality constraint (a statement about the $\sigma_{ij}$ quantities) that is implied by the model. Is it still true with $\phi_{12}=0$? $\beta_1 = 0$? $\beta_2 = 0$?
        \item Suppose you wanted to estimate $\beta_1$. Suggest a \emph{statistic} (function of the sample data) to serve as an estimator.
        \item Is your estimator consistent? Under what circumstances? You don't have to prove anything in detail. 
        \item If the primary interest is in $\beta_1$, do we really need the response variable $Y_{i,2}$?
        \item Does any variable in this model qualify as an instrumental variable?
  \end{enumerate}

\newpage  

    \item In this problem, $W_i$ and the $Y_{ij}$ variables are observable, and $X_i$ is latent. The response variable of primary interest is $Y_{i,1}$, while $Y_{i,2}$ and $Y_{i,3}$ are included to help with identifiability. The point of the question is that the error terms need not all be independent for this to work.

Independently for $i=1, \ldots, n$,
\begin{eqnarray} 
    Y_{i,1}  & = &  \beta_{0,1} + \beta_{1,1} X_i + \epsilon_{i,1}  \nonumber  \\
    Y_{i,2}  & = &  \beta_{0,2} + \beta_{1,2} X_i + \epsilon_{i,2}  \nonumber  \\
    Y_{i,3}  & = &  \beta_{0,3} + \beta_{1,3} X_i + \epsilon_{i,3}  \nonumber  \\
    W_{i~}   & = &  X_i + e_i         \nonumber 
\end{eqnarray}
where 
\begin{itemize}
     \item $X_i \sim N(\mu_x,\phi)$ is a latent variable
     \item $e_i \sim N(0,\omega)$
     \item $\boldsymbol{\epsilon}_i = 
           (\epsilon_{i,1},\epsilon_{i,2},\epsilon_{i,3})^\top$
     \item $X_i$, $e_i$ and $\boldsymbol{\epsilon}_i$ are independent of one another
     \item $\boldsymbol{\epsilon}_i$ is multivariate normal with mean zero and covariance matrix
\end{itemize}
    \begin{displaymath}
    \boldsymbol{\Psi} = 
     \left[ \begin{array}{c c c}
                 \psi_{1,1} & \psi_{1,2} & 0 \\
                 \psi_{1,2} & \psi_{2,2} & \psi_{2,3}  \\
                 0          & \psi_{2,3} & \psi_{3,3} 
    \end{array} \right] .
    \end{displaymath}
        \begin{enumerate}
        \item Make a path diagram for this model 
            \item What is the parameter vector $\boldsymbol{\theta}$ for this model? 
            \item How many moment structure equations are there? You do not have to say what they are; just give a number. Don't forget the means.
            \item Does this problem pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers.
            \item Calculate the variance-covariance matrix of the observable variables. Remember that some covariances between errors are non-zero. Show your work.
            \item Solving the complete set of moment structure equations can be 
            done\footnote{Even the intercepts are identifiable from the mean vector $\boldsymbol{\mu}$, because there is no measurement bias term in this model. That's unrealistic, of course.}
            but it's a big chore. The primary interest is in the parameter $\beta_{1,1}$. Show that just this parameter is identifiable.
            \item Does any variable in this model qualify as an instrumental variable?
        \end{enumerate}

