1 The Tennessee STAR experiment [30 points]

“Education production” is a topic much explored by economists. The terminology reflects that we think of school environment features as inputs that cost money, while student learning is the output that schools produce. A major question in the field is which inputs have the highest benefit/cost ratio, and one very costly input is class size - most likely, you~ll need to hire more teachers to reduce class size. An important experiment conducted in Tennessee was designed to precisely answer the question “Does class size impacts student performance?”.

Krueger (1999) analyzed the Project STAR, a longitudinal study that randomly assigned kindergarten students and their teachers to one of three groups beginning in the 1985–1986 school year. The three groups were small classes (13–17 students per teacher), regular-size classes (22–25 students), and regular/aide classes (22–25 students) which also included a full-time teacher’s aide. After their initial assignment, the design called for students to remain in the same class type for four years. Around 6000–7000 students were involved in the project each year. You can find part of the sample related to students who entered STAR in kindergarten here to answer the following questions.

  1. Create the dummy variables Free_lunch (takes 1 if lunch is “free”), White_asian (equal 1 if ethnicity is either “cauc” or “asian”) and Female - takes 1 if gender is “female”. Also, define the variable age as 1986-birth, i.e., compute the age of the children in 1986.

  2. The first question to ask about a randomized experiment is whether the randomization successfully balanced the subject’s characteristics across different groups. Although the STAR data failed to include any pretreatment test scores, we can look at some characteristics of students such as race, gender, age, and free lunch status, which is a good measure of family income since only poor children qualify for free school lunch. Compare the values of Free_lunch, White_asian, Female, and age across the three groups small, regular, regular+aide. Are those variables balanced?

  3. One way to get the causal effect of interest is to run a regression of the outcome score (in percentage points) on the treatment classtype. Run a regression of score on classtype using robust standard errors and explain the results.

Hint: Use feols() from the fixest package setting se="hetero".

CIA and Regression

  1. Schools with at least three classes in each grade could choose to enroll in the experiment. Hence, students were randomized but not schools, and that might be a source of bias. One way to wash out the bias is to control for schoolid. Run a regression of the outcome score (in percentage points) on the treatment classtype, controlling for schoolid and using robust standard errors. How do these results compare to what you got in (c)?

  2. Run a regression of the outcome score (in percentage points) on the treatment classtype, controlling for schoolid, Free_lunch, White_asian, female, and experience. Use robust standard errors. Did the estimates related to small and regular+aide change when you added covariates? Why/Why not?

  3. Do your new results represent the causal effect of class size on students’ scores? Why/Why not?

2 Omitted Variable Bias [30 points]

The standard economic theory of criminal behavior frames the engagement in illicit activities as a result of a rational choice, a decision that considers costs and benefits. The straightforward prediction of this theory is that people respond to changes in the expected costs and benefits of committing a crime. For instance, if there are more police on the streets or better police intelligence, that would increase the probability of arrest \((P_{A})\) faced by potential offenders. The increase in \(P_{A}\) generates a drop in crime rates since it increases the cost of committing a crime/decreases the expected utility of crime. This behavioral response of individuals is known as a deterrent effect.

In this question, we’ll use part of the dataset from Cornwell and Trumbuil (1993). Download the .RDS file here. The authors have data from counties in North Carolina from 1981 to 1987, and their goal was to estimate by how much crime would go down with an increase in the probability of arrest \((P_{A})\), i.e., the elasticity of crime with respect to \(P_{A}\).

  1. Read the dataset CT1993.RDS and get the average crime rate crmte and the average probability of arrest prbarr for each year from 1981 to 1987. Is there any clear pattern between those variables over time across North Carolina’s counties?

Hint: you want to gropu_by(year) and summarize() the data.

  1. Filter the data by year==1987. After that, construct a scatterplot with lcrmrte (log crime per person) in the y-axis and prbarr (probability of arrest) in the x-axis. How do those two variables relate?

  2. Estimate the following model by OLS with robust s.e. using the 1987 data:

\[lcrmrte_{i}=\beta_{0}+\beta_{1}prbarr_{i}+v_{i}\] and interpret the results.

  1. Estimate the following model by OLS with robust s.e. using the 1987 data:

\[lcrmrte_{i}=\gamma_{0}+\gamma_{1}prbarr_{i}+\gamma_{2}density_{i}+\eta_{i}\] where density is the population density (number of people per sq. mile). Why are the coefficients related to prbarr different in the two regressions?

  1. Estimate the following model by OLS using the 1987 data:

\[density_{i}=\theta_{0}+\theta_{1}prbarr_{i}+\varepsilon_{i}\] and show how \(\theta_{1}\) relates to \(\beta_{1}\) and \(\gamma_{1}\).

  1. Is either of the regressions likely to provide the true causal effect of the probability of arrest on crime rates? Why/Why not?


xkcd 552: Correlation

3 Propensity Score Matching [40 points]

In Brazil, there is a growing interest in the effect of studying in a Military School versus going to a regular public school. To attribute differences in student achievement to the type of school attended is complicated because students are not randomly allocated to Military or other public institutions. It is important to mention that there are two ways to enroll in a Military school: the student can either take a highly competitive entrance exam or attend a Military school for having a parent in the Army.

The dataset here1 contains (simulated) information about 3,704 students who attended different schools in Fortaleza-CE. At the end of Middle school, students from various institutions take a standardized exam. Your job is to get the causal effect of Military school attendance on students’ scores.

The variables are described below:

Table 3.1: Variables Description
Variable Definition
id Students’s Identification
test_score Standardized Test Scores
military 1 if attended a Military school, 0 otherwise
white 1 if student is white, 0 otherwise
income Student’s family income
mothers_edu 1 if mother’s education level is high school or below, 0 if some college or above
mothers_age Age of mother
  1. What is the result of a naive comparison of average test_score between the treatment (Military school attendance) and control groups (other public schools)? Is that difference statistically significant?

Hint: use the function read_csv() from tidyverse to import the dataset.

  1. Run t-tests to check for balance between treatment and control groups for all the covariates income, white, mothers_edu, and mothers_age. Are those variables balanced?

  2. Based on your answer for b), can you guess the direction of the bias? i.e., do you think that the naive comparison using non-experimental data underestimates or overestimates the causal effect of Military school attendance? Explain.

3.1 Regression with multiple covariates

  1. Run a regression adding the covariates income, white, mothers_edu, and mothers_age beside the treatment variable military. Is the effect of Military school attendance different using regression and using a simple difference in means? Why?

3.2 Nearest-Neighbor Matching

  1. Run the logit model to get the predicted probabilities of getting treated given the observed covariates. Plot the distributions of the propensity score in the treated and control groups. What do you see? Can we rely on the existence of common support?

  2. Restrict the sample finding the nearest neighbor of each treated student. Then, run two regressions (one with and the other without covariates) with the matched data. Contrast the results you got from the naive comparison and regression with the full sample. What assumptions do you rely on to claim that this is the true causal effect of Military school attendance on scores?

3.3 Weighting on the Propensity Score

  1. Trim the data based on the propensity score dropping observations with p-score<0.1 and p-score>0.9. Then run the weighted regression using the IPW as weights. Contrast the results you got with the ones from the nearest neighbor.

  2. Is any of those regressions likely to provide the true causal effect of Military school attendance on student scores? Why/Why not?

  1. Disclaimer: this is a simulated dataset. The purpose of this question is to improve your understanding of selection on observables. We cannot draw any conclusions about the effect of Military Schools based on this example.↩︎