6  Relationships Between Variables

Learning Objectives

By the end of this module you should be able to:

  1. Compute and interpret a correlation coefficient.
  2. Distinguish correlation from causation.
  3. Fit a simple linear regression and interpret the slope and intercept.
  4. Check a regression visually with a scatter plot and trend line.
  5. Explain the difference between \(R^2\) and correlation.
  6. Recognize when a linear model is not appropriate.

6.1 From One Variable to Two

Modules 1-5 were about one variable at a time: how is yield distributed? what is the mean? the spread? With Module 6 we start asking about relationships: how does yield change with acres? does fertilizer application predict yield? is price related to quality?

Almost every interesting question in data analysis is a question about a relationship between variables. The rest of this course is, in one way or another, about formalizing these questions.

6.2 Covariance and Correlation

The Intuition

Suppose you have two variables — say, fertilizer rate and yield — measured across 100 fields. You make a scatter plot. What does it look like?

  • If the points form an upward-sloping cloud, yield tends to increase with fertilizer.
  • If they form a downward-sloping cloud, yield tends to decrease with fertilizer.
  • If they are a shapeless blob, there is no obvious relationship.
  • If they form a tight line, the relationship is strong.
  • If they are widely scattered, the relationship is weak.

The correlation coefficient turns this visual intuition into a single number between -1 and +1.

The Formula

First, the covariance between two variables \(X\) and \(Y\), for a sample of size \(n\):

\[ \text{Cov}(X, Y) = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) \]

The covariance is positive when \(X\) and \(Y\) tend to be above (or below) their means together, and negative when one is above while the other is below. But its magnitude depends on the units of both variables, which makes it hard to interpret directly.

The (Pearson) correlation coefficient normalizes the covariance by the standard deviations:

\[ r = \frac{\text{Cov}(X, Y)}{s_X \, s_Y} \]

The result is always between -1 and +1, regardless of units.

  • \(r = +1\): perfect positive linear relationship.
  • \(r = 0\): no linear relationship.
  • \(r = -1\): perfect negative linear relationship.
  • \(|r|\) between 0 and 1 measures the strength of the (linear!) relationship.

In Excel and R

Excel:

=CORREL(A2:A101, B2:B101)

R:

cor(yields$fertilizer, yields$yield_bu_acre)

Or for a whole data frame:

cor(yields[, c("fertilizer", "acres", "yield_bu_acre")])

What Correlation Does Not Measure

Correlation measures linear relationships. If the true relationship between \(X\) and \(Y\) is a curve — say, \(Y\) rises with \(X\) and then falls — the correlation can be zero even though there is a very strong relationship. This is why you should always plot your data before computing a correlation. A single scatter plot tells you more than a single number.

There is a famous dataset called Anscombe’s Quartet — four datasets that all have the same mean, standard deviation, correlation, and regression line, but look completely different when plotted. We will discuss them in the worked example.

6.3 Correlation Is Not Causation

If you remember nothing else from this course, remember this sentence. Correlation does not imply causation.

There are several reasons two variables can be correlated without one causing the other:

  1. Coincidence. If you look at enough pairs of variables, some will be correlated by chance alone. There is a famous website (tylervigen.com) that collects absurd correlations: per-capita cheese consumption correlates with deaths by bedsheet entanglement. Neither causes the other. They are both just trending over time.
  2. Reverse causation. You observe that people who exercise more have lower weights. Does exercise cause low weight, or do people with low weight exercise more? Usually a bit of both.
  3. Confounding. A third variable causes both. Ice cream sales correlate with drowning deaths. Does ice cream cause drowning? No. Both are caused by summer.
  4. Selection. The dataset was assembled in a way that induces a spurious correlation. (We will see examples of this in Module 10.)

To establish causation, you generally need either a controlled experiment (randomly assign treatment and control) or some clever reasoning about the source of variation. This is the domain of causal inference, which you will meet in AREC 262.

For AREC 261, the point is: any time you see a correlation, the first question should be “is there a plausible causal story, and what else could explain this?” The second question should be “could it be the reverse direction?” And the third should be “what could be confounding this?”

6.4 Simple Linear Regression

Correlation summarizes the strength and direction of a linear relationship. Linear regression goes one step further: it estimates the equation of the line.

The Model

We write the simple linear regression model as:

\[ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i \]

where:

  • \(y_i\) is the response variable (the thing we want to predict), for observation \(i\).
  • \(x_i\) is the predictor variable (the thing we’re using to predict), for observation \(i\).
  • \(\beta_0\) is the intercept — the expected value of \(y\) when \(x = 0\).
  • \(\beta_1\) is the slope — the expected change in \(y\) per unit change in \(x\).
  • \(\varepsilon_i\) is the error term — the deviation of the actual \(y_i\) from the line. Accounts for everything that affects \(y\) but isn’t captured by \(x\).

The goal is to estimate \(\beta_0\) and \(\beta_1\) from data. The standard method is ordinary least squares (OLS), which picks the values that minimize the sum of squared residuals:

\[ \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \sum_{i=1}^{n} (y_i - (\hat{\beta}_0 + \hat{\beta}_1 x_i))^2 \]

You don’t need to memorize the derivation. The formulas turn out to be:

\[ \hat{\beta}_1 = \frac{\text{Cov}(X, Y)}{\text{Var}(X)} = r \cdot \frac{s_Y}{s_X} \]

\[ \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} \]

Notice that the slope involves the correlation, scaled by the ratio of standard deviations. And the intercept is chosen so that the line passes through the point \((\bar{x}, \bar{y})\).

In Excel

For a simple regression, you can use =SLOPE(y_range, x_range) and =INTERCEPT(y_range, x_range). Or add a trendline to a scatter plot (right-click a series → Add Trendline → Linear → check “Display equation”).

For more detail, use the Data Analysis ToolPak (File → Options → Add-ins → Analysis ToolPak). Once enabled, Data → Data Analysis → Regression gives you a full regression output with standard errors, \(R^2\), and so on.

In R

model <- lm(yield_bu_acre ~ fertilizer, data = yields)
summary(model)

The lm() function fits a linear model. The formula y ~ x means “\(y\) as a function of \(x\).” summary() prints the estimated coefficients, standard errors, \(t\)-statistics, \(p\)-values, and \(R^2\).

You can extract pieces of the model:

coef(model)           # the coefficients
fitted(model)         # the predicted values
residuals(model)      # the residuals
predict(model, newdata = data.frame(fertilizer = 100))  # predict for new data

6.5 Interpreting the Output

Suppose you fit yield_bu_acre ~ fertilizer and get:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   30.2      2.1       14.3   <2e-16 ***
fertilizer     0.15     0.02       7.5   1.2e-10 ***

Interpretation:

  • Intercept (30.2): the expected yield when fertilizer is zero. Sometimes this is meaningful (what would yield be with no fertilizer?) and sometimes it is an extrapolation to a region you do not have data for. Be careful.
  • Slope (0.15): for every additional unit of fertilizer, expected yield increases by 0.15 bushels per acre. The units matter. If fertilizer is in kg/ha and yield is in bu/ac, the slope is in (bu/ac) per (kg/ha).

Always state the interpretation in words that include the units. “Yield goes up by 0.15” is vague and wrong. “Each additional kilogram per hectare of fertilizer is associated with an expected 0.15 bu/ac increase in yield” is the kind of sentence you should be writing.

Note the careful word “associated.” I did not say “causes.” Whether a regression coefficient measures a causal effect is a separate question that depends on how the data was collected — see Section 6.3.

6.6 \(R^2\): How Well Does the Line Fit?

The regression line is the “best” line in the least-squares sense, but “best” does not mean “good.” A dataset where the points are scattered all over the place still has a best line; the question is whether the line explains any meaningful fraction of the variation.

\(R^2\) (pronounced “R squared”) measures the fraction of the variance in \(Y\) that is explained by the regression. It ranges from 0 (the line explains nothing) to 1 (the line explains everything, meaning all points lie exactly on the line).

For a simple regression, \(R^2 = r^2\) — the square of the correlation coefficient. So a correlation of 0.7 gives \(R^2 = 0.49\): about half the variance in \(Y\) is accounted for by \(X\).

In R, summary(model) reports it as Multiple R-squared (and also Adjusted R-squared, which you will meet in AREC 262). In Excel, it’s shown on the regression trendline if you check “Display R-squared value.”

Rough rules of thumb (very rough; context matters):

  • \(R^2 < 0.1\): the relationship explains very little.
  • \(R^2 \approx 0.3\): a noticeable but modest relationship.
  • \(R^2 \approx 0.5\): a strong relationship.
  • \(R^2 > 0.8\): either a very strong true relationship, or you are overfitting.

Agricultural yield data often has \(R^2\) in the 0.2-0.5 range for any single predictor, because yields depend on many things. Don’t be disappointed by a modest \(R^2\) if the model is nonetheless useful.

6.7 When Linear Regression Is (and Isn’t) Appropriate

Linear regression makes assumptions. The main ones:

  1. The relationship is approximately linear. If the true relationship is curved, a straight line will fit badly. Check with a scatter plot.
  2. Observations are independent. If you have repeated measurements on the same field, or fields in the same farm, the observations are not independent and your standard errors will be wrong. (Fixing this needs more advanced methods you will meet in AREC 262.)
  3. Residuals have constant variance. If the scatter around the line grows as \(X\) grows (a “fan shape”), this assumption is violated.
  4. Residuals are roughly normal. Matters for small samples; less important for large ones.
  5. No extreme outliers driving the result. A single weird point can dramatically change the slope.

Always make a scatter plot with the regression line on top before trusting a regression. If the data looks nothing like a line, don’t report a linear regression.

6.8 Worked Example: Fertilizer and Yield

[TBD: walk through a complete example with a canola fertilizer trial dataset, showing the scatter plot, correlation, regression, interpretation, and a sensible written summary of the findings.]

6.9 Test Bank Sample

  1. (Concept.) You find a correlation of 0.6 between two variables. What does that mean? What doesn’t it mean?
  2. (Formula.) Write the formula for the Pearson correlation coefficient.
  3. (Correlation vs causation.) Give an example of two variables that are correlated but where neither causes the other.
  4. (Regression.) You fit yield ~ fertilizer and get a slope of 0.2. Write a one-sentence interpretation.
  5. (R-squared.) Your regression has \(R^2 = 0.35\). What does this tell you?
  6. (Diagnostics.) Name three things you should check before trusting a regression result.

6.10 Practice Exercises

  1. Compute the correlation between two variables in the canola dataset. Interpret.
  2. Fit a simple linear regression and interpret the coefficients.
  3. Reproduce Anscombe’s quartet. What does it teach you?
  4. [TBD: a regression exercise with a deliberately misleading dataset.]