Introduction: How to Think About Data Analysis

Before we touch a spreadsheet or write a line of R code, we need to spend a little time on how to think about data analysis. The specific tools change every few years — the version of Excel you learn in this class will be replaced, R may one day be replaced by something newer, and whatever machine learning library is hot right now will be a footnote in a decade. What doesn’t change are the principles: the habits of mind that separate careful analysis from careless analysis.

Think of this introduction as the rules of the road. The rest of the book is about learning to drive.

Why Data Analysis?

You are probably reading this because AREC 261 is required for your degree program. Fair enough. But let me try to convince you that this material matters beyond a checkbox on your transcript.

Agriculture is now one of the most data-intensive sectors of the economy. A single modern combine harvester generates gigabytes of yield, moisture, and position data in a day. A dairy operation tracks the milk output, health, and reproductive status of every cow in near-real-time. Futures markets react to satellite measurements of crop greenness in Ukraine before most farmers have finished breakfast. Input suppliers use machine learning to recommend seed varieties, fertilizer rates, and spray timings. Government programs — crop insurance, AgriStability, environmental cross-compliance — all depend on data, and the people who design them need to understand data.

Whether you end up working as a farm manager, an agronomist, a commodity trader, a crop insurer, a policy analyst, a consultant, a researcher, or something none of us can predict yet, you will be surrounded by data. The question is whether you will be good at using it or used by people who are. This course is about being the former.

And even if you never touch a spreadsheet for work, there is a more basic reason to care: data analysis is a form of clear thinking. Learning to ask “how was this measured?”, “what is being compared to what?”, and “could this pattern be a coincidence?” is a life skill. It will save you from scams, bad advice, and political nonsense.

The Four Principles

I want to introduce four principles that will recur throughout the book. They are not tools. They are habits of mind. You should be thinking about them on every project you do, in this class and after.

Principle 1: Reproducibility and Transparency

Imagine you produce a report for your manager in March. It contains a chart showing that input costs per acre rose 18% over the last five years. In September, your manager asks you to update the chart with new data. You open the spreadsheet — but which spreadsheet? There are seven versions of it. You open the most recent one and realize you can’t remember how you calculated the cost per acre. Did you include seed? Fuel? Labour? Crop insurance premiums? You stare at the formulas for twenty minutes and then give up, rebuild the whole thing from scratch, and get a slightly different answer. Now you have two reports with two different numbers and no way to explain why.

This happens all the time. It happens to professionals with decades of experience. The solution is not “be smarter” or “have a better memory”. The solution is to build your analyses so they are reproducible: someone else — including future-you — should be able to take the raw data and the steps you wrote down and arrive at exactly the same answer.

A few terms that are easy to confuse:

Reproducibility means: given the same data, I can re-run the analysis and get the same results. This is about the plumbing of your analysis — the spreadsheet formulas, the R scripts, the way you imported the data.
Replicability means: a different person with a different dataset on the same underlying question gets a similar answer. This is a much harder standard and not something you can enforce on your own work.
Transparency means: someone reading your analysis can see what you did and why, not just the final number. A chart with no methods section is not transparent, no matter how pretty it is.

The thing to keep in mind is that every piece of analysis has an audience — and that audience is almost never just the person who signs your paycheque. The audience might be:

Your co-worker who inherits the file when you get promoted.
Your manager, who needs to defend the number in a meeting.
A regulator or auditor reviewing a compliance report.
Your professor, marking your assignment.
Most importantly: you, two months from now, when you have completely forgotten what any of this means.

Future-you is the most neglected audience in all of analytics. Be nice to her. Leave comments. Name your files sensibly. Write down what you did.

A related concept is version control. Version control means keeping a systematic record of how your analysis changes over time. At the professional level this is done with tools like Git (which you will meet briefly later in the course). At the beginner level it can be as simple as naming files like yield_analysis_2026-04-10.xlsx instead of yield analysis FINAL FINAL v3 (real one).xlsx. We will talk more about file-naming conventions in 1 Describing Data in Excel.

I should warn you up front: reproducibility is much harder in Excel than in R or any other scripting language. In Excel, your analysis is a tangle of cells that reference other cells. There is no “log” of what you did — no record of the order in which you typed things. In R, by contrast, your analysis is a script: a file you can re-run from top to bottom, which either produces the same answer it did last time or throws an error telling you where it broke. This is one of the major reasons we transition to R in 2 Introduction to R and Positron.

But don’t let that be an excuse to write sloppy Excel. One of the marks of a thoughtful analyst is that their Excel workbooks are still legible a year later, even though the tool does not force them to be.

Principle 2: Data Quality, or GIGO

There is a saying in computing: garbage in, garbage out (GIGO). It means that no amount of clever analysis can rescue bad data. If your yield column is measured in bushels per acre on some rows and tonnes per hectare on others, the prettiest regression in the world will give you nonsense. If your “organic” label was applied by someone who didn’t know what “organic” meant, your comparison of organic vs. conventional yields is meaningless.

A huge — often the majority — of real analysis work is spent on what we call data cleaning: checking that values are in the expected range, dealing with missing observations, reconciling units, fixing typos, matching up records across datasets. Students often find this work tedious. Experienced analysts know that it is the part of the job that most often determines whether your final answer is right.

Some things to ask about any dataset before you do anything else:

Where did it come from? Who collected it, and why?
What was measured, and how? Self-report? Sensor? Administrative record?
What are the units? (A surprising number of analyses have been ruined by mixed units.)
Are there missing values? How are they coded — blank, “NA”, -99, 9999?
Is the dataset a complete population, or a sample? If a sample, how was it drawn?
Has the data been edited, and if so, by whom? Could those edits have introduced bias?

We will revisit these questions throughout the book. For now just internalize the principle: the data is where the real work is, not the model.

Principle 3: Methods Quality

Once you have data you trust, the next question is: am I using the right method?

I want to introduce two ideas here that many students find uncomfortable at first:

All models are wrong. Every statistical or mathematical model you will meet in this course (and in life) is a simplification of reality. A linear regression assumes the relationship between \(X\) and \(Y\) is a straight line — but in the real world it almost never is exactly. A normal distribution assumes outcomes are symmetric and that extreme values are exponentially rare — but real yield distributions have fatter tails than that. A forecasting model assumes the future will behave like the past — but the future sometimes doesn’t.

Does this mean models are useless? No. It means your job is not to find a “true” model — there isn’t one — but to find one that is useful for the question you are actually trying to answer. The statistician George Box put it memorably: all models are wrong, but some are useful. Your job is to pick the least-wrong one for your purpose, and to be honest about its limitations.

Know your assumptions. Every technique you learn in this course comes with assumptions. Sometimes they are explicit (“the residuals should be approximately normally distributed”). Sometimes they are implicit (“this sample is representative of the population I want to draw conclusions about”). A responsible analyst does not just run the tool — they check the assumptions, and they state up front when the assumptions are shaky.

This is not about being exhaustively cautious. It is about being honest about the ground you are standing on.

Principle 4: Bias

Bias is a word that gets thrown around a lot, usually in the political sense of “people I disagree with are biased.” In data analysis, bias has a more specific meaning: a systematic tendency for your analysis to be wrong in a particular direction.

Two kinds to watch for:

Researcher bias. This is when the person doing the analysis — you — has an incentive, conscious or unconscious, to arrive at a particular answer. Maybe your boss is expecting the new fertilizer to work and you know it. Maybe you picked the subset of data where the effect showed up and ignored the rest. Maybe you ran ten different regressions and only reported the one with the significant p-value. All of these are researcher bias, and all of them are extraordinarily common even among honest, well-intentioned people.

The defence against researcher bias is procedure. Decide what you are going to do before you look at the results. Write it down. When you deviate from the plan, note that you deviated and why. Be suspicious of yourself.

Sample bias. This is when the data you have is not representative of the population you want to draw conclusions about. A classic example: if you survey farmers at a canola growers’ conference about what they think of canola policy, you will get different answers than if you surveyed all farmers, because the ones who go to canola conferences are systematically different from the ones who don’t. A more subtle example: if your yield dataset only includes fields that were actually harvested and recorded, it systematically excludes the worst fields — the ones that failed and were never harvested. Your average yield is therefore biased upward.

We will talk much more about sampling in 10 Sampling. For now, the principle is: the data you see is never all the data that exists, and the difference matters.

A Word About AI

You cannot read anything about data analysis in 2026 without encountering artificial intelligence. I want to say a few things about how AI fits into this course.

First: you are allowed and encouraged to use AI tools — GitHub Copilot, Claude, ChatGPT, whatever — to help you learn R and write code. These tools are very good at scaffolding a script, explaining an error message, or suggesting which function you want. Professional analysts use them every day and so should you.

Second: AI is not a substitute for understanding. If you have an AI write a script you don’t understand and it produces the right answer on the test bank, you will still fail the tests, because the tests are drawn from a bank of 200 questions and the AI’s code will not generalize to all of them unless you understand what it’s doing. More importantly, you will fail in your career, because every real analysis project involves noticing when the output is subtly wrong — which you cannot do if you don’t know what right looks like.

Third: AI is very good at producing plausible-sounding nonsense. It will confidently tell you that a function exists when it does not. It will confidently tell you that a dataset shows a pattern that it does not. It will confidently misremember the formula for a standard deviation. Your job, as a competent analyst, is to check its work, not trust it. This is the same skill you need for reading anything on the internet — AI just makes the nonsense come at you faster.

We will come back to AI specifically when we get to R (2 Introduction to R and Positron), but the principle is the same everywhere: AI is a power tool. It will make you more productive if you know what you’re doing, and more dangerous if you don’t.

How the Rest of This Book Is Organized

The book is divided into three parts that mirror the twelve modules of AREC 261:

Part 1 (Modules 1-6): Data Foundations. We start in Excel, because most of you have seen it before, and build up the core skills of loading, cleaning, transforming, summarizing, and visualizing data. Around Module 2 we introduce R and run the two tools in parallel for a few modules. By Module 6 you will be comfortable describing and visualizing relationships between variables.
Part 2 (Modules 7-9): Probability and Simulation. We introduce the idea of a random variable and the basics of probability, then spend two modules on simulation as a way to answer probability questions without having to know any formulas. Simulation is the foundation we use for inference in the last three modules.
Part 3 (Modules 10-12): Inference Through Simulation. We learn about sampling variability, hypothesis testing via randomization, and bootstrap confidence intervals. The goal is to give you a solid, intuitive grasp of statistical inference before you meet the formulas in AREC 262.

Each module ends with a test bank sample (the kind of question you will see on the actual test) and a practice exercises section.

Let’s get started.