With a dataset in R and inspected, we can start doing things to it. Most data analysis involves repeating a surprisingly small number of operations: keep certain rows, keep certain columns, create or change a column, sort the rows, calculate summary statistics, and do those calculations separately for different groups.
The tidyverse package dplyr gives us a function for each of these jobs (you will often see them called the dplyr “verbs”). None of the operations are new – each is the R form of something from the Excel modules:
Question
dplyr function
Excel counterpart
Which rows do I want?
filter()
filtering
Which columns do I want?
select()
keeping certain columns
What new column do I want?
mutate()
a formula in a new column
In what order do I want the rows?
arrange()
sorting
What summary do I want?
summarise()
summary formulas
That summary separately for groups?
group_by()
a PivotTable
Once you understand these operations, most everyday data manipulation in R is a matter of combining them.
We will continue with the canola trial data from the last chapter, in the same project folder. Since this chapter does a new job, it gets a new script: code/2_data_manipulation.R. Start it with a header, then load the tidyverse and read the data in:
# ---# Title: Manipulating the canola trial data# Author: Your Name# Date: 2026-10-06# Description:# Filters, transforms, and summarizes the canola trial data# using the dplyr functions.# ---# Load packageslibrary(tidyverse)# Read in the datatrial_data <-read_csv("data/canola_trial.csv")
> # ---
> # Title: Manipulating the canola trial data
> # Author: Your Name
> # Date: 2026-10-06
> # Description:
> # Filters, transforms, and summarizes the canola trial data
> # using the dplyr functions.
> # ---
>
> # Load packages
> library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
>
> # Read in the data
> trial_data <- read_csv("data/canola_trial.csv")
Rows: 120 Columns: 5
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): field_id, variety
dbl (3): fertilizer_kg_ha, rainfall_mm, yield_bu_acre
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
6.1 Data Manipulation Functions
NoteVideo walkthrough of this section
We will learn four functions in this section. Two affect the rows:
filter() chooses rows, and
arrange() changes their order.
Two affect the columns:
select() chooses columns, and
mutate() creates or changes them.
Each function takes the data frame as its first argument, followed by what to do with it.
6.1.1filter(): Keep Certain Rows
filter() keeps the rows that satisfy a condition. It’s two arguments are the data frame and the condition. For example, to keep only the fields with yields above 50 bushels per acre:
Notice that filter() displays the high-yielding fields but does not remove the other fields from trial_data – check nrow(trial_data) and all 120 rows are still there. Every function in this chapter works the same way: it takes a data frame as input and returns a new data frame as output. Running the above code will only display that output, R does not save it. To keep a result, assign it to an object:
(An assignment prints nothing – the result went into high_yield instead of the console.)
Now there are two objects: trial_data, the original 120 fields, and high_yield, the filtered subset. That is a useful feature – you can experiment without changing the original data. Throughout the chapter, an example without <- is just displaying a result.
Note the two equals signs: == asks whether two things are equal, so variety == "InVigor" asks “is variety equal to "InVigor"?”. Also note the quotation marks: "InVigor" with quotes is text; InVigor without quotes would be read as the name of an object, which probably doesn’t exist and would cause an error.
You can combine conditions with & (and – both must be true), | (or – at least one must be true), and ! (not):
##Filter for yields above 50 with the variety InVigorfilter(trial_data, yield_bu_acre >50& variety =="InVigor")##Filter for fertilizer below 80 or above 160filter(trial_data, fertilizer_kg_ha <80| fertilizer_kg_ha >160)##Filter for yields above 60 with the variety not equal to InVigorfilter(trial_data, yield_bu_acre >60&!variety =="InVigor")
When you want to match several possible values of the same variable, %in% is easier to read:
##The following two filters are identicalfilter(trial_data, variety =="InVigor"| variety =="DEKALB")filter(trial_data, variety %in%c("InVigor", "DEKALB"))
The function arrange() changes the order of the rows. It also takes a minimum of two arguments: the data frame and the variable to sort by. The code arrange(trial_data, yield_bu_acre) sorts from lowest yield to highest. If you want to arrange from highest to lowest then use the desc() (descending order) function.
##Sort data by yield (lowest to highest)arrange(trial_data, yield_bu_acre)##Sort data by yield (highest to lowest)arrange(trial_data, desc(yield_bu_acre))
You can sort by more than one variable – arrange(trial_data, variety, desc(yield_bu_acre)) sorts by variety and then, within each variety, from highest yield to lowest. Like the other functions, arrange() does not permanently rearrange trial_data unless you save the result.
## Sort data by variety first, then yield (highest to lowest)arrange(trial_data, variety, desc(yield_bu_acre))
select() chooses variables rather than observations. It’s arguments are the data frame and the names of the variables to keep. For example, to keep only the field ID, variety, and yield:
The result has the same rows but only those three columns – especially useful with datasets containing dozens or hundreds of variables. You can also drop a column by putting - in front of its name, or keep a range of adjacent columns:
## Select everything except rainfallselect(trial_data, -rainfall_mm) # Select columns from field_id through rainfall_mmselect(trial_data, field_id:rainfall_mm)
A relative of select() is rename(), which changes a column’s name without dropping anything: rename(trial_data, yield = yield_bu_acre).
6.1.4mutate(): Create or Change Columns
Potentially the most useful function is mutate(), which creates a new variable or changes an existing one. Suppose we want yield in tonnes per hectare rather than bushels per acre. We can accomplish this by running the code:
## Create a new variable which is yield in tons per ha using the conversion factor of 0.056. Save as trial datatrial_data <-mutate(trial_data, yield_t_ha = yield_bu_acre *0.0560)## Print trial data to see our new columnhead(trial_data)
> ## Create a new variable which is yield in tons per ha using the conversion factor of 0.056. Save as trial data
> trial_data <- mutate(trial_data, yield_t_ha = yield_bu_acre * 0.0560)
>
> ## Print trial data to see our new column
> head(trial_data)
# A tibble: 6 × 6
field_id fertilizer_kg_ha rainfall_mm variety yield_bu_acre yield_t_ha
<chr> <dbl> <dbl> <chr> <dbl> <dbl>
1 C001 146. 199. InVigor 56.7 3.18
2 C002 136. 282. DEKALB 62 3.47
3 C003 113. 274. Clearfield 53.6 3.00
4 C004 112. 173 InVigor 51.2 2.87
5 C005 103. 216. InVigor 49.2 2.76
6 C006 140. 151. InVigor 51.2 2.87
Note that we have saved the resulting data frame back into trial_data. In R we can always overwrite an object. The result contains all the original columns plus a new yield_t_ha. Read the expression as: create yield_t_ha and set it equal to yield_bu_acre × 0.0560.
You can create several variables in one call:
## Create new rainfall and fertilizer variablestrial_data <-mutate(trial_data,rainfall_cm = rainfall_mm /10,kg_n_per_t = fertilizer_kg_ha / yield_t_ha)## Print trial data to see our new columnsprint(trial_data)
In the second variable the hectares cancel – fertilizer and yield are both per hectare – leaving kilograms of nitrogen per tonne of grain.
6.2 The Pipe |>
NoteVideo walkthrough of this section
Many questions that we want to resolve require more than one function. Suppose we want to know: among the InVigor fields, what are the field IDs and yields, ordered from highest to lowest? We know every operation we need – filter to InVigor, select the two columns, arrange by descending yield. We can combine these using the pipe, which is written as |>. The pipe takes whatever is on its left and feeds it as the first argument to the function on its right. Essentially, think of the pipe as a command that says “then”. For example, suppose we want to
filter the data to InVigor, THEN
select the columns field_id and yield_bu_acre, THEN
arrange by yield (highest to lowest).
We can write this in R code as:
trial_data |>##Take our trial data THEN filter(variety =="InVigor") |>##Filter it to InVigor THENselect(field_id, yield_bu_acre) |>##Select field_id and yield_bu_acre columns THENarrange(desc(yield_bu_acre)) ##Sort by yield (highest to lowest)
> trial_data |> ##Take our trial data THEN
+ filter(variety == "InVigor") |> ##Filter it to InVigor THEN
+ select(field_id, yield_bu_acre) |> ##Select field_id and yield_bu_acre columns THEN
+ arrange(desc(yield_bu_acre)) ##Sort by yield (highest to lowest)
# A tibble: 42 × 2
field_id yield_bu_acre
<chr> <dbl>
1 C112 70.4
2 C045 62.8
3 C033 62.5
4 C009 61
5 C120 60.1
6 C119 60
7 C011 59.5
8 C052 59.5
9 C076 59.2
10 C059 58.4
# ℹ 32 more rows
One thing to note is that we have split this code across multiple lines. We could have written it all one one line as:
This becomes difficult to read. R knows that when a line ends with the pipe |>, the command is not finished, so it continues reading the next line. This lets us write a long sequence of operations across several lines rather than cramming everything onto one line.
This is not something special about the pipe. More generally, R will keep reading whenever it can tell that a command is incomplete. For example, we can also split a function across multiple lines:
##These two commands are identicalfilter(trial_data, yield_bu_acre >50)filter(trial_data, yield_bu_acre >50)
NoteVideo walkthrough of this section and Writing Output
The four functions so far rearrange or add information while keeping observations as rows. Sometimes we want to collapse the observations into summary statistics. That is the job of summarise().
6.3.1summarise(): Many Rows Become One
Suppose we want to get the mean of yields in the data. We can use the summarise function and inside define a column that is the mean of yields. The result is a new data frame with a single row and a single column, mean_yield:
The names on the left (mean_yield, sd_yield, n_fields) become column names in the result; the expressions on the right calculate the values. n() counts the rows. Note again that we have split the arguments of summarise across multiple lines. We could also write:
The contrast with mutate() matters. mutate() keeps the observations and adds information; summarise() collapses the observations into summaries. So trial_data |> mutate(mean_yield = mean(yield_bu_acre)) still has 120 rows, with the overall mean repeated on every row, while the summarise() version returns one row.
Note that we could have used potentially simpler commands from “base R” to get the mean and standard deviation above:
##Get mean and st dev of yields using base Rmean(trial_data$yield_bu_acre)sd(trial_data$yield_bu_acre)
> ##Get mean and st dev of yields using base R
>
> mean(trial_data$yield_bu_acre)
[1] 53.39917
> sd(trial_data$yield_bu_acre)
[1] 5.767724
The power of summarise comes from being able to combine it with other commands via the pipe. For example, we can combine it with filter() to get the mean of only the InVigor fields:
Take trial_data, then divide the observations by variety, then calculate the statistics separately for each variety. The result has one row per variety. This is the R equivalent of a PivotTable.
The central pattern – data |> group_by(group_variable) |> summarise(...) – appears throughout the rest of the book.
6.4 Writing Output
Results so far have existed only inside R. To save one as a file that can be opened in Excel or sent to someone else, pass the object and a filename to write_csv. With our project layout, results go in output/:
## Summarise mean yields by varietyvariety_summary <- trial_data |>group_by(variety) |>summarise(mean_yield =mean(yield_bu_acre, na.rm =TRUE),n_fields =n())## Write the summary table to the output folderwrite_csv(variety_summary, "output/variety_summary.csv")
> ## Summarise mean yields by variety
> variety_summary <- trial_data |>
+ group_by(variety) |>
+ summarise(mean_yield = mean(yield_bu_acre, na.rm = TRUE),
+ n_fields = n())
>
> ## Write the summary table to the output folder
> write_csv(variety_summary, "output/variety_summary.csv")
Notice that write_csv prints nothing to the console – the result is the file itself, which appears in output/. read_csv brings data into R; write_csv sends a data frame out to a file. Because the output is created by the script, you can always recreate it by running the script again.
6.5 Combining It All
We now have enough tools to answer a realistic question: for each variety, among fields that received at least 100 kg/ha of nitrogen, what was the mean yield in tonnes per hectare, and how many fields were observed? Show the highest-yielding variety first.
Do not read it as one giant command; read it one step at a time:
Filter to the fields that received at least 100 kg/ha.
Create the variable the question needs: metric yield.
Group the observations by variety.
Summarise each group down to one row: its mean yield and its number of fields.
Sort from highest mean yield to lowest.
Each line does one simple thing, and that is the reason for pipes: a good pipeline makes the logic of an analysis visible. Saving the table is one more line: write_csv(variety_summary_100, "output/variety_summary_100.csv").
6.6 Debugging
Your code will break. That is normal – everyone’s does – and finding the problem is a skill of its own. It has two parts: reading what R tells you, and narrowing down where things went wrong.
Reading error messages
R’s error messages name the problem more precisely than they first appear to. Here are the three you will meet most often, each with its usual cause.
A misspelled function name (or a package you forgot to load):
quantle(trial_data$yield_bu_acre, 0.5)
> quantle(trial_data$yield_bu_acre, 0.5)
Error: could not find function "quantle"
R is exact about names – quantle is not quantile, and no amount of context will make R guess. The same message appears when the function is spelled correctly but its package is not loaded; then the fix is library(tidyverse).
A misspelled object name, or an object that was never created:
mean(trial_dta$yield_bu_acre)
> mean(trial_dta$yield_bu_acre)
Error: object 'trial_dta' not found
Check the Variables pane: is the object there, spelled exactly that way? If it is missing entirely, the line that creates it has not been run – which often means the script was not run from the top.
Using = where == belongs:
filter(trial_data, variety ="InVigor")
> filter(trial_data, variety = "InVigor")
Error: We detected a named input.
ℹ This usually means that you've used `=` instead of `==`.
ℹ Did you mean `variety == "InVigor"`?
This one diagnoses itself – the message even suggests the fix.
The wrong answer with no error
The harder case is code that runs and returns something wrong. R is exact about the values in the data too:
No error – just zero rows, because no variety is spelled "invigor" with a small i. A comparison matches text exactly, capitals included.
For a pipeline that produces a wrong-looking result, do not stare at the whole thing and guess. Run it one step at a time: run the first step alone, check the result, add the second step, check again, and keep going. The first point where the data stops looking the way you expect is where the problem is.