make

Lecture 17

Dr. Colin Rundel

make

  • build tool for the creation of software / libraries / documents by specifying dependencies

    • Almost any process that has files as input and outputs can be automated via make

  • Originally created by Stuart Feldman in 1976 at Bell Labs

  • Almost universally available (all flavors of unix / linux / macOS / Windows)

  • Dependencies are specified using a text-based Makefile with a simple syntax

Makefile

A Makefile provides a list of target files along with their dependencies, and the steps necessary to generate each of the targets from the dependencies.

target1: depend1 depend2 depend3 ...
    step1
    step2
    step3
    ...

depend1: depend4 depend5
    step1
    step2
    ...

In the above example target* and depend* are all just files (given by a relative or absolute path).

Makefile (basic example)

paper.html: paper.qmd fig1/fig.png fig2/fig.png
    quarto render paper.qmd

fig1/fig.png: fig1/fig.R
    Rscript fig1/fig.R

fig2/fig.png: fig2/fig.R
    Rscript fig2/fig.R

Smart Building

Because the Makefile specifies the dependency structure make knows when a file has changed (by examining the file’s modification timestamp) and only runs the steps that depend on the file(s) that have changed.


  • After running make the first time, I edit paper.qmd, what steps run if I run make again?

  • What about editing fig1/fig.R?

Variables

Like R or other languages we can define variables, this can help avoid repetition.

R_OPTS=--no-save --no-restore --no-site-file --no-init-file --no-environ

fig1/fig.png: fig1/fig.R
    cd fig1;Rscript $(R_OPTS) fig.R

fig2/fig.png: fig2/fig.R
    cd fig2;Rscript $(R_OPTS) fig.R

Special Targets

By default if you run make without arguments it will attempt to build the first target in the Makefile (whose name does not start with a .). By convention we often include an all target which explicitly specifies how to build everything within the project.

all is an example of what is called a phony target - because there is no file named all in the directory. Other common phony targets:

  • clean - remove any files created by the Makefile, restores to the original state

  • install - for software packages, installs the compiled programs / libraries / header files

Optionally, we specify all phony targets by including a line with .PHONY as the target and the phony targets as dependencies, i.e.:

.PHONY: all clean install

This can help avoid rare cases where a file with the same name as a phony target exists in the directory, which can cause make to skip the steps for that target.

Builtin / Automatic Variables

  • $@ - the file name of the target

  • $< - the name of the first dependency

  • $^ - the names of all dependencies

  • $(@D) - the directory part of the target

  • $(@F) - the file part of the target

  • $(<D) - the directory part of the first dependency

  • $(<F) - the file part of the first dependency

Pattern Rules

Often we want to build several files in the same way, in these cases we can use % as a special wildcard character to match both targets and dependencies.

So we can go from

fig1/fig.png: fig1/fig.R
    cd fig1;Rscript fig.R

fig2/fig.png: fig2/fig.R
    cd fig2;Rscript fig.R

to

fig%/fig.png: fig%/fig.R
    cd $(<D);Rscript $(<F)

Makefile (fancier example)

all: paper.html

paper.html: paper.qmd fig1/fig.png fig2/fig.png
    quarto render paper.qmd

fig%/fig.png: fig%/fig.R
    cd $(<D);Rscript $(<F)

clean:
    rm -f paper.html
    rm -f fig*/*.png

.PHONY: all clean

HW4 Makefile

flowchart RL
    classDef script fill:#E69F00,stroke:#D55E00,color:#000
    classDef data   fill:#56B4E9,stroke:#0072B2,color:#000
    classDef output fill:#009E73,stroke:#006B4F,color:#fff
    html[hw4.html]:::output


flowchart RL
    classDef script fill:#E69F00,stroke:#D55E00,color:#000
    classDef data   fill:#56B4E9,stroke:#0072B2,color:#000
    classDef output fill:#009E73,stroke:#006B4F,color:#fff
    qmd([hw4.qmd]):::script
    html[hw4.html]:::output
    qmd      --> html


flowchart RL
    classDef script fill:#E69F00,stroke:#D55E00,color:#000
    classDef data   fill:#56B4E9,stroke:#0072B2,color:#000
    classDef output fill:#009E73,stroke:#006B4F,color:#fff
    qmd([hw4.qmd]):::script
    lqrds[data/lq.rds]:::data
    dennyrds[data/dennys.rds]:::data
    html[hw4.html]:::output
    lqrds    --> qmd
    dennyrds --> qmd
    qmd      --> html


flowchart RL
    classDef script fill:#E69F00,stroke:#D55E00,color:#000
    classDef data   fill:#56B4E9,stroke:#0072B2,color:#000
    classDef output fill:#009E73,stroke:#006B4F,color:#fff
    parselq([parse_lq.R]):::script
    lqrds[data/lq.rds]:::data
    parsedennys([parse_dennys.R]):::script
    dennyrds[data/dennys.rds]:::data
    qmd([hw4.qmd]):::script
    html[hw4.html]:::output
    parselq      --> lqrds
    lqrds        --> qmd
    parsedennys  --> dennyrds
    dennyrds     --> qmd
    qmd          --> html


flowchart RL
    classDef script fill:#E69F00,stroke:#D55E00,color:#000
    classDef data   fill:#56B4E9,stroke:#0072B2,color:#000
    classDef output fill:#009E73,stroke:#006B4F,color:#fff
    parselq([parse_lq.R]):::script
    lqhtml[data/lq/*.html]:::data
    lqrds[data/lq.rds]:::data
    parsedennys([parse_dennys.R]):::script
    dennyshtml[data/dennys/*.html]:::data
    dennyrds[data/dennys.rds]:::data
    qmd([hw4.qmd]):::script
    html[hw4.html]:::output
    lqhtml       --> parselq
    parselq      --> lqrds
    lqrds        --> qmd
    dennyshtml   --> parsedennys
    parsedennys  --> dennyrds
    dennyrds     --> qmd
    qmd          --> html


flowchart RL
    classDef script fill:#E69F00,stroke:#D55E00,color:#000
    classDef data   fill:#56B4E9,stroke:#0072B2,color:#000
    classDef output fill:#009E73,stroke:#006B4F,color:#fff
    getlq([get_lq.R]):::script
    parselq([parse_lq.R]):::script
    lqhtml[data/lq/*.html]:::data
    lqrds[data/lq.rds]:::data
    getdennys([get_dennys.R]):::script
    parsedennys([parse_dennys.R]):::script
    dennyshtml[data/dennys/*.html]:::data
    dennyrds[data/dennys.rds]:::data
    qmd([hw4.qmd]):::script
    html[hw4.html]:::output
    getlq        --> lqhtml
    lqhtml       --> parselq
    parselq      --> lqrds
    lqrds        --> qmd
    getdennys    --> dennyshtml
    dennyshtml   --> parsedennys
    parsedennys  --> dennyrds
    dennyrds     --> qmd
    qmd          --> html

Live Demo

HW4 Makefile

all: hw4.html

hw4.html: hw4.qmd data/lq.rds data/dennys.rds
    quarto render hw4.qmd

data/lq.rds: parse_lq.R data/lq/*.html
    Rscript parse_lq.R

data/dennys.rds: parse_dennys.R data/dennys/*.html
    Rscript parse_dennys.R

data/lq/*.html: get_lq.R
    Rscript get_lq.R

data/dennys/*.html: get_dennys.R
    Rscript get_dennys.R

clean:
    rm -f hw4.html
    rm -rf data/

.PHONY: all clean