Breaking the borrow checker on purpose
TL;DR — In five days, Vx's borrow checker had twelve fixes for programs it accepted and should have refused, and five for correct programs it refused. Then we tested the tests. We broke the checker on purpose, 108 ways, one at a time, and ran the tests after each break. They caught 93. Of the 15 they missed, 11 were rules no test checked, and those tests are now added. Three changed code that no Vx program reaches today. One led to a real bug: a borrow made inside a block was forgotten when the block ended, and a program read freed memory. Then we translated 78 of Rust's borrow checker bug reports into Vx. They led to 14 more programs that Vx accepts and should refuse, now filed, and to two correct programs that Vx accepts and Rust still refuses.
The previous post went through the programs Vx's borrow checker let through, and how each was fixed. Every fix came with a test, and every test passed. That shows a test and its fix agree. It does not show what the tests miss. This post is about how we went looking for that, and what the whole effort added up to.
Five days of fixes
The work started on 10 October with a review of the checker, tracked in #1501. A borrow checker can be wrong in two directions. It can accept a program that reaches memory it should not, which is the dangerous direction: the optimizer trusts the checker, and nothing later catches the mistake. Or it can refuse a correct program, which costs the person writing it.
| What was wrong | Fixed | Open |
|---|---|---|
| A program compiled that should have been refused | 12 | 15: #1778, found by breaking the checker, whose fix is in review, and #1791 to #1804, found from Rust's bug reports |
| A correct program was refused | 5 | 1, #1784 |
The twelve are in the previous post's table, with one more since: a second &mut
x was accepted while a reference to a reference still reached the first
(#1768). The five correct programs that
were refused include Euclid's algorithm, which broke gcd in a bigint library written in
Vx (#1646).
Some changes added new rules:
Tensor<..>::uninit()gives memory whose elements hold whatever was there before. Reading one before writing it reads data the program never wrote. A call outsideunsafeis now an error (#1509). Before the error went in, every call in the repository moved intounsafe, and the olderTensor<T>([..])spelling was changed to start at zero.- A
spawnloop that could run in parallel, and runs on one thread because something blocks it, now gets a warning that names what blocks it (#1515). - A field moved out of a struct is now tracked on its own. It can be moved out only once, and the rest of the struct is still dropped (#1512).
And a set of tests tries each of seven borrow rules inside each kind of block Vx has, 266
programs in all (#1577). The bug in the
previous post whose answer changed with the optimization level was a rule that worked in a function
body and failed inside an if arm, so the tests try every rule in every place.
Do the tests check the checker?
Here is a question those tests cannot answer by passing: if one part of the checker stopped working, would any test notice? The direct way to find out is to stop it on purpose and run the tests. This is called mutation testing. Each change to the program under test is a mutant, and a mutant the tests do not catch marks code that no test depends on. Tools such as cargo-mutants do this automatically for Rust code, by replacing function bodies and changing operators. The repository's rules already ask for this one test at a time: before a new test is trusted, break the code it covers and watch it fail. Here the same idea is applied to the whole checker.
We wrote the breaks by hand, so each one meant something for a borrow checker. There were two kinds:
- A report turned off. Each of the 42 places where the checker reports a borrow error, E4001 to E4013, was skipped in turn.
- A decision turned off or weakened. These are the questions the reports
depend on: when a borrow ends, what
let r2 = r1copies, whether two places such asq.aandq[i]can overlap, where a reference points, what a move does, and whether one lifetime fits another. A first round turned each one off. A second round weakened them in narrower ways. For example, a borrow ended one statement early, a copy kept only one of two borrows, the moves made in anelsearm were forgotten, or anif's value was taken to point only where its first branch points.
Building the compiler takes minutes. So we did not rebuild it for each break. We put every break behind a switch that an environment variable turns on, built once, and ran the tests once per switch:
if !sabotaged("A29") {
self.errors.error_with_code(DiagnosticCode::E4009, message, span);
}
The tests for each switch were the compiler's 595 unit tests and its test programs, almost 1,500 of them, about a third of which must be refused. A run took about two minutes, and all 108 took about three and a half hours on one machine. The switches were a local change only, and never went into the repository.
What got through
The first round broke the checker 79 ways, and the tests caught 78. The second round broke it 29 ways, and the tests caught 15. For each of the 15 that got through, we wrote a short program that the break should change, and compiled it twice: once with the real checker, once with the break on.
- If the real checker refused it and the broken one accepted it, the checker was right and no test checked that rule. That was the case for 11.
- If the program could not be written at all, the code the break changed is never reached. That was the case for 3.
- If both accepted it, the real checker was wrong too. That was the case for 1.
Eleven rules no test checked
Here is one. A value moved in the else arm of an if may have been moved
when the code after the if runs, so using it there is refused:
fn moved_in_else(c : bool) -> i32 {
let s = S { v : 1 };
if c {
print!(1);
} else {
take(s);
}
return take(s); // E4001: s may have been moved
}
The checker refuses this. But the tests only ever moved a value in the first branch, so forgetting the moves of the second branch broke nothing they looked at. The other ten were of the same kind:
| The rule | Error |
|---|---|
A store through a reference that a call returns, such as *slot(h) = &local, can put a local's address into the caller's memory | E4005 |
let t2 = t gives t2 every borrow t holds, even two of the same variable, such as &q.a and &q.b | E4009 |
A value moved in a later arm of a match is moved after it | E4001 |
A value that points at a local cannot be returned through the else of an if, a later match arm, an unsafe block, a *p, an enum payload, or a closure call such as (|| &local)() (six rules, one per form) | E4005 |
A method that returns a &mut keeps its receiver borrowed as &mut while the result is used | E4004 |
Each now has a test (#1783). Each test was checked the same way: with its break on, the error it expects goes away, and the other tests' errors stay.
Three breaks that changed nothing
One break made reading an element, a[i], forget where the element points. No Vx
program can show this today: arrays and tensors hold only numbers, so no element holds a reference.
Two others changed how a method call is checked. Before those questions are asked, the checker
has already turned a.f(b) into a function call f(a, b), so that code never
runs. These will matter once arrays can hold references, or once a method call reaches those
checks unchanged.
One real bug
One break made a copied borrow keep the block it was made in, instead of the block of the variable it was copied to. To show what that changes, the program needs a borrow made inside a block and given to a variable declared outside it. Written that way, the real checker accepted the program too:
import std::vec;
struct W {
r : &i32,
}
fn main() -> i32 {
let mut v = Vec<i32>::new();
v.push(7);
let y : i32 = 2;
let mut outer = W { r : &y };
if true {
outer = W { r : v.as_slice() }; // outer now borrows v
}
for i in 0..1000 {
v.push(i); // v grows, and moves its elements
}
let mut z = Vec<i32>::new();
for i in 0..8 {
z.push(9999);
}
print!(*outer.r); // reads the old, freed buffer
print!("\n");
return 0;
}
It compiled and printed a different large number on each run, such as
-151189382. The right answer is 7, or better, a compile error at the first
push, since outer still borrows v. Rust refuses the same
program. With no if around the assignment, Vx refused it too.
The checker lost the borrow in two places (#1778):
- An
if,match,unsafeorwhilewritten as a statement is checked as an expression. After an expression statement, the checker puts back the table of borrows it had before the statement. That is how the borrows a call makes end with the call. It also dropped the borrow thatouternow held. - At the end of a block, the checker dropped every borrow made in that block, whoever held it.
That covered the
forloop, whose body is a block.
The struct is not needed. With outer a plain reference,
outer = v.as_slice() inside the if read freed memory as well.
outer = &v was accepted too, and so was the push after it, when the
type of outer was written out, as in let mut outer : &Vec<i32> =
&w;. When that type is left to be inferred from &w, a different check
(E3004) refuses outer = &v first, which is why a first look suggested the hole
needed a struct. The fix (#1786) keeps a
borrow whose holder is still in scope after the block or statement, for as long as the holder
lives. Built from that change, each of these forms is refused at the first push.
The tests then caught the fix's first version. A program in the test suite does this:
if disagg == 1 {
logits = transformer<1>(token, pos, &config, &mut decode_state, &decode_weights);
} else {
logits = transformer<0>(token, pos, &config, &mut decode_state, &decode_weights);
}
With the borrow from the first branch kept in logits, the second branch's
&mut decode_state was refused as a second mutable borrow. But only one branch runs.
The checker already handled moves this way: each branch starts from the moves made before the
if, and the branches are joined after it. Borrows now get the same treatment.
What breaking the checker cannot find
A break shows whether some test depends on a rule. A rule that was never written cannot be
broken. So this method could not have found two of the bugs in the previous post. A reference into
a value with no name, kept after the value was freed
(#1752), needed a check that did not exist
yet. So did a struct holding both a &mut to an outer variable and a reference to an
inner one (#1746). We found the first by
reading an open item of an older lifetime issue
(#860), and the second from a bug report
against Rust.
Rust's issue tracker has years of such reports, many with a short program attached. Many of them describe rules Vx has, written in a language Vx follows closely. Translating the ones Vx can express into Vx gives tests that nobody here had to think of, so we did that next.
Rust's bug reports, run in Vx
We took 78 reports from Rust's issue tracker, each with a short program, from the labels Rust
uses for soundness bugs and for the borrow checker (I-unsound, A-borrow-checker, A-NLL, NLL-sound
and NLL-complete). Each program was translated into Vx and run with main and the fix
for #1778:
- 38 gave the right answer: Vx refused what should be refused, and accepted the rest.
- 15 could not be written in Vx. Most need written lifetimes. Some need a
matchguard, or aletwith no value. - 13 were correct programs that Vx refused.
- 12 led to a program that Vx accepted and should have refused: for six, the report's own program, and for six, a close variant of it or the closest program Vx can write.
Following those twelve, and three more programs written along the way, gave 14 holes, filed as #1791 to #1804. All but one compile and then read a value that has changed, drop a value twice, or read freed memory.
Some go past the Rust bug they came from. The previous post showed
rust-lang/rust#164163, where a
borrow is made inside a method call's argument and the argument also points p
somewhere else. In Vx the second step is not needed
(#1800):
import std::vec;
fn main() -> i32 {
let mut a = Vec<i64>::with_capacity(4);
for i in 0..4 {
a.push(1i64);
}
let zero : i64 = 0i64;
let mut s : &i64 = &zero;
// The push grows `a` and frees its old buffer, which `s` points into.
a.push(if true {
s = a.get_ref(0);
0i64
} else {
0i64
});
let mut c = Vec<i64>::with_capacity(4);
c.push(99i64);
println!("{}", *s);
return 0;
}
The argument of a.push(..) keeps a reference to a[0] in
s, and the push then moves the buffer. The program compiles and prints 99, a value
from the other vector, which took over the freed memory. Rust refuses this form with E0502.
Two go the other way.
rust-lang/rust#54663 and
rust-lang/rust#145032 are correct
programs that rustc 1.95 still refuses, and their Vx versions compile and print the right answer.
In #54663, a function returns early from inside a match on a call that borrows its
argument, and then borrows the argument again. Vx ends the call's borrow with the
match, because only the arm that returns holds it. Keeping the call's result and using
it after a write to the argument is still refused, as it should be.
What is left
- The 14 holes found from Rust's bug reports, #1791 to #1804, and the 13 correct programs from those reports that Vx refuses.
- #1784:
r = &ydoes not endr's earlier borrow ofx, so a later change toxis refused whileris used. Rust accepts this program. - Vx gives a reference the block in which it was made, not the block of what it points at. So
outer = inneris refused when the two references were made in different blocks, even when both point at variables that outlive them. Rust accepts this program too. - The ideas from the post about the Zeta language that started this work are still design
questions: building a value one field at a time
(#1684), proving that
a[i]anda[j]are different elements (#1513), and letting a signature say which parts of an argument a function uses (#1514).
If you find a program that compiles and should not, or one that should compile and does not, please open an issue with the program.
Vx is Apache 2.0 with the LLVM exception — install it and try the examples.