Design visibility, builds and versions together
TL;DR — Three tools in a language depend on the public interface of a package: the visibility rules, the build system and the package manager. When a language designs them separately, they disagree, and programmers pay for it. In C++, a private field makes other files rebuild. In Rust, splitting a slow crate makes private items public. Swift added a new keyword nine years after 1.0 to match its package manager. The idea of this post is to define the interface once and let all three tools read it. In Vx, one visibility keyword will answer three questions: who can call this, what rebuilds when it changes, and which version bump the change needs.
This post is about one idea in programming language design, with small examples from languages you may already use.
Three tools, three questions
Each of the three tools asks one question:
| Tool | Its question |
|---|---|
| Visibility rules, in the compiler | Can this code call that function? |
| Build system | A file changed. What must I rebuild? |
| Package manager | Can version 1.4 replace version 1.3? |
Each answer depends on the same thing: the public interface of the package, meaning the part that other code can see and use. If the language defines that interface exactly, the build system and the package manager can derive their answers from it. If the language leaves gaps, each tool guesses, and the guesses disagree. Here are three such disagreements.
C++: a private field is public to the build
// widget.h
class Widget {
public:
void draw();
private:
int cache_size_; // only Widget's own code can use this
};
Only the code of Widget can use cache_size_. But the field is in the
header, so every file that includes widget.h depends on it. Add a second private
field, and the build system rebuilds all of those files. The size of Widget also
changes, so code that someone compiled against the old header can break.
The language hides the field from other code. It cannot hide the field from the build system, or from the layout of the compiled program. C++ programmers hide such fields by hand with the pimpl idiom: the header keeps only a pointer to a separate struct.
Rust: build speed makes items public
In Rust, pub(crate) means "visible anywhere in this crate". A crate is also the unit
that the compiler compiles at one time, and a large crate compiles slowly. So teams split a large
crate into several smaller ones. Rust has no visibility level for "the crates in my workspace", so
each pub(crate) item that a new crate uses must become pub:
// Before the split: one large crate.
pub(crate) fn parse_header(bytes: &[u8]) -> Header { ... }
// After the split, the new crate `codec` calls it, so it must be `pub`.
pub fn parse_header(bytes: &[u8]) -> Header { ... }
When the team publishes the library, crates.io also needs the new crate, with a version number
of its own. A decision about build speed made the public interface larger. Now the team must keep
parse_header compatible in each release, though they never meant users to call it.
Swift: a keyword to match the package manager
In Swift, internal means "this module", and one Swift package can hold many
modules. So code that two modules of one package shared had to be public, and every
user of the package could call it. In 2023, nine years after Swift 1.0, Swift 5.9 added a
package level
(SE-0386).
The language changed its visibility rules to fit its package manager.
Late fixes are hard
Each of these fixes came late, and a late fix must keep all existing code working. Java shows
how long that takes. It added modules in 2017, 21 years after
Java 1.0. By then, so many libraries used the internal class sun.misc.Unsafe that
Java 9 had to leave it available. A new language has one chance to draw its boundaries cheaply:
before it has users.
The idea: define the interface once
In each example, one tool drew a boundary that the other two could not see. The fix is to give the three tools one definition of the interface. The language defines the interface exactly, and the build system and the package manager derive their answers from it. This needs two things from the language.
First, the signature must hold the whole interface. In Vx, a signature is the whole contract. Each parameter type and each return type is in the signature, and nothing about the type of a function comes from its body.
Rust shows what can go wrong without this rule. The future that an async fn returns
can be sent to another thread only if it is Send. Rust decides that from the values
that the body holds across each .await, and the signature does not say it. Add one line that holds an Rc
across an .await, and each caller that sends the future to another thread stops
compiling. The signature did not change, so a tool that compares signatures cannot see the break.
The Rust RFC for impl Trait calls this "a silent breaking change"
(RFC 1522).
Second, the visibility levels must match the units that the other two tools use. Those units are the package, and the group of packages that a team releases together. Then one keyword can answer all three questions.
Elm already does part of this for versions. elm publish compares the interfaces of
the old and the new release, and it accepts only a version number that matches the change. Cargo
has no such check. Someone asked for one in 2014, and the
request is still open.
One keyword, three answers
Vx plans four visibility levels. pub and the private default exist today.
pkg and ws are a design:
fn helper() // private: this module only (the default)
pkg fn plan() // any module in this package
ws fn lower() // any package in this workspace
pub fn compile() // any package that depends on this one
A package is the unit that the compiler checks and the registry publishes. A workspace is a group of packages that the registry publishes together, at one version. Each level then gives all three answers at once:
| Level | Who can call it | A change to its signature rebuilds | Version bump |
|---|---|---|---|
| private | Code in this module | Nothing outside this package | Patch |
pkg | Code in this package | Nothing outside this package | Patch |
ws | Packages in this workspace | The packages in this workspace | None: the workspace shares one version |
pub | Any package that depends on it | Each package that depends on it | Major, if the change breaks callers |
Because the signature holds the whole interface, a change to the body of a function rebuilds nothing outside its package, at any level. Generic and inline functions still need a rebuild, because other packages generate code from their bodies.
Why must a workspace share one version? Suppose that package A calls a ws function
in package B, and B 1.3 changes that function. A ws function has no version promise of
its own. So with separate versions, nothing would stop a user from getting A 1.2 with B 1.3. The
build would then fail, and no version number would warn about it. With one shared version, that
pair cannot exist.
This is the central idea of the post. The programmer writes one keyword. The compiler, the build system and the package manager all read their answer from it, so they cannot disagree.
What Vx has today
Part of this exists. The rule that a signature is the whole contract is in the language, because
the parallel compiler of Vx needs it. vxc --emit-interface writes the interface of a
package to a .vxlib file. vx-api diff compares two of these files and
prints the smallest safe version bump. Here is the first version of a small library:
fn scale(x: i32) -> i32 {
return x * 2;
}
pub fn area(w: i32, h: i32) -> i32 {
return scale(w) * h;
}
The second version changed area to take and return i64.
vx-api diff compares the interface files of the two versions:
$ vx-api diff v1/shapes.vxlib v2/shapes.vxlib
major function `area` changed from `fn(i32, i32) -> i32` to `fn(i64, i64) -> i64`
not compared: traits, enums whose variants carry data, constants, and the bounds of generic parameters
bump: major
A change to only the body of scale gives bump: patch. But Vx also shows the
drift that this post describes. pkg and ws do not exist yet, so
vx-api treats every item as public. A change to the signature of the private helper
scale also asks for a major release. The .vxlib file also holds function
bodies today, so its bytes change after any edit. A build system that compares these files cannot
skip a rebuild yet.
Visibility, builds and versions all depend on one fact: what a package shows to the code outside it. A new language can define that fact once, before it has users, and let all three tools read it.
The design is in docs/discussions/visibility-and-semver-package-management.md. The scorecard compares 15 pairs of compiler and build system, and Vx, against six stricter tests. Vx is Apache 2.0 with the LLVM exception — install it and try the examples.