Design visibility, builds and versions together

TL;DR — Three tools in a language depend on the public interface of a package: the visibility rules, the build system and the package manager. When a language designs them separately, they disagree, and programmers pay for it. In C++, a private field makes other files rebuild. In Rust, splitting a slow crate makes private items public. Swift added a new keyword nine years after 1.0 to match its package manager. The idea of this post is to define the interface once and let all three tools read it. In Vx, one visibility keyword will answer three questions: who can call this, what rebuilds when it changes, and which version bump the change needs.

This post is about one idea in programming language design, with small examples from languages you may already use.

Three tools, three questions

Each of the three tools asks one question:

ToolIts question
Visibility rules, in the compilerCan this code call that function?
Build systemA file changed. What must I rebuild?
Package managerCan version 1.4 replace version 1.3?

Each answer depends on the same thing: the public interface of the package, meaning the part that other code can see and use. If the language defines that interface exactly, the build system and the package manager can derive their answers from it. If the language leaves gaps, each tool guesses, and the guesses disagree. Here are three such disagreements.

C++: a private field is public to the build

// widget.h
class Widget {
public:
  void draw();
private:
  int cache_size_;   // only Widget's own code can use this
};

Only the code of Widget can use cache_size_. But the field is in the header, so every file that includes widget.h depends on it. Add a second private field, and the build system rebuilds all of those files. The size of Widget also changes, so code that someone compiled against the old header can break.

The language hides the field from other code. It cannot hide the field from the build system, or from the layout of the compiled program. C++ programmers hide such fields by hand with the pimpl idiom: the header keeps only a pointer to a separate struct.

Rust: build speed makes items public

In Rust, pub(crate) means "visible anywhere in this crate". A crate is also the unit that the compiler compiles at one time, and a large crate compiles slowly. So teams split a large crate into several smaller ones. Rust has no visibility level for "the crates in my workspace", so each pub(crate) item that a new crate uses must become pub:

// Before the split: one large crate.
pub(crate) fn parse_header(bytes: &[u8]) -> Header { ... }

// After the split, the new crate `codec` calls it, so it must be `pub`.
pub fn parse_header(bytes: &[u8]) -> Header { ... }

When the team publishes the library, crates.io also needs the new crate, with a version number of its own. A decision about build speed made the public interface larger. Now the team must keep parse_header compatible in each release, though they never meant users to call it.

Swift: a keyword to match the package manager

In Swift, internal means "this module", and one Swift package can hold many modules. So code that two modules of one package shared had to be public, and every user of the package could call it. In 2023, nine years after Swift 1.0, Swift 5.9 added a package level (SE-0386). The language changed its visibility rules to fit its package manager.

Late fixes are hard

Each of these fixes came late, and a late fix must keep all existing code working. Java shows how long that takes. It added modules in 2017, 21 years after Java 1.0. By then, so many libraries used the internal class sun.misc.Unsafe that Java 9 had to leave it available. A new language has one chance to draw its boundaries cheaply: before it has users.

The idea: define the interface once

In each example, one tool drew a boundary that the other two could not see. The fix is to give the three tools one definition of the interface. The language defines the interface exactly, and the build system and the package manager derive their answers from it. This needs two things from the language.

First, the signature must hold the whole interface. In Vx, a signature is the whole contract. Each parameter type and each return type is in the signature, and nothing about the type of a function comes from its body.

Rust shows what can go wrong without this rule. The future that an async fn returns can be sent to another thread only if it is Send. Rust decides that from the values that the body holds across each .await, and the signature does not say it. Add one line that holds an Rc across an .await, and each caller that sends the future to another thread stops compiling. The signature did not change, so a tool that compares signatures cannot see the break. The Rust RFC for impl Trait calls this "a silent breaking change" (RFC 1522).

Second, the visibility levels must match the units that the other two tools use. Those units are the package, and the group of packages that a team releases together. Then one keyword can answer all three questions.

Elm already does part of this for versions. elm publish compares the interfaces of the old and the new release, and it accepts only a version number that matches the change. Cargo has no such check. Someone asked for one in 2014, and the request is still open.

One keyword, three answers

Vx plans four visibility levels. pub and the private default exist today. pkg and ws are a design:

fn helper()        // private: this module only (the default)
pkg fn plan()      // any module in this package
ws fn lower()      // any package in this workspace
pub fn compile()   // any package that depends on this one

A package is the unit that the compiler checks and the registry publishes. A workspace is a group of packages that the registry publishes together, at one version. Each level then gives all three answers at once:

LevelWho can call itA change to its signature rebuildsVersion bump
privateCode in this moduleNothing outside this packagePatch
pkgCode in this packageNothing outside this packagePatch
wsPackages in this workspaceThe packages in this workspaceNone: the workspace shares one version
pubAny package that depends on itEach package that depends on itMajor, if the change breaks callers

Because the signature holds the whole interface, a change to the body of a function rebuilds nothing outside its package, at any level. Generic and inline functions still need a rebuild, because other packages generate code from their bodies.

Why must a workspace share one version? Suppose that package A calls a ws function in package B, and B 1.3 changes that function. A ws function has no version promise of its own. So with separate versions, nothing would stop a user from getting A 1.2 with B 1.3. The build would then fail, and no version number would warn about it. With one shared version, that pair cannot exist.

This is the central idea of the post. The programmer writes one keyword. The compiler, the build system and the package manager all read their answer from it, so they cannot disagree.

What Vx has today

Part of this exists. The rule that a signature is the whole contract is in the language, because the parallel compiler of Vx needs it. vxc --emit-interface writes the interface of a package to a .vxlib file. vx-api diff compares two of these files and prints the smallest safe version bump. Here is the first version of a small library:

fn scale(x: i32) -> i32 {
  return x * 2;
}

pub fn area(w: i32, h: i32) -> i32 {
  return scale(w) * h;
}

The second version changed area to take and return i64. vx-api diff compares the interface files of the two versions:

$ vx-api diff v1/shapes.vxlib v2/shapes.vxlib
major  function `area` changed from `fn(i32, i32) -> i32` to `fn(i64, i64) -> i64`
not compared: traits, enums whose variants carry data, constants, and the bounds of generic parameters
bump: major

A change to only the body of scale gives bump: patch. But Vx also shows the drift that this post describes. pkg and ws do not exist yet, so vx-api treats every item as public. A change to the signature of the private helper scale also asks for a major release. The .vxlib file also holds function bodies today, so its bytes change after any edit. A build system that compares these files cannot skip a rebuild yet.

Visibility, builds and versions all depend on one fact: what a package shows to the code outside it. A new language can define that fact once, before it has users, and let all three tools read it.

The design is in docs/discussions/visibility-and-semver-package-management.md. The scorecard compares 15 pairs of compiler and build system, and Vx, against six stricter tests. Vx is Apache 2.0 with the LLVM exception — install it and try the examples.