Statically Linking Go in 2022

tl;dr

I.e.

• To get a static binary, under most circumstances: CGO_ENABLED=0 go build
• To use the libc functions for net and os/user, and still get a static binary (for containers): go build -ldflags "-linkmode 'external' -extldflags '-static'"
  • On dev machines, don’t try to static link: go build

Introduction

In this blog I assume you know roughly what’s meant by source files, compilation (i.e. translation of compilation units), object files, archive files, linking, shared objects, and dynamic linking.

Statically linking a Go programme is one of those things I often want to do but rarely think about. Whenever I set up a new project I just copy&paste the build instructions from a previous project. My recent experiments to Chainguard’s Melange and Apko have forced me to think about this again, and I’ve realised a lot of the options I used to use are either out-of-date or redundant. Googling this stuff finds a few articles, but they’re all at least two years old, and none explains what’s actually going on.

Why might Go not be statically linked?

All the Golang code in your project does get statically linked; any Go packages you import, including things from the standard library like fmt, are built into the binary. Any Go programme that uses just these will be a static binary; there’s nothing else for it to link to. For example, Hello World in Go is statically-linked.

package main

import (
    "fmt"
)

func main() {
    fmt.Println("Hello, world!")
}

$ ldd main
      not a dynamic executable

Go wants to statically link. Static binaries were one of the big selling points of Go in the early days. So what gives? Go has support for calling C code, a mechanism called CGO, which you can use to call arbitrary code in arbitrary libraries. If you start calling out to C code, Go will, by default, produce a dynamically-linked binary, linked to those libraries, just like a C compiler would. But that’s rare right? It only ever happens if you explicitly ask for it right? It turns out: no. Go’s standard library uses CGO itself in a couple of places. Specifically, common functions in the packages net and os/user call out to libc. This is what makes so many Go programmes dynamically-linked, to the point that a lot of people think Go dynamically links by default.

They do this to make use of advanced libc functionality like nsswitch (and thus LDAP etc) for name resolution, and advanced user/group ID sources (again mostly LDAP these days). Re-implementing all this in Go’s stdlib would be a huge undertaking and hasn’t been done yet.

Because of these calls into libc, your binary ends up dynamically linked against it. And because the binary is dynamically linked, it also links against the dynamic loader, and VDSO (on a typical, modern, linux system - this stuff is full of exceptions and edge cases).

package main

import (
    "fmt"
    "net"
)

func main() {
    fmt.Println(net.LookupHost("google.com"))
}

$ ldd main
      linux-vdso.so.1 (0x00007ffee81ea000)
      libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3695075000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f36952a7000)

Why would you want a statically linked programme?

Just to recap: a Go programme that doesn’t use net or os/user (or deliberately call other C libraries) will be statically linked out of the box. All the options I’m going to show here aren’t necessary in this circumstance, and are often applied when they don’t need to be. But, if you are ending up with a dynamic binary, you might want a static one instead, and this is how to do that.

I won’t spend much time on why, but it’s usually for ease of packaging and deployment. If you need a binary to run on any and every system no matter its distro or age, then not depending on any system libraries can be really useful. This was a big part of Go’s early appeal because it was a pretty unique feature back then. Likewise if you’re building a binary that’s going to be packaged into a container image, then it’s a lot easier to have a static binary, not least if your base image is static/distroless/wolfi, because then you don’t have to arrange to have (the right version of) (the right type of) libc in it.

How do I make my binary static again?

Do stop and think: this isn’t something you always want. By linking against libc you get to use those sophisticated hostname lookup, user and group info functions, and more. If your programme needs to work in, say, an LDAP environment, you might need that. It also benefits from host system updates (less applicable in containers): using the system’s libc and other libraries means you automatically use any upgraded versions, too.

Caveats aside, there’s two main options:

Avoid the whole issue

The Go standard library has pure-Go versions of all the functions in net and os/user. These are less full-featured than the versions in libc, but they’re often fine. The libc calls are the default, but you can specify build tags to opt-in to the pure Go versions: netgo and osusergo.

$ go build -tags netgo,osusergo main.go
$ ldd main
      not a dynamic executable

Or you can apply a blanket ban on CGO callouts, forcing the native Go implementations everywhere, with the environment variable CGO_ENABLED.

$ CGO_ENABLED=0 go build main.go
$ ldd main
      not a dynamic executable

You don’t need both the tags and the environment variable.

This also has the advantage that it’s easy to cross-compile (eg from your macos/arm64 laptop to linux/amd64 for a container image). Linking against C libraries needs their headers and the libraries themselves available on the build machine, which can be very fiddly if you’re trying to cross-compile.

Statically link the C code in

Go likes static linking; that’s how it includes all the Go packages you use. When you start calling C libraries, the reason it dynamically links against them is because static linking is hard. The toolchain would have to understand all the intricacies of ELF, Mach-O, PIE, etc, etc. So it just… doesn’t. Instead it emits a dynamically-linked binary, and lets the dynamic loader (eg ld-linux) sort it out later.

However we can statically link all this C code into the binary if we want. The docs imply this isn’t a 100% supported path, so consider if you really need this. This can give you the best of both worlds - libc’s advanced functionality, and a run-anywhere static binary.

To do this, we tell Go’s toolchain to use an external linker rather than its own (it’ll go find one, usually GCC’s ld): go build -ldflags "-linkmode 'external'".

This is called external linking, as opposed to the default internal linking where Go’s own linker is used. Deep dive.

You might be worried about the availability of GCC’s ld on build systems, and while that’s a concern, compiling any CGO code needs GCC anyway, so it’s no worse.

We then need to tell that linker to produce a static binary, or rather we need to tell the go driver programme to tell its linker component to call out to ld and to tell that. So we end up with: go build -ldflags "-linkmode 'external' -extldflags '-static'".

You’ll see a lot of stuff on the internet saying you only need the -extldflags bit, but that’s not been true for a while: Go started using its own linker by default a few years back.

Note that you’re in fiddly, fragile territory here. These are the current incantations, for Linux. Your mileage will definitely vary (these were the runes in 2019, I’ve not even tested them. There don’t seem to be any official docs for this). There’s been talk of adding a -static flag that Just Works TM, but that issue’s been open since 2018.

For example, when I do this, I use the above command when building binaries for containers, and do that in a controlled environment (like a Dockerfile). When I build a programme like this on my dev machine for testing or whatever, I leave CGO enabled (to make it pretty representative) but I don’t try to statically link, since that’s a nightmare across OSes, distros, version, and arches. (The command for this is simply: go build main.go).

Cross-Compilation

To clarify, there’s two things commonly meant when talking about cross-compiling Go:

1. The Go toolchain’s own cross-compilation support, eg running the native Go toolchain on macos/arm64 and saying GOOS="linux" GOARCH="amd64"
2. Emulating the target platform’s Go toolchain, eg on an arm64 mac, running the amd64 go tool under qemu (to get that toolchain’s native amd64 binaries)

Cross-compilation is very useful for example in CI. You might have a project on GitHub for which you want binaries to be available for download. The CI runner (eg GitHub actions) is probably Linux/amd64, but you want to also produce Darwin/arm64, Linux/armv7 (for Raspberry Pis), etc.

Cross-compilation and CGO/static linking are not good friends. Indeed, CGO is default off when cross-compiling (opposite to normal). Case 1 basically doesn’t work, at least not easily, when you’re using CGO and/or external static linking (the two often go hand-in-hand). I won’t go into the details, but even if you can get it to work once it’s hard to maintain.

If you can, it’s easiest to just not use CGO for cross-compiles. You’ll get a build without issues, and the resulting binaries will be statically linked.

If you need CGO, your best bet is case 2; run the target’s native Go under emulation. However note that this isn’t so quick, especially in throttled compute like a free CI runner. There’s various ways to do this, I’ll list a few common cases below:

• If you’re building a container image with docker buildx or Chainguard’s melange+apko, this emulation will happen automatically. Don’t give Go any cross-compilation instructions, and it’ll just work.
• If you’re running under GitHub actions (and trying to build in the workflow itself, rather than calling docker buildx or melange), you could try https://github.com/uraimo/run-on-arch-action (NB: I’ve not tested this).

A note on OpenSSL

Anyone who’s used Python or Rust (and I’m sure many more) has hit the pain of OpenSSL. Those languages often want to link against OpenSSL (because their tls&crypto packages call it, like Go does with libc), and doing that often means they try to download and compile the OpenSSL source, which often fails. I’m no expert, but I’ve not seen Go try to do this. I wrote a minimal programme which constructed an HTTPS server, and while it linked against libc, libssl was nowhere to be seen. Google hasn’t given me anything concrete, but it seems like all crypto code is implemented in pure Go… That seems like a lot of work, more than implementing net I’d have thought, but 🤷‍♀️

Just because Go has a net/tls, crypto/x509, etc doesn’t mean they couldn’t be just thin wrappers around OpenSSL / BoringSSL / GnuTLS, but they don’t seem to be.