Running Linux Programs as Unikernels on macOS
Someone emailed me the other day and was having some trouble getting
their program to work under OPS. He had downloaded a binary release from github and knowing that OPS only runs linux binaries chose the linux version. Further, there are numerous
examples of people running linux programs under ops on osx. Hell, just do a
user@users-MacBook-Pro Desktop % ops run solana-validator *errors.errorString libstdc++.so.6: file does not exist /Users/eyberg/go/src/github.com/nanovms/ops/lepton/ldd_darwin.go:100 (0x4c55779) /Users/eyberg/go/src/github.com/nanovms/ops/lepton/ldd_darwin.go:129 (0x4c5607d) /Users/eyberg/go/src/github.com/nanovms/ops/lepton/image.go:261 (0x4c52d0e) /Users/eyberg/go/src/github.com/nanovms/ops/lepton/image.go:24 (0x4c50f5f) /Users/eyberg/go/src/github.com/nanovms/ops/cmd/run.go:15 (0x4ca4189) /Users/eyberg/go/src/github.com/nanovms/ops/cmd/run.go:118 (0x4ca4f75) /Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 (0x4c8bf3e) /Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 (0x4c8cb5b) /Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 (0x4ca8d88) /Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 (0x4ca8d83) /usr/local/go/src/runtime/proc.go:200 (0x402f57c) main: return /usr/local/go/src/runtime/asm_amd64.s:1337 (0x405a671) goexit: GLOBL shifts>(SB),RODATA,$256 panic: libstdc++.so.6: file does not exist After I saw the word mac and then libstdc++ I knew immediately what was wrong but I also realized that it might not be readily apparent to many people and so this is what this blogpost is about. The binary in question is dynamically linked. What this means is that
there are several libraries that the dynamic loader (ld) will try and
load at runtime versus statically linked where all the libraries are
packaged inside the binary. This leads to a fatter binary but you can be
assured that everything is present when needed. I won’t jump into a
static/dynamic flamewar as they each have their own usecases. I should also point out that normally you can’t run linux binaries directly on a mac — this is actually a uniquely interesting unikernel aspect as we don’t need to boot up vagrant or a linux vm. Traditional virtualization software virtualizes the operating system. You can see using a tool like OPS as virtualizing the application instead. One of the reasons I decided to write this post was not just the email but I’ve noticed in the past few years I’ve seen quite a lot of people that use containers and go to refer to their programs as statically linked when it is in fact absolutely not. Let’s look at this example go webserver:
package main import ( "fmt" "net/http" ) func main() http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) fmt.Fprintf(w, "Welcome to my website!") >) fs := http.FileServer(http.Dir("static/")) http.Handle("/static/", http.StripPrefix("/static/", fs)) http.ListenAndServe(":8080", nil) > eyberg@box:~/z$ ldd z linux-vdso.so.1 (0x00007ffe8b1db000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2e091f4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2e08e03000) /lib64/ld-linux-x86-64.so.2 (0x00007f2e09413000) eyberg@box:~/z$ file z z: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, not stripped VDSO gives us a fast clock. Libpthread gives us threads. Libc contains everything you’d find in section 3 of the man pages and finally LD is our loader. You can link your go programs statically via something like this:
eyberg@box:~/z$ go build -buildmode=pie -ldflags "-linkmode external -extldflags -static" # _/home/eyberg/z /tmp/go-link-057347370/000004.o: In function `_cgo_7e1b3c2abc8d_C2func_getaddrinfo': /tmp/go-build/cgo-gcc-prolog:57: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking eyberg@box:~/z$ ldd z not a dynamic executable eyberg@box:~/z$ file z z: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=0671d22e52baf812ed9f500a8d9ee7848b8c809e, not stripped Both ldd && file now report that it is statically linked. However, you’ll notice (at least in go 1.12) it now outputs a message stating that getaddrinfo is still required! That’s cause even though ldd output and file is telling us that it is indeed static we still rely on libnss for dns. You can force go to use its built in resolver instead. Getaddrinfo is found here:
eyberg@box:~/s/solana-release/bin$ readelf --dyn-syms /lib/x86_64-linux-gnu/libc-2.27.so | grep getaddr 873: 0000000000107bc0 3261 FUNC GLOBAL DEFAULT 13 getaddrinfo@@GLIBC_2.2.5 1601: 00000000001413c0 43 FUNC GLOBAL DEFAULT 13 inet6_rth_getaddr@@GLIBC_2.5 You’ll note that your libraries are probably stripped which means you can’t use something like nm to find the symbols but readelf surfaces them up in the .dynsym section. Also, even that is not enough — you’ll probably make use of various functions found in these files as well:
eyberg@box:~/z$ ls /lib/x86_64-linux-gnu/libnss_files.so.2 /lib/x86_64-linux-gnu/libnss_files.so.2 eyberg@box:~/z$ ls /lib/x86_64-linux-gnu/libnss_dns.so.2 /lib/x86_64-linux-gnu/libnss_dns.so.2 The point here is that just sticking a go program in a container does not inherently make your binary ‘statically linked’ and the problem is that, that is not what a lot of people have been stating and this source of
confusion is probably what leads to misunderstandings such as this. So let’s go back to our example at the beginning of the article. The program in question is the solana blockchain. You can download and unzip like so:
wget https://github.com/solana-labs/solana/releases/download/v0.23.2/solana-release-x86_64-unknown-linux-gnu.tar.bz2 bunzip2 solana-release-x86_64-unknown-linux-gnu.tar.bz2 tar xf solana-release-x86_64-unknown-linux-gnu.tar You can see from the ldd output that it is trying to load a library that is dynamically linked (libstdc++) and that’s what our error in the trace was.
eyberg@box:~/s/solana-release/bin$ ldd solana linux-vdso.so.1 (0x00007ffe338df000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f06976a8000) /lib64/ld-linux-x86-64.so.2 (0x00007f06984a2000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f06974a4000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f069729c000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f069707d000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0696e65000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0696ac7000) eyberg@box:~/s/solana-release/bin$ ops run -c config.json solana-validator [solana-validator --ledger /] booting /home/eyberg/.ops/images/solana-validator.img . qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] assigned: 10.0.2.15 solana-validator 0.23.2 log file: solana-validator-7wgZy3ozTRyvLtRN5o3Xh6m5aEvEo2XBxP5yPfaCny1t-20200210-204507.log eyberg@box:~/s/solana-release/bin$ cat config.json < "Args": ["solana-validator", "--ledger", "/"] >This is all fine but if you want to run it on osx you’ll need to do one
of two things: 1) Either build from source and statically link it (so ldd doesn’t show
output). or 2) Download the libraries it’s linked to and manually add them to the
filesystem. The reason this works out of the box on linux is that OPS looks at and
try to load the libraries as we build the disk image. This isn’t
possible on a mac cause mac uses mach-o which is a different format that Nanos does not and probably won’t ever support as nanos is explicitly designed to run linux server-side programs. For example when we build common packages like nginx or node, you’ll see from the ops output that all required libraries are placed on the filesystem in a known location.
➜ ~ ops pkg contents node_v13.6.0 File :/node File :/package.manifest Dir :/sysroot Dir :/sysroot/lib Dir :/sysroot/lib/x86_64-linux-gnu File :/sysroot/lib/x86_64-linux-gnu/libc.so.6 File :/sysroot/lib/x86_64-linux-gnu/libdl.so.2 File :/sysroot/lib/x86_64-linux-gnu/libgcc_s.so.1 File :/sysroot/lib/x86_64-linux-gnu/libm.so.6 File :/sysroot/lib/x86_64-linux-gnu/libnss_dns.so.2 File :/sysroot/lib/x86_64-linux-gnu/libnss_files.so.2 File :/sysroot/lib/x86_64-linux-gnu/libpthread.so.0 Dir :/sysroot/lib64 File :/sysroot/lib64/ld-linux-x86-64.so.2 Dir :/sysroot/proc File :/sysroot/proc/meminfo Dir :/sysroot/usr Dir :/sysroot/usr/lib Dir :/sysroot/usr/lib/x86_64-linux-gnu File :/sysroot/usr/lib/x86_64-linux-gnu/libstdc++.so.6 Without getting into the semantics of building your own package which you can find here the easiest way to do this with your own app is in your project directory you can create a directory:
ops run -c config.json myprogram After that you can always grab the image that was created in ~/.ops/images. Hope this clears up any confusion.
Is there a way to run a Linux binary on macOS?
Is there a way to run a Linux binary in macOS? I tried to run a binary but it said it isn’t executable.
Well OS X does not use elf binaries, so it’s a bit like trying to run windows .exe. However, a brief search finds this: osxbook.com/software/xbinary If you can recompile your code, it’s not an issue.
XBinary requires a program that knows how to run your binary; it in and of itself doesn’t run anything.
@ott— You can use «wine» to run Windows binary on Linux. Maybe there is a wine-like thing to run Linux binary on macOS.
4 Answers 4
Update years later: The Noah repo has now been archived. I haven’t found a good alternative.
I recently starting using Noah to run Linux binaries in macOS. You can install using homebrew ( brew install linux-noah/noah/noah ). Then you should be able to do this:
In my experience the behavior of the binary matches what I see on my Ubuntu machine.
Outdated answer, it does not work anymore. noah has been removed from homebrew and the source code is archived.
These answers are half correct, because virtualization is a choice but there is another. May I present.
History
- First there was UNIX, circa 1972
- Then the Timeline Split
- In 1977, for $90, Bob Fabry and others, compiled/built the first versions of BSD, short for Berkeley Systems Distribution.
- In 1991, Linus Torvalds posted in a Newsgroup, about software he used from Richard Stallman, who started GNU in 1983, and Linus’es UNIX was born.
- Apple reacquired NeXT Software in 1996, after Steve Jobs was fired from Apple in 1984, and used the software and people there to build OS X. OS X is the Darwin OS + the NeXT Desktop Environment. And now we’re back to Bullet #1, as Darwin is a closed source fork of BSD. For the Open Source Project, see PureDarwin. For the Official Apple Developer Page, see Apple Open Source.
How We Can Use This
BSD’s traditionally use the Ports system for Package Management. The most widely used of these are the FreeBSD Ports. Ports are packages installed directly from source. Since the same Linux applications come from the same sources, you can run a Linux application if its port exists. Don’t use these Ports on a Mac because.
Since all of Apple’s GUI’s are written using the Cocoa API — WikiEntry, bundled with XCode — OS X for Developers, the Ports can be tuned to take advantage of this:
- Install XCode for your version of OS X.
- Bundled inside XCode is Apple’s version of the GCC Compiler, and all the other associated tools. To update the tools, see this post on StackOverflow. The tools are OS Version dependant, ie they are not backwards compatible, to my knowledge (in short, don’t install XCode for 10.8 on 10.6, etc.)
- Having installed XCode you now have a compiler, and can head on over to the MacPorts page and browse for the port you need installed, after installing the MacPorts .pkg installer
Caveat
The ports system doesn’t necessarily do dependency checking, unless the port was well written. I lightly touched on the problem in what does elibc_FreeBSD mean in gentoo portage overlays?