How to determine the maximum number to pass to make -j option?
I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the -j option. How can I programmatically choose that value, e.g. in a shell script? Is the output of nproc equivalent to the number of threads I have available to compile with? make -j1 make -j16
8 Answers 8
nproc gives the number of CPU cores/threads available, e.g. 8 on a quad-core CPU supporting two-way SMT.
The number of jobs you can run in parallel with make using the -j option depends on a number of factors:
- the amount of available memory
- the amount of memory used by each make job
- the extent to which make jobs are I/O- or CPU-bound
make -j$(nproc) is a decent place to start, but you can usually use higher values, as long as you don’t exhaust your available memory and start thrashing.
For really fast builds, if you have enough memory, I recommend using a tmpfs , that way most jobs will be CPU-bound and make -j$(nproc) will work as fast as possible.
If I use a tmpfs , will I be limited to a directory size that always smaller than my physical RAM size?
It’s not a great answer, but in the strict spirit of the question of programmatically determining the fastest «j» value, you could loop j from 1 to some reasonable upper limit (2x nproc??) and wrap the make in a time call. Clean up the results, lather rinse repeat— and end up sorting the times/j values.
@terdon No. Make is all about resolving dependencies, which means the jobs still have to be run in a certain order. GNU parallel doesn’t care about that. On a side note, deciding which jobs are safe to run in parallel and which aren’t is a hard problem. All make programs that offered parallel builds took years until they became somewhat usable.
The most straight-foward way is to use nproc like so:
The command nproc will return the number of cores on your machine. By wrapping it in the ticks, the nproc command will execute first, return a number and that number will be passed into make .
You may have some anecdotal experience where doing core-count + 1 results in faster compile times. This has more to do with factors like I/O delays, other resource delays and other availability of resource constraints.
To do this with nproc+1 , try this:
Unfortunately even different portions of the same build may be optimal with conflicting j factor values, depending on what’s being built, how, which of the system resources are the bottleneck at that time, what else is happening on the build machine, what’s going on in the network (if using distributed build techniques), status/location/performance of the many caching systems involved in a build, etc.
Compiling 100 tiny C files may be faster than compiling a single huge one, or viceversa. Building small highly convoluted code can be slower than building huge amounts of straight-forward/linear code.
Even the context of the build matters — using a j factor optimized for builds on dedicated servers fine tuned for exclusive, non-overlapping builds may yield very dissapointing results when used by developers building in parallel on the same shared server (each such build may take more time than all of them combined if serialized) or on servers with different hardware configurations or virtualized.
There’s also the aspect of correctness of the build specification. Very complex builds may have race conditions causing intermittent build failures with occurence rates that can vary wildly with the increase or decrease of the j factor.
I can go on and on. The point is that you have to actually evaluate your build in your very context for which you want the j factor optimized. @Jeff Schaller’s comment applies: iterate until you find your best fit. Personally I’d start from the nproc value, try upwards first and downwards only if the upwards attempts show immediate degradation.
Might be a good idea to first measure several identical builds in supposedly identical contexts just to get an idea of the variability of your measurements — if too high it could jeopardise your entire optimisation effort (a 20% variability would completely eclipse a 10% improvement/degradation reading in the j factor search).
Lastly, IMHO it’s better to use an (adaptive) jobserver if supported and available instead of a fixed j factor — it consistently provides a better build performance across wider ranges of contexts.
Parallel make: set -j8 as the default option
I can set number of threads for the build process using -j argument. For example, I have 4 cores +4 virtual. When I write: make -j8 the speed increases 4 times. Is it possible to set that value as default? (For example, in Linux Gentoo, in config file, it’s possible to set this default value). p.s. I have Arch Linux
Pubby: Most makefile authors have numerous flaws and don’t get the dependencies right, ultimately leading to bad experiences when trying parallel builds 😉
But often, just rerunning make once more makes everything right while still having built most of the stuff much faster.
@K3—rnc I have a question that illustrates what you have said: Error building wxWidgets using -j (jobs) option with MinGW
3 Answers 3
Your question is not about threads, but processes (jobs) executed by make.
The simple, way to set this, when make is used from the console is adding:
You can also use setenv MAKEFLAGS ‘-j 8’ , but MAKEFLAGS can ignore this parameter in some scenarios, because keeping desired number of processes requires communicating with recursive make calls. Happily this method works with current versions of GNU Make.
Since this mentioning of «recursive make» doesn’t stop, I’d like to once again point to aegis.sourceforge.net/auug97.pdf . besides, the make file needs to be well-written to even allow for parallel make. Often, especially automated methods, disregard the actual dependencies and cause targets to be built multiple times and the likes .
In case the link is not working in future, the paper is titled «Recursive Make Considered Harmful» and should be easy enough to find. It is well worth a read.
Most people should NOT pass the -j option value down to recursive calls. Jobserver in the top-level make does this much better: all the child processes combined get exactly as many jobs as the cores assigned. Now imagine if you set -j16 for 16 cores, and EACH of 10 child make s gets -j16 (which overrides jobserver). Now you have 160 jobs doing context switches on a CPU-bound tasks.
Note that setenv belongs to (t)csh and is not available in bash (see unix.stackexchange.com/a/85100/200009).
CORES ?= $(shell sysctl -n hw.ncpu || echo 1) all:; @$(MAKE) _all -j$(CORES) _all: install lint test .PHONY: all _all … I’ve basically «aliased» my default target all to a «private» _all . The command to figure out the number of cores is OSX specific, AFAIK, so you could just improve it to be more cross platform if you will. And because of the ?= assignment, we can just override it with and env variable if/when needed.
You can also append to your MAKEFLAGS from within the makefile itself, like so:
CPUS ?= $(shell sysctl -n hw.ncpu || echo 1) MAKEFLAGS += --jobs=$(CPUS) … You may also use the following, ff you want it to be more cross-platform:
CPUS ?= $(shell (nproc --all || sysctl -n hw.ncpu) 2>/dev/null || echo 1)