What do ‘real’, ‘user’ and ‘sys’ mean in the output of time(1)?
What do real , user and sys mean in the output of time? Which one is meaningful when benchmarking my app?
If your program exits that fast, none of them are meaningful, it’s all just startup overhead. If you want to measure the whole program with time , have it do something that will take at least a second.
It is really important to note that time is a bash keyword. So typing man time is not giving you a man page for the bash time , rather it is giving the man page for /usr/bin/time . This has tripped me up.
8 Answers 8
Real, User and Sys process time statistics
One of these things is not like the other. Real refers to actual elapsed time; User and Sys refer to CPU time used only by the process.
- Real is wall clock time — time from start to finish of the call. This is all elapsed time including time slices used by other processes and time the process spends blocked (for example if it is waiting for I/O to complete).
- User is the amount of CPU time spent in user-mode code (outside the kernel) within the process. This is only actual CPU time used in executing the process. Other processes and time the process spends blocked do not count towards this figure.
- Sys is the amount of CPU time spent in the kernel within the process. This means executing CPU time spent in system calls within the kernel, as opposed to library code, which is still running in user-space. Like ‘user’, this is only CPU time used by the process. See below for a brief description of kernel mode (also known as ‘supervisor’ mode) and the system call mechanism.
User+Sys will tell you how much actual CPU time your process used. Note that this is across all CPUs, so if the process has multiple threads (and this process is running on a computer with more than one processor) it could potentially exceed the wall clock time reported by Real (which usually occurs). Note that in the output these figures include the User and Sys time of all child processes (and their descendants) as well when they could have been collected, e.g. by wait(2) or waitpid(2) , although the underlying system calls return the statistics for the process and its children separately.
Origins of the statistics reported by time (1)
The statistics reported by time are gathered from various system calls. ‘User’ and ‘Sys’ come from wait (2) (POSIX) or times (2) (POSIX), depending on the particular system. ‘Real’ is calculated from a start and end time gathered from the gettimeofday (2) call. Depending on the version of the system, various other statistics such as the number of context switches may also be gathered by time .
On a multi-processor machine, a multi-threaded process or a process forking children could have an elapsed time smaller than the total CPU time — as different threads or processes may run in parallel. Also, the time statistics reported come from different origins, so times recorded for very short running tasks may be subject to rounding errors, as the example given by the original poster shows.
A brief primer on Kernel vs. User mode
On Unix, or any protected-memory operating system, ‘Kernel’ or ‘Supervisor’ mode refers to a privileged mode that the CPU can operate in. Certain privileged actions that could affect security or stability can only be done when the CPU is operating in this mode; these actions are not available to application code. An example of such an action might be manipulation of the MMU to gain access to the address space of another process. Normally, user-mode code cannot do this (with good reason), although it can request shared memory from the kernel, which could be read or written by more than one process. In this case, the shared memory is explicitly requested from the kernel through a secure mechanism and both processes have to explicitly attach to it in order to use it.
The privileged mode is usually referred to as ‘kernel’ mode because the kernel is executed by the CPU running in this mode. In order to switch to kernel mode you have to issue a specific instruction (often called a trap) that switches the CPU to running in kernel mode and runs code from a specific location held in a jump table. For security reasons, you cannot switch to kernel mode and execute arbitrary code — the traps are managed through a table of addresses that cannot be written to unless the CPU is running in supervisor mode. You trap with an explicit trap number and the address is looked up in the jump table; the kernel has a finite number of controlled entry points.
The ‘system’ calls in the C library (particularly those described in Section 2 of the man pages) have a user-mode component, which is what you actually call from your C program. Behind the scenes, they may issue one or more system calls to the kernel to do specific services such as I/O, but they still also have code running in user-mode. It is also quite possible to directly issue a trap to kernel mode from any user space code if desired, although you may need to write a snippet of assembly language to set up the registers correctly for the call.
More about ‘sys’
There are things that your code cannot do from user mode — things like allocating memory or accessing hardware (HDD, network, etc.). These are under the supervision of the kernel, and it alone can do them. Some operations like malloc or fread / fwrite will invoke these kernel functions and that then will count as ‘sys’ time. Unfortunately it’s not as simple as «every call to malloc will be counted in ‘sys’ time». The call to malloc will do some processing of its own (still counted in ‘user’ time) and then somewhere along the way it may call the function in kernel (counted in ‘sys’ time). After returning from the kernel call, there will be some more time in ‘user’ and then malloc will return to your code. As for when the switch happens, and how much of it is spent in kernel mode. you cannot say. It depends on the implementation of the library. Also, other seemingly innocent functions might also use malloc and the like in the background, which will again have some time in ‘sys’ then.
What specifically are wall-clock-time, user-cpu-time, and system-cpu-time in Unix?
I can take a guess based on the names, but what specifically are wall-clock-time, user-cpu-time, and system-cpu-time in Unix? Is user-cpu time the amount of time spent executing user-code while kernel-cpu time the amount of time spent in the kernel due to the need of privileged operations (like I/O to disk)? What unit of time is this measurement in? And is wall-clock time really the number of seconds the process has spent on the CPU or is the name just misleading?
4 Answers 4
Wall-clock time is the time that a clock on the wall (or a stopwatch in hand) would measure as having elapsed between the start of the process and ‘now’.
The user-cpu time and system-cpu time are pretty much as you said — the amount of time spent in user code and the amount of time spent in kernel code.
The units are seconds (and subseconds, which might be microseconds or nanoseconds).
The wall-clock time is not the number of seconds that the process has spent on the CPU; it is the elapsed time, including time spent waiting for its turn on the CPU (while other processes get to run).
@Pacerier: on a single core machine, yes, but multi-core machines and multi-threaded programs can use more than 1 CPU second per elapsed second.
@JonathanLeffler thank you for the answer, I wanted to get the number of nanoseconds that has been elapsed but calculalting the CPU time using the formula CPUtime = #clock_cycles / clock_rate cannot be the same as calculating the elapsed time. Do you know if I can get the elapsed time from the CPU time?
@Bionix1441: You cannot derive elapsed time from CPU time for a number of reasons. First, a process can be idle, not consuming any CPU time, for arbitrary periods (for example, a daemon process waiting for a client to connect to it over the network), so it may do nothing for days at a time of elapsed time. Second, if it is running, it may have multiple threads, and if it has, say, 4 threads and there are 4 or more cores on the system, it might rack up 4 CPU seconds of expended effort per second of elapsed time. These show that there’s no simple (or even complex) formula that you could use.
@Catbuilts: Are you aware that the Unix kernel runs separately from user programs. When your program makes a system call (for example, read() or getpid() ), the kernel executes code on behalf of your program. The kernel also handles pre-emptive multi-tasking so that other programs get their turn to run, and does some general housekeeping work to keep the system running smoothly. This code is executed in ‘kernel code’ (also in ‘kernel mode’). This is distinct from the code you wrote, and the user libraries (including the system C library) that you run.
User CPU time vs System CPU time?
Could you explain more about «user CPU time» and «system CPU time»? I have read a lot, but I couldn’t understand it well.
4 Answers 4
The difference is whether the time is spent in user space or kernel space. User CPU time is time spent on the processor running your program’s code (or code in libraries); system CPU time is the time spent running code in the operating system kernel on behalf of your program.
@user472221 My answer is based on UNIX/Linux; the same concept should apply on Windows. I would guess that most DLLs are user-space, although I am not sure where Windows draws the line. If you really want to find out where your program is using CPU time, use a profiler.
Is «user space» and «kernel space» the same as «user/kernel mode»? If the kernel is running, but in user mode, does this account as user time or system time? Or is this of practical difference only when on a microkernel?
@JohnMudd: the programmer may have some intuition about where they expect time to be spent, and if that isn’t what’s really happening, may want to change their code accordingly. For example, if the program is expected to be waiting in a call to epoll() for I/O to react to, only doing very transient bursts of processing, but it’s unexpectedly spending a significant percentage of its time in user mode, then the programmer may want to investigate where and why. For example, if a web server was doing that, it may be someone’s managed to get a javascript coin miner to run.