Linux systemd analyze blame

No, this is not a disparaging or bashing article of the new Linux boot process technology designed to replace the older, classic System V init daemon. This is a tutorial that will teach you how to use the built-in functionality of the systemd framework to profile and analyze your boot process. Remember Bootchart? Well, almost the same.

Today, we will have a brief introduction into systemd, mention some alternatives, see the basic locations and commands, and then delve deeper into the boot sequence, using the available tools to try to identify and resolve bottlenecks. And we will do some fancy graphs, so stay put and enjoy. Tested on Fedora 16 KDE.

Brief introduction to systemd

systemd is too complex to fully explain in this one page. At the same time, grasping the basic concepts is not that difficult. The overall concept of bringing your machine up and running has not changed. The big difference is that systemd tries to parallelize the startup of services, whereas this process has been serial with the init daemon. In theory, you will have your machine booting faster. At the same time, systemd remains fully backward compatible, so you can use the old commands and methods of managing your box.

systemd keeps its startup scripts under /lib/systemd . As simple as that. This is not much different from having them kept under /etc/init.d. Then, you also have /etc/systemd directory, which contains symbolic links to the /lib/systemd directory.

Under /lib/systemd, you will find a handful of sub-directories. For example, the system directory contains the startup scripts for your services.

Enabled scripts are nothing but shell scripts with the executable bit turned on. So if you want to disable a service altogether, you can just chmod it off. Again, this is no different than changing the permissions on scripts located under /etc/init.d, without touching the runlevel directories.

If you want to work the proper way, there are other ways than tampering with /lib/systemd. The control of systemd scripts is done using systemctl utility, which is a sort of a mother tool for the systemd framework. Think of it as the service wrapper, similar to chkconfig. Enabling/disabling services becomes a simple matter of creating or deleting symbolic links for relevant services.

Now, if you want to disable a service, then you merely do:

You will see the actual action on the command line:

Similarly, enabling a service:

ln -s ‘/lib/systemd/system/bluetooth.service’ ‘/etc/systemd/system/bluetooth.target.wants/bluetooth.service’

Likewise, you can stop, start, reload and whatnot. Just like in the past. Now, if you wonder what a systemd script looks like, it’s no black magic either. Let’s focus on the Bluetooth example:

So what do we have here. First, the use of the word target . Don’t worry, targets are merely the replacement term for runlevels, though extended a bit to allow additional flexibility and usage. For backward compatibility, the standard runlevels are fully mapped to runlevelX.target scripts , where X is one of the numbers. In other words, issuing init or telinit is equivalent to issuing init runlevel.

But then, there are additional targets, which you might treat as subgroups within runlevels, allowing you a lot of customization that was not previously possible. You can sort of create boot checkpoints, identified by various targets, to make sure one or more services are started before you move on, all this without altering the purpose of runlevels or ruining the backward compatibility.

All right, so Bluetooth is designed to start after the syslog target. You have the executable path, the type of service, where the log should be written to, as well as the dependencies list. This is very similar to classic init scripts, but much simpler, because you do not need to know shell, you just need to understand a few basic terms.

Now, I must confess I’m omitting quite a bit, but it’s not really important at the moment. We’re not here to digest systemd in complete detail, as we might have a separate article on this topic, we’re here to reduce anxiety and then focus on the boot process.

And here’s a nice cheatsheet on how to use systemd.

Analyze the boot process with the blame command

Yup, that’s it. What do you do when the system is misbehaving? You run systemd-analyze blame. Boom. So now the title pun finally and fully registers. Blame is one of those handy commands that you want to use when you identify a problematic boot process.

Just running systemd-analyze without any extra commands will display the last boot statistics, for example:

Startup finished in 958ms (kernel) + 1862ms (initramfs) + 127651ms (userspace) = 130472ms

Now, execute systemd-analyze blame to get a verbose output, with the services times shown in ms, from the worst offender down. This gives you a quick overview of where you might want to place most of your focus. Now, note that the total times need not be summed up! Some of these services loaded at the same time. The table view should only be a rough pointer, as investing effort in pruning down services that took only a fraction of a second may be a waste of time, where those exceeding 20-30 seconds might be worth examining.

This means you will need a graphical representation of your boot sequence to try to understand what the critical points are where only single services loaded and delayed the subsequent steps.

Enter systemd-analyze plot + eog

All right. Now we want to dump our boot info into a plot data. systemd-analyze will do that for us, as well. Then, we want to take the dumped information and display it as a graph. Here is how it’s done, more info available yonder.

systemd-analyze plot > /tmp/plot.svg
eog /tmp/plot.svg

eog is a tool that will plot SVG maps in a nice, lovely fashion, much like you’ve seen with Bootchart. The tool is available in Fedora repositories, however it is a GTK application, so it might look a little out of place on top of a KDE desktop.

The difficult part is now trying to understand what is taking so long and how you can resolve the issue, if at all. You will have to carefully walk through the graph and figure out which services might be slow or plain unneeded.

systemctl

In theory, we’re done with blaming the boot process. But let’s explore systemd and the accompanying tools and utilities some more. Profiling your system startup can be fun, especially if you take full advantage of all the built-in systemd features.

dot option

This sounds like something graphical, and this is in fact an option to create a dependency list for all services, output in XML form. As to the usage itself, if you issue the command without any arguments, you will get a help hint at the end:

— You probably want to process this output with graphviz’ dot tool.
— Try a shell pipeline like ‘systemctl dot | dot -Tsvg > systemd.svg’!

Interesting. All right. We do want to see the dependency graph of our boot process. First, we need to install dot, which brings along some 20+ LaTeX dependencies. Now, follow the command as written above and wait until the XML data is turned into a graph:

Conclusion

There ought to be two separate conclusions for this tutorial. One, all about systemd as the new boot framework for handling services. And I will not be doing that, because I have not properly explored and reviewed this technology. Two, the boot-process related debug capabilities, which are aplenty. Using blame or dot, you gain a wealth of knowledge about your hanging or stalling services, allowing you to trim, slim and weightloss your boot sequence, improve overall performance and gain a better understanding of your system. Best of all, the analytic functionality is included in this new framework. On top of that, we produced some fancy graphs that should help us understand and resolve problems.

I hope you liked this article. It should reduce your anxiety about systemd, as well as teaches you how to use it to control and profile your boot times. So if you suspect there might be a problem somewhere, grab the SVG files and start charting. Have fun.

Источник

Linux systemd analyze blame

Systemd-analyze — анализ системных процессов