Java: Spring Boot application as a service using systemd on Ubuntu 16.04
Although in modern architectures you typically see Spring Boot executable jars running as the primary process of a container, there are still many deployment scenarios where running the jar as a service at boot time is required.
With Ubuntu 16.04, we can use the built-in systemd supervisor to run a Spring Boot application at boot time. This will enable the Java based service to run in the background as a distinct user, fully integrated into the syslog framework.
Build the echo service
The first step is to build the echo service, which is a Spring Boot based application that receives an HTTP request and outputs the context path and request parameters as an HTTP response.
$ sudo apt-get install openjdk-8-jdk git curl -y $ sudo ufw allow 8080; sudo ufw allow 80 $ cd ~ $ git clone https://github.com/fabianlee/spring-echo-example.git $ cd spring-echo-example $ ./gradlew clean assemble $ java -jar build/libs/spring-echo-example-1.0.0.jar --server.port=8080
When run, it starts by outputting a Spring banner to the console, and then you will see messages from the embedded Jetty container, and finally the message “Alive and kicking. ” when the Spring application is started.
2018-04-17 15:30:00.422 INFO 2539 --- [ main] com.waiamu.open.SpringEchoApp : Alive and kicking.
From a different console, use curl to invoke the echo service and you will see output like below.
$ curl http://localhost:8080/echo?message=hello < "headers" : < "Accept" : "*/*", "User-Agent" : "curl/7.47.0", "Host" : "localhost" >, "path" : "/echo", "protocol" : "HTTP/1.1", "method" : "GET", "body" : null, "parameters" : < "message" : [ "hello" ] >, "cookies" : null
This is basic validation of the service when run as a normal foreground process.
Creating systemd service
Turning this into a service for systemd requires that we create a unit service file “springechoservice.service” in the “/lib/systemd/system” directory.
$ sudo cp src/main/resources/scripts/springechoservice.service /lib/systemd/system/. $ sudo sed -i -e "s#/home/vagrant/spring-echo-example#`pwd`#g" /lib/systemd/system/springechoservice.service $ sudo chown root:root /lib/systemd/system/springechoservice.service $ sudo chmod 755 /lib/systemd/system/springechoservice.service
For your convenience, the content of the file is also shown below.
[Unit] Description=Spring Echo service After=syslog.target # CHANGE TO YOUR ENV . Environment=MYDIR=/home/vagrant/spring-echo-example Environment=SVCPORT=8080 ConditionPathExists=$ [Service] Type=simple User=springecho Group=springecho LimitNOFILE=1024 Environment=SVCNAME=springechoservice Restart=on-failure RestartSec=10 startLimitIntervalSec=60 WorkingDirectory=$ ExecStart=/usr/bin/java -jar $/build/libs/spring-echo-example-1.0.0.jar --server.port=$ -Xms256m -Xmx768m # make sure log directory exists and owned by syslog PermissionsStartOnly=true ExecStartPre=/bin/mkdir -p /var/log/$ ExecStartPre=/bin/chown syslog:adm /var/log/$ ExecStartPre=/bin/chmod 755 /var/log/$ ExecStartPre=/usr/bin/touch /var/log/$/$.log ExecStartPre=/bin/chown syslog:adm /var/log/$/$.log ExecStartPre=/bin/chmod 755 /var/log/$/$.log StandardOutput=syslog StandardError=syslog SyslogIdentifier=$ [Install] WantedBy=multi-user.target
We have instructed systemd to run the process as the user ‘springecho’, so we need to create that user and group.
$ sudo useradd springecho -s /sbin/nologin -M
Now, you should be able to enable the service, start it, then monitor the logs by tailing the systemd journal:
$ sudo systemctl enable springechoservice.service $ sudo systemctl start springechoservice $ sudo journalctl -f -u springechoservice
Listing the process should should you that the process is indeed running as the “springecho” user, and netstat should show a service running at port 8080.
$ sudo systemctl status springechoservice $ sudo ps -ef | grep spring-echo | grep -v color springe+ 4827 1 5 17:01 ? 00:00:04 /usr/bin/java -jar /home/vagrant/spring-echo-example/build/libs/spring-echo-example-1.0.0.jar --server.port=8080 -Xms256m -Xmx768m $ netstat -an | grep "LISTEN " | grep 8080 tcp6 0 0 . 8080 . * LISTEN
Logging to syslog
The systemd journal is stored as a binary file, so it cannot be tailed directly. But we have syslog forwarding enabled in the systemd service file, so it is just a matter of configuring our syslog server. For full instructions on configuring syslog on Ubuntu, read my article here. But here are quick instructions for Ubuntu 16.04.
First modify “/etc/rsyslog.conf” and uncomment the lines below which tell the server to listen for syslog messages on port 514/TCP.
module(load="imtcp") input(type="imtcp" port="514")
Then, create “/etc/rsyslog.d/30-springechoservice.conf” with the following content:
if $programname == 'springechoservice' or $syslogtag == 'springechoservice' then /var/log/springechoservice/springechoservice.log & stop
Now restart the rsyslog and springechoservice and you should see the logs going to the log file specified.
$ sudo systemctl restart rsyslog $ sudo systemctl restart springechoservice $ tail -f /var/log/springechoservice/springechoservice.log
In another terminal, if you now make a request to the service using curl
$ curl http://localhost:8080/echo?message=helloagain
You will see a message similar to the following appended to “springechoservice.log”.
Apr 15 22:51:38 xenial1 springechoservice[1859]: 2018-04-15 22:51:38.789 INFO 1859 — [tp1006485584-15] com.waiamu.open.SpringEchoApp : REQUEST:
Privileged ports
In the above example, we have the springechoservice listening on port 8080. But if we used a port less than 1024 as a non-root user, special privileges would need to be granted for this to run as a service (or in the foreground for that matter).
If we changed the server port to 80 in the systemd “springechoservice.service” file, and then restarted the service, it would not come back up properly.
$ sudo systemctl daemon-reload $ sudo systemctl restart springechoservice
Instead, we would see errors similar to below in the log.
Apr 15 19:12:29 xenial1 springechoservice[9572]: org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Jetty servlet container . Apr 15 19:12:29 xenial1 springechoservice[9572]: Caused by: java.net.SocketException: Permission denied
Instead of running the service as root, we will run ‘setcap’ against the Java binary which will allow it to bind to these ports. But then any JVM that uses the java binary would be allowed to bind to these privileged ports…so we will create a copy of the OpenJDK8 that will have these privileges.
In this way, we could use chown/chmod and the Linux file permissions to control access to this copy of the Java binary (removing group and world permissions). I won’t go through this exercise here, but I think the concept is clear enough.
$ sudo update-alternatives --config java $ cd /usr/lib/jvm $ sudo cp -r java-8-openjdk-amd64 java-8-openjdk-amd64-setcap $ sudo setcap 'cap_net_bind_service=+eip' /usr/lib/jvm/java-8-openjdk-amd64-setcap/jre/bin/java
The java binary in our new directory now has privileges to bind to ports less than 1024. So we edit “/lib/systemd/system/springechoservice.service” so that it runs the jar using this specific binary like below:
ExecStart=/usr/lib/jvm/java-8-openjdk-amd64-setcap/jre/bin/java -jar /home/vagrant/spring-echo-example/build/libs/spring-echo-example-1.0.0.jar --server.port=80 -Xms256m -Xmx768m
Then reload the systemd configuration and restart the service.
$ sudo systemctl daemon-reload $ sudo systemctl restart springechoservice
And now the logs once again reflect success and requests can be made successfully to port 80.
$ curl http://localhost:80/echo?message=port80
Signals and process monitoring
Java has only rudimentary signal handling, so if you send a signal to the springechoservice, the process will be killed. But systemd will restart it after 10 seconds as defined in the “RestartSec” key of the service file.
Although the Java program cannot determine all the signal types received, systemd certainly can and it is reported in the journal log.
The one exception is the QUIT signal which generates a thread dump to the console and journal.
Sending QUIT
$ kill -s QUIT $(ps -ef | grep spring-echo-example | grep -v color | awk )
Results in a thread dump to the journal as well as the syslog file.
Sending SIGUSR1
$ kill -s SIGUSR1 $(ps -ef | grep spring-echo-example | grep -v color | awk )
Results in the following message in the journal log, and then the process is restarted after 10 seconds.
Apr 15 18:35:15 xenial1 systemd[1]: springechoservice.service: Main process exited, code=killed, status=10/USR1 Apr 15 18:35:15 xenial1 systemd[1]: springechoservice.service: Unit entered failed state. Apr 15 18:35:15 xenial1 systemd[1]: springechoservice.service: Failed with result 'signal'. Apr 15 18:35:26 xenial1 systemd[1]: springechoservice.service: Service hold-off time over, scheduling restart.
Sending SIGINT
$ kill -s SIGINT $(ps -ef | grep spring-echo-example | grep -v color | awk )
Results in the following message in the journal log, and then the process is restarted after 10 seconds.
Apr 15 18:35:58 xenial1 systemd[1]: springechoservice.service: Main process exited, code=exited, status=130/n/a Apr 15 18:35:58 xenial1 systemd[1]: springechoservice.service: Unit entered failed state. Apr 15 18:35:58 xenial1 systemd[1]: springechoservice.service: Failed with result 'exit-code'.
Sending SIGKILL
$ kill -s SIGKILL $(ps -ef | grep spring-echo-example | grep -v color | awk )
Results in the following message in the journal log, and then the process is restarted after 10 seconds.
Apr 15 18:36:24 xenial1 systemd[1]: springechoservice.service: Main process exited, code=killed, status=9/KILL Apr 15 18:36:24 xenial1 systemd[1]: springechoservice.service: Unit entered failed state. Apr 15 18:36:24 xenial1 systemd[1]: springechoservice.service: Failed with result 'signal'.