- Generating a thread dump
- Windows environment
- Jira running from startup.bat
- Jira running as a Windows service
- Using jstack
- Linux/Unix/OS X environment
- Linux/Unix command line
- Linux/Unix Alternative: Generating thread dumps using jstack
- Linux/Unix Alternative: Generating thread dumps using scripts
- Steps for Atlassian Docker containers
- Analysis tools
- TDA
- Issue processing thread dump with TDA (NumberFormatException)
Generating a thread dump
Occasionally, Jira may appear to ‘freeze’ during execution of an operation. During these times, it is helpful to retrieve a thread dump — a log containing information about currently running threads and processes within the Java Virtual Machine. Taking thread-dumps is a non-destructive process that can be run on live systems. This document describes the steps necessary to retrieve a thread dump.
The steps necessary to retrieve the thread dump are dependent on the operating system Jira is running in — please follow the appropriate steps below.
On this page:
Windows environment
Jira running from startup.bat
You need to run the Command console as an administrator.
- In the Command console window where Jira is running, open the properties dialog box by right-clicking on the title bar and select Properties.
- Select the Layout tab.
- Under Screen buffer size, set the Height to 3000.
- Select Ok.
- With the same command console in focus, press CTRL-BREAK. This will output the thread dump to the command console.
- Scroll back in the command console until you reach the line containing «Full thread dump».
- Right-click the title bar and select Edit >Mark. Highlight the entire text of the thread dump.
- Right-click the title bar and select Edit >Copy. The thread dump can then be pasted into a text file.
Jira running as a Windows service
Using jstack
The JDK ships with a tool named jstack for generating thread dumps.
- Identify the process. Launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java (Jira) process. You may need to add the PID column using View -> Select Columns .
- Run jstack to capture a single thread dump or multiple thread dumps at set intervals:
jstack.exe -l > threaddump.txt
jstack.exe -l 22668 > threaddump.txt
Use the following command to capture 6 thread dumps of the process id
for /L %n in (1,1,6) do timeout 10 | jstack.exe -l > threaddump-%n.txt
for /L %n in (1,1,6) do timeout 10 | jstack.exe -l 22668 > threaddump-%n.txt
This will output to threaddump-%n.txt , where %n is the number of the loop iteration (starting at 1).
Common issues with jstack:
- You must run jstack as the same user that is running Jira.
- If you get the error «Not enough storage is available to process this command», download the ‘psexec’ utility from here, and then run one of the following commands, where is the Jira process ID (for example, 22668):
- To capture a single thread dump:
jstack.exe -l > threaddump.txt
1..6|foreach |Out-File -FilePath "app_threads.$(Get-Date -uformat %s).txt";sleep 10>
Linux/Unix/OS X environment
Linux/Unix command line
- Identify the java process that Jira is running in. This can be achieved by running a command similar to:
keithb 910 873 1 17:01 pts/3 00:00:18 /usr/java/jdk/bin/java -Xms128m -Xmx256m -Xms128m -Xmx256m -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.endorsed.dirs=/tmp/atlassian-jira-enterprise-3.6-standalone/common/endorsed -classpath :
for i in $(seq 6); do top -b -H -p -n 1 > jira_cpu_usage.`date +%s`.txt; kill -3 ; sleep 10; done
Linux/Unix Alternative: Generating thread dumps using jstack
If you have trouble using kill -3 to obtain a thread dump, try using jstack a java utility that will output stack traces of Java threads for a given process.
- Identify the java process that Jira is running in. This can be achieved by running a command similar to:
adam 22668 0.3 14.9 1691788 903928 ? Sl Jan27 9:36 /usr/lib/jvm/java-6-sun-1.6.0.14/bin/java -Djava.util.logging.config.file=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone/conf/logging.properties -XX:MaxPermSize=256m -Xms128m -Xmx1048m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Datlassian.mail.senddisabled=false -Datlassian.mail.fetchdisabled=false -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone/common/endorsed -classpath /home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone/bin/bootstrap.jar -Dcatalina.base=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone -Dcatalina.home=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone -Djava.io.tmpdir=/home/adam/Products/installs/atlassian-jira-enterprise-4.0.1-standalone/temp org.apache.catalina.startup.Bootstrap start
jstack 22668 > threaddump.txt
Use the following command to capture 6 thread dumps of the process ID
in 10-second intervals between each thread dump: for i in $(seq 6); do top -b -H -p -n 1 > threaddump.$(date +%s).txt; kill -3 ; sleep 10; done
for i in $(seq 6); do top -b -H -p 22668 -n 1 > threaddump.$(date +%s).txt; kill -3 22668; sleep 10; done
The output will be saved to threaddump.$(date +%s).txt in the current directory, where date +%s is the current Unix timestamp.
You can modify the seq command parameter to adjust the the number of thread dumps to capture and the sleep command parameter to adjust the time between each thread dump.
If you are connecting to the server through RDP, jstack might fail with following error:
Not enough storage is available to process this command
Linux/Unix Alternative: Generating thread dumps using scripts
You can also generate a thread dump by using scripts prepared by our Support team. That’s an easy process, and the scripts will do everything for you. As an addition to generating a thread dump, the scripts also allow you to generate heap dumps, check the disk access speed, or the Java SSL connection.
- (Optional) Install the Thready app if it’s supported by your Jira version. Thready renames Tomcat threads to include the URL they serve, which will make it easier to analyze the threads.
- Download and install the scripts from https://bitbucket.org/atlassianlabs/atlassian-support/.
- Execute the scripts when your Jira instance behaves slowly or is unresponsive.
- (Optional) The scripts also allow you to test the disk access speed. This is described in more detail in Testing disk access speed.
- When asked whether you want to capture thread dumps, enter Y.
- When asked whether you want to capture heap dumps, enter N.
- (Optional) The scripts also allow you to check the Java SSL connection.
Steps for Atlassian Docker containers
If you’re running Jira on a container, follow the steps below:
/opt/atlassian/support/thread-dumps.sh can be run via docker exec to easily trigger the collection of thread dumps from the containerized application. For example:
docker exec my_container /opt/atlassian/support/thread-dumps.sh
By default this script will collect 10 thread dumps at 5 second intervals. This can be overridden by passing a custom value for the count and interval, by using -c / —count and -i / —interval respectively. For example, to collect 20 thread dumps at 3 second intervals:
docker exec my_container /opt/atlassian/support/thread-dumps.sh --count 20 --interval 3
If you’re running the Docker container in a Kubernetes environment, you can execute the command as below:
kubectl exec -it jira-1 -n jira -- bash -c "/opt/atlassian/support/thread-dumps.sh --count 20 --interval 3"
Replace -it jira-1 with the pod name, and -n jira with the namespace where the Jira pods are running.
Thread dumps will be written to $APP_HOME/thread_dumps/ .
Note: By default this script will also capture output from top run in ‘Thread-mode’. This can be disabled by passing -n / —no-top
The Troubleshooting section on https://hub.docker.com/r/atlassian/jira-software has additional information.
Analysis tools
Try Watson, TDA, or Samurai to inspect your thread dump.
TDA
java -jar -Xmx512M ~/tda-bin-1.6/tda.jar
Issue processing thread dump with TDA (NumberFormatException)
Should you get an error on TDA console like:
Exception in thread "AWT-EventQueue-0" java.lang.NumberFormatException: For input string: "5 os_prio=0" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:441) at java.lang.Long.(Long.java:702) at com.pironet.tda.utils.ThreadsTableModel.getValueAt(ThreadsTableModel.java:80) at com.pironet.tda.utils.TableSorter.getValueAt(TableSorter.java:285) at javax.swing.JTable.getValueAt(JTable.java:2717) at javax.swing.JTable.prepareRenderer(JTable.java:5719) at javax.swing.plaf.basic.BasicTableUI.paintCell(BasicTableUI.java:2114) at javax.swing.plaf.basic.BasicTableUI.paintCells(BasicTableUI.java:2016) at javax.swing.plaf.basic.BasicTableUI.paint(BasicTableUI.java:1812) at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161) at javax.swing.JComponent.paintComponent(JComponent.java:778) at javax.swing.JComponent.paint(JComponent.java:1054) at javax.swing.JComponent.paintChildren(JComponent.java:887) at javax.swing.JComponent.paint(JComponent.java:1063) at javax.swing.JViewport.paint(JViewport.java:731) at javax.swing.JComponent.paintChildren(JComponent.java:887) at javax.swing.JComponent.paint(JComponent.java:1063) at javax.swing.JComponent.paintChildren(JComponent.java:887) at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1047) at javax.swing.JComponent.paint(JComponent.java:1063) at javax.swing.JComponent.paintToOffscreen(JComponent.java:5230) at javax.swing.BufferStrategyPaintManager.paint(BufferStrategyPaintManager.java:295) at javax.swing.RepaintManager.paint(RepaintManager.java:1249) at javax.swing.JComponent._paintImmediately(JComponent.java:5178) at javax.swing.JComponent.paintImmediately(JComponent.java:4989) at javax.swing.RepaintManager$3.run(RepaintManager.java:808) at javax.swing.RepaintManager$3.run(RepaintManager.java:796) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76) at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:796) at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:769) at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:718) at javax.swing.RepaintManager.access$1100(RepaintManager.java:62) at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1677) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733) at java.awt.EventQueue.access$200(EventQueue.java:103) at java.awt.EventQueue$3.run(EventQueue.java:694) at java.awt.EventQueue$3.run(EventQueue.java:692) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76) at java.awt.EventQueue.dispatchEvent(EventQueue.java:703) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138) at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
Apply the following command on the thread dump(s) to fix the thread header format to make it processable:
sed -i ‘s/prio=9\ os_prio=4\/prio=5/g’