Посмотреть загрузку видеокарты linux

GPU usage monitoring (CUDA)

I installed CUDA toolkit on my computer and started BOINC project on GPU. In BOINC I can see that it is running on GPU, but is there a tool that can show me more details about that what is running on GPU — GPU usage and memory usage?

20 Answers 20

For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU. There also is a list of compute processes and few more options but my graphic card (GeForce 9600 GT) is not fully supported.

Sun May 13 20:02:49 2012 +------------------------------------------------------+ | NVIDIA-SMI 3.295.40 Driver Version: 295.40 | |-------------------------------+----------------------+----------------------+ | Nb. Name | Bus Id Disp. | Volatile ECC SB / DB | | Fan Temp Power Usage /Cap | Memory Usage | GPU Util. Compute M. | |===============================+======================+======================| | 0. GeForce 9600 GT | 0000:01:00.0 N/A | N/A N/A | | 0% 51 C N/A N/A / N/A | 90% 459MB / 511MB | N/A Default | |-------------------------------+----------------------+----------------------| | Compute processes: GPU Memory | | GPU PID Process name Usage | |=============================================================================| | 0. Not Supported | +-----------------------------------------------------------------------------+ 

@donlucacorleone man watch tells us the -d flag highlights differences between the outputs, so it can aid in highlighting which metrics are changing over time.

For linux, use nvidia-smi -l 1 will continually give you the gpu usage info, with in refresh interval of 1 second.

I prefer to use watch -n 1 nvidia-smi to obtain continuous updates without filling the terminal with output

Using watch means your starting a new process every second to poll the cards. Better to do -l, and not every second, I’d suggest every minute or every 5 minutes.

Recently I have written a simple command-line utility called gpustat (which is a wrapper of nvidia-smi ) : please take a look at https://github.com/wookayin/gpustat.

I had to use sudo -H pip install gpustat comamnd to install and then i could just run gpustat -a -i 3 and works like charm

For Intel GPU’s there exists the intel-gpu-tools from http://intellinuxgraphics.org/ project, which brings the command intel_gpu_top (amongst other things). It is similar to top and htop , but specifically for the Intel GPU.

 render busy: 18%: ███▋ render space: 39/131072 bitstream busy: 0%: bitstream space: 0/131072 blitter busy: 28%: █████▋ blitter space: 28/131072 task percent busy GAM: 33%: ██████▋ vert fetch: 0 (0/sec) GAFS: 3%: ▋ prim fetch: 0 (0/sec) VS: 0%: VS invocations: 559188 (150/sec) SF: 0%: GS invocations: 0 (0/sec) VF: 0%: GS prims: 0 (0/sec) DS: 0%: CL invocations: 186396 (50/sec) CL: 0%: CL prims: 186396 (50/sec) SOL: 0%: PS invocations: 8191776208 (38576436/sec) GS: 0%: PS depth pass: 8158502721 (38487525/sec) HS: 0%: TE: 0%: GAFM: 0%: SVG: 0%: 

nvidia-smi does not work on some linux machines (returns N/A for many properties). You can use nvidia-settings instead (this is also what mat kelcey used in his python script).

nvidia-settings -q GPUUtilization -q useddedicatedgpumemory 

watch -n0.1 "nvidia-settings -q GPUUtilization -q useddedicatedgpumemory" 

for continuous monitoring.

Thanks, this is what worked for me, since I have a GeForce card which is not supported by nvidia-smi.

You can do nvidia-settings -q all to see what other parameters you can monitor. I’m monitoring GPUCurrentProcessorClockFreqs and GPUCurrentClockFreqs .

@Hossein: That might be because nvidia-settings looks at the X Display variable $DISPLAY . In a GPGPU server, that won’t work — if only because such servers typically have multiple GPU’s

I have a GeForce 1060 GTX video card and I found that the following command give me info about card utilization, temperature, fan speed and power consumption:

$ nvidia-smi --format=csv --query-gpu=power.draw,utilization.gpu,fan.speed,temperature.gpu 

You can see list of all query options with:

You can use nvtop , it’s similar to htop but for NVIDIA GPUs. Link: https://github.com/Syllo/nvtop

For Linux, I use this HTOP like tool that I wrote myself. It monitors and gives an overview of the GPU temperature as well as the core / VRAM / PCI-E & memory bus usage. It does not monitor what’s running on the GPU though.

Recently, I have written a monitoring tool called nvitop , the interactive NVIDIA-GPU process viewer.

It is written in pure Python and is easy to install.

pip3 install --upgrade nvitop 

Install the latest version from GitHub:

pip3 install git+https://github.com/XuehaiPan/nvitop.git#egg=nvitop 

Run as a resource monitor:

nvitop will show the GPU status like nvidia-smi but with additional fancy bars and history graphs.

For the processes, it will use psutil to collect process information and display the USER , %CPU , %MEM , TIME and COMMAND fields, which is much more detailed than nvidia-smi . Besides, it is responsive for user inputs in monitor mode. You can interrupt or kill your processes on the GPUs.

nvitop comes with a tree-view screen and an environment screen:

In addition, nvitop can be integrated into other applications. For example, integrate into PyTorch training code:

import os from nvitop.core import host, CudaDevice, HostProcess, GpuProcess from torch.utils.tensorboard import SummaryWriter device = CudaDevice(0) this_process = GpuProcess(os.getpid(), device) writer = SummaryWriter() for epoch in range(n_epochs): # some training code here # . this_process.update_gpu_status() writer.add_scalars( 'monitoring', < 'device/memory_used': float(device.memory_used()) / (1 , global_step ) 

Note: nvitop is dual-licensed by the GPLv3 License and Apache-2.0 License. Please feel free to use it as a dependency for your own projects. See Copyright Notice for more details.

Источник

How to Check GPU (Intel/AMD/NVIDIA) Usage in Ubuntu 22.04 | 20.04

Unlike Windows, Ubuntu so far does not have real-time GPU usage info displayed in its system monitor application. If you want to check how much your graphic card is in use, then this simple tutorial may help!

Check Intel GPU usage in Ubuntu:

For the integrated Intel graphics card, there’s a command line tool intel_gpu_top can do the job.

1. First, press Ctrl+Alt+T on terminal to open a terminal window. When it opens, run command to install the package:

sudo apt install intel-gpu-tools

Type user password when it asks (no asterisk feedback) and hit Enter.

2. Then, run command to start it:

As you can see, it shows real-time IMC read and write speed, power usage, as well as percentage usage of 3D/Render, Blitter, Video, and VideoEnhance.

Monitor AMD/NVIDIA GPU usage in Ubuntu

For NVIDIA and AMD graphics card, there’s a htop like task monitor called nvtop (Neat Videocard TOP).

It shows real-time GPU and GPU Memory usage in both total and per process basis, along with temperature, power usage, and graph information.

nvtop, image from https://github.com/Syllo/nvtop

nvtop also supports Intel GPU, however, it does not work in my case with i5-4590 (HD4600), and i3-6006U (HD 520).

Install nvtop in Ubuntu:

The tool is available in Ubuntu system repository, but old. For the latest version, it has an official PPA contains the packages for Ubuntu 20.04, Ubuntu 22.04, Ubuntu 22.10, and next Ubuntu 23.04

1. First, press Ctrl+Alt+T on keyboard to open terminal. Then, run command to add the PPA:

sudo add-apt-repository ppa:flexiondotorg/nvtop

Type user password (no asterisk feedback) when it asks and hit Enter to continue.

2. Then, run command to install the tool:

For Linux Mint, run sudo apt update first to refresh package cache.

Finally, either search for and launch it from ‘Activities’ overview (or start menu), or run nvtop command to start monitoring your GPU.

(Optional) Remove nvtop

To remove the PPA repository, open terminal and run command:

sudo add-apt-repository --remove ppa:flexiondotorg/nvtop

And remove nvtop if you want via command:

Источник

Как узнать статистику использования GPU в Ubuntu?

Ubuntu до сих пор не отображает информацию об использовании графического процессора в системном мониторе. Однако график использования GPU можно легко посмотреть, для этого есть соответствующие программы. Сейчас мы их и рассмотрим.

NVTOP - мониторинг загрузки видеокарт AMD/NVIDIA

Аббревиатура NVTOP расшифровывается как Neat Videocard TOP, она поддерживает большинство современных видеочипов от AMD, NVIDIA. Программа NVTOP отображает использование графического процессора, а также видеопамяти как для отдельного процесса, так и в целом. Дополнительно отображается температура, частоты и энергопотребление.

Теоретически есть поддержка даже видеокарт Intel, но она пока находится в зачаточном состоянии (только драйвер i915). Для AMD поддерживаются все GPU с драйвером amdgpu. Видеокарты NVIDIA поддерживаются, начиная с поколения Kepler (GeForce 600, GeForce 800M и выше).

Для установки NVTOP лучше воспользоваться PPA, так как в официальных репозиториях хранится старая версия (в PPA на текущий момент есть версии для Ubuntu 20.04, Ubuntu 22.04, Ubuntu 22.10 и Ubuntu 23.04).

sudo add-apt-repository ppa:flexiondotorg/nvtop

После установки запустите ее командой nvtop.

Intel GPU TOP - для видеокарт от Intel

Для графики от Intel также есть аналогичная программа - Intel GPU TOP. Ее интерфейс немного попроще, но основную информацию она отображает: загрузку видеочипа, скорость обмена с видеопамятью, частоты и энергопотребление.

Для установки Intel GPU TOP воспользуйтесь командой:

sudo apt install intel-gpu-tools

Источник

How to measure GPU usage?

Using the top command in the terminal lists processes, sorted by their CPU usage (and you can change it to sort by another parameter) Is there an equivalent for the GPU? This fellow is asking about RAM used by GPU

Deppending, if you are using a radeon you can use radeontop, for nvidia there's another tool but I don't have the name at hand.

For a nvidia gpu, you can use nvidia-smi -l 5 , which will provide an update every 5 seconds. (Change this number to update at a different interval.)

10 Answers 10

• For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU.
• For Intel GPU's you can use the intel-gpu-tools.
• AMD has two options
• fglrx (closed source drivers):

aticonfig won't work over SSH. Claims it needs an X server running to work (there is one running). However, RadeonTop ( sudo apt-get radeontop ) does work with the fglrx (needs root). Hurrah! Sadly RadeonTop doesn't provide any temperature readings.

aticonfig WILL work over SSH, but an X server with tcp enabled needs to be running. This can be done by configuring lightdm via xserver-allow-tcp=true . Searching around this site with these keywords should lead to the result.

Not sure why but watch -n 1 nvidia-smi gave me real-time updates. watch nvidia-smi has a 2 sec update delay.

You can use gpustat , which is a simple command-line script (wrapper for nvidia-smi) for querying and monitoring GPU status:

Install intel-gpu-tools (its likely that they are installed already)

sudo apt-get install intel-gpu-tools 

Nvidia: to continuously update the output of nvidia-smi , you can use nvidia-smi --loop=1 (refresh interval of 1 second) or nvidia-smi --loop-ms=1000 (refresh interval of 1000 milliseconds).

 -l SEC, --loop=SEC Continuously report query data at the specified interval, rather than the default of just once. The application will sleep in-between queries. Note that on Linux ECC error or XID error events will print out during the sleep period if the -x flag was not specified. Pressing Ctrl+C at any time will abort the loop, which will otherwise run indef‐ initely. If no argument is specified for the -l form a default inter‐ val of 5 seconds is used. -lms ms, --loop-ms=ms Same as -l,--loop but in milliseconds. 

Источник