Control NVIDIA GPU Fan Speed on Linux Ubuntu 16.04

NVIDIA GPUs have fan speed profiles that control the fan to keep the noise to a minimum. In high-performance computing applications when the system is in a datacenter, it is often preferable to run the fan at maximum speeds for lowest possible operating temperature to possibly prolong the life of the hardware.

The following assumes you installed Ubuntu Server without desktop environment, you do not want to run a desktop manager, and you need a command prompt after reboot. This generally applies to datacenter and remote server environments.

A simple search on the the Internet was not sufficient to make things work so here is a write-up that works. Unfortunately it appears that we cannot avoid installing and keeping running an Xorg server, just to change GPU settings and keep them that way. The following steps are needed to make this permanent, and without any display attached (a headless system.)

The procedure was tested on Ubuntu 16.04.2 LTS with EVGA GTX 1050 Ti GPUs, on Ubuntu 16.04.3 LTS with EVGA and ASUS GTX 1070 and 1080 Ti GPUs. Our post on Mining Pods using CentOS 7.4 with CUDA 9.1, covers the procedure for CentOS, and was tested with NVIDA FE GPU cards and some Zotac cards.

[ ] Install Xorg and create xorg.conf configuration file
[ ] Create a control script
[ ] Make the script run on boot

Install and Configure Xorg

First we need to have Xorg installed and configured properly.

sudo apt install -y xorg

Without the “cool-bits” option set to a proper value in the xorg.conf file, trying to set fan speeds will just not work. You can check any existing configuration file /etc/X11/xorg.conf for the coolbits setting, or just create a new one using the command below.

sudo nvidia-xconfig --enable-all-gpus --allow-empty-initial-configuration --cool-bits=7

If you want to set all cool-bits binary bits to ‘1’ to control other aspects of the GPU, you can just set –cool-bits=31. Above command creates sections for each GPU in the system. Coolbits are set in Section “Screen”.

Fan speed configuration

Create a script (named gpufansetting in this example) in your user home or somewhere convenient. This will probably need to be executed at boot. The script should not terminate Xorg, or the custom speed settings may revert to their defaults. It is also necessary to have graphical.target enabled to start an Xorg server. Optionally you can disable it at the end of the script by changing to multi-user.target which then give you your command line prompt on the terminal after reboot.

vi ~/gpufansetting
[ ] Add the following
#!/bin/bash
#a delay can be specified on the command line if not used from termonal session
DEFAULT_DELAY=0
if [ "x$1" = "x" -o "x$1" = "xnone" ]; then
 DELAY=$DEFAULT_DELAY
else
 DELAY=$1
fi
sleep $DELAY
#write log file for debugging
exec 2> /tmp/gpufansetting.log
exec 1>&2
set -x
#make sure an Xorg server is running which you can connect to
# ":1" can be any unused display number.
# there may be better ways using an already running server, but this works.
#more on targets: https://www.systutorials.com/239880/change-systemd-boot-target-linux/
#not tested: isolate could be the better way than using set-default!
systemctl set-default graphical.target
X :1 &
sleep 10
export DISPLAY=:1
#set GPU 0..5 fan to 100%. (This assumes six GPUs)
nvidia-settings -a [gpu:0]/GPUFanControlState=1 -a [fan-0]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:1]/GPUFanControlState=1 -a [fan-1]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:2]/GPUFanControlState=1 -a [fan-2]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:3]/GPUFanControlState=1 -a [fan-3]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:4]/GPUFanControlState=1 -a [fan-4]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:5]/GPUFanControlState=1 -a [fan-5]/GPUTargetFanSpeed=100
#set back to command line prompt after boot (optional)
sudo systemctl set-default multi-user.target
#terminating Xorg will reset fan speeds to default so do not terminate Xorg!
#sudo killall Xorg
[ ] Save and exit
<Esc>:wq<Enter>
[ ] Make the script executable:
sudo chmod +x gpufansetting

The portion after the first systemctl starts Xorg. Once started you can execute the commands that follow from the command line. This works from a remote terminal also.

If you want to try it from your (remote ssh) terminal first without rebooting or running scripts, use e.g. sudo X :2 &, where the number following the colon (e.g. 2) must be different from an already running server, and then export DISPLAY=:2, and a command to change a parameter: sudo nvidia-settings …

A GPU that ran at 82C at fan speed 51% now runs at a much nicer 74C at fan speed 100%! Results will depend on your ambient temperature.

Run Script at Boot

Finally we add a line to /etc/rc.local, before exit(0) to automatically run the script at boot. Don’t forget to replace the path /home/userid/gpufansetting by the path to your own script.

[ ] sudo vi /etc/rc.local
[ ] Add:
exec /home/userid/gpufansetting
[ ] Save and exit editor: <Esc>:wq<Enter>

To be able to debug your script you can add the following before the line you added. You will be able to see what happened in the log file /tmp/rc.local.log.

exec 2> /tmp/yourlogfile.log
exec 1>&2
set -x

Reboot the system and keep your fingers crossed: sudo reboot. If things do not work, check /tmp/rc.local.log and /tmp/gpufansetting.log for any errors.

If you are mining, confirm the hashing performance. On one occasion during our experimentation we found severely degraded performance to less than 20%. However, there should be no degradation if your follow these instructions and after a clean reboot.

To adjust the fan speed during operation of the system you can use from a remote terminal, e.g.

sudo env DISPLAY=:1 nvidia-settings -a [gpu:0]/GPUFanControlState=1 -a [fan-0]/GPUTargetFanSpeed=89

or equivalently

sudo nvidia-settings -c :1 -a [gpu:0]/GPUFanControlState=1 -a [fan-0]/GPUTargetFanSpeed=89