CUDA升级，从9.2到11.0

故事背景

系统配置

操作系统

1
2
$ cat /proc/version
Linux version 4.4.0-138-generic (buildd@lcy01-amd64-006) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #164-

算法模型 Backbone

  • ResNet V2
  • ResNet 101
  • ResNet 50
  • Mask RCNN

故事描述

在测试环境中，升级组件时，误把CUDA通过命令进行了升级，但由于cuda版本与cudnn版本不匹配，结果导致nvidia-smi命令报错，基于TensorFlow Serving的GPU图像识别推理也无法正常进行，于是做了些尝试终于将两者匹配，此时CUDA版本已经升级至11.0，经过测试意外地发现，此前的图像识别速度竟然提升了好几倍，猜测可能是CUDA相关升级导致的识别速度提升，但与算法同事和leader沟通没有结论后，我决定在新的测试机器上正式升级CUDA版本并验证是否速度会有提升

故事经过

安装CUDA驱动

先在NVIDIA官网找到对应版本的cuda，执行命令

1
2
wget http://developer.download.nvidia.com/compute/cuda/11.0.2/local_installers/cuda_11.0.2_450.51.05_linux.run
sudo sh cuda_11.0.2_450.51.05_linux.run

安装过程中全选，收到报错提示，查看cuda安装log 

1
cat /var/log/cuda-installer.log

提示

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[INFO]: Driver installation detected by command: apt list --installed | grep -e nvidia-driver-[0-9][0-9][0-9] -e nvidia-[0-9][0-9][0-9]
[INFO]: Cleaning up window
[INFO]: Complete
[INFO]: Checking compiler version...
[INFO]: gcc location: /usr/bin/gcc
[INFO]: gcc version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
[INFO]: Initializing menu
[INFO]: Setup complete
[INFO]: Components to install:
[INFO]: Driver
[INFO]: 450.51.05
[INFO]: Executing NVIDIA-Linux-x86_64-450.51.05.run --ui=none --no-questions --accept-license --disable-nouveau --no-cc-version-check --install-libglvnd  2>&1
[INFO]: Finished with code: 256
[ERROR]: Install of driver component failed.
[ERROR]: Install of 450.51.05 failed, quitting
huqian@aly-hb3-retail-resident-gpu05:~$ cat /var/log/nvidia-installer.log
nvidia-installer log file '/var/log/nvidia-installer.log'
creation time: Sun Sep 27 00:48:09 2020
installer version: 450.51.05
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
nvidia-installer command line:
    ./nvidia-installer
    --ui=none
    --no-questions
    --accept-license
    --disable-nouveau
    --no-cc-version-check
    --install-libglvnd
Using built-in stream user interface
-> Detected 8 CPUs online; setting concurrency level to 8.
-> The file '/tmp/.X0-lock' exists and appears to contain the process ID '1165' of a runnning X server.
ERROR: You appear to be running an X server; please exit X before installing.  For further details, please see the section INSTALLING THE NVIDIA DRIVER in the README available on the Linux driver download page at www.nvidia.com.
ERROR: Installation has failed.  Please see the file '/var/log/nvidia-installer.log' for details.  You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.
1
sudo rm -rf /tmp/.X0-lock

并再次尝试安装，再次报错，查看log

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
nvidia-installer log file '/var/log/nvidia-installer.log'
creation time: Sun Sep 27 00:51:35 2020
installer version: 450.51.05
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
nvidia-installer command line:
    ./nvidia-installer
    --ui=none
    --no-questions
    --accept-license
    --disable-nouveau
    --no-cc-version-check
    --install-libglvnd
Using built-in stream user interface
-> Detected 8 CPUs online; setting concurrency level to 8.
ERROR: An NVIDIA kernel module 'nvidia-uvm' appears to already be loaded in your kernel.  This may be because it is in use (for example,by an X server, a CUDA program, or the NVIDIA Persistence Daemon), but this may also happen if your kernel was configured without support for module unloading.  Please be sure to exit any programs that may be using the GPU(s) before attempting to upgrade your driver.  If no GPU-based programs are running, you know that your kernel supports module unloading, and you still receive this message, then an error may have occured that has corrupted an NVIDIA kernel module's usage count, for which the simplest remedy is to reboot your computer.
ERROR: Installation has failed.  Please see the file '/var/log/nvidia-installer.log' for details.  You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.

从网上查阅解决方案后，执行

1
2
3
4
sudo service lightdm stop
sudo stop nvidia-digits-server # 使用报错
sudo service docker stop # 未使用
sudo rmmod nvidia-uvm # 提示不存在

此时发现TensorFlow Serving进程还有，于是kill后再次尝试安装，

继续报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
nvidia-installer log file '/var/log/nvidia-installer.log'
creation time: Sun Sep 27 01:11:14 2020
installer version: 450.51.05
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
nvidia-installer command line:
    ./nvidia-installer
    --ui=none
    --no-questions
    --accept-license
    --disable-nouveau
    --no-cc-version-check
    --install-libglvnd
Using built-in stream user interface
-> Detected 8 CPUs online; setting concurrency level to 8.
-> Installing NVIDIA driver version 450.51.05.
-> The NVIDIA driver appears to have been installed previously using a different installer. To prevent potential conflicts, it is recommended either to update the existing installation using the same mechanism by which it was originally installed, or to uninstall the existing installation before installing this driver.
Please review the message provided by the maintainer of this alternate installation method and decide how to proceed:
The package that is already installed is named nvidia-410.
You can upgrade the driver by running:
`apt-get install nvidia-410 nvidia-modprobe nvidia-settings`
You can remove nvidia-410, and all related packages, by running:
`apt-get remove --purge nvidia-410 nvidia-modprobe nvidia-settings`
This package is maintained by NVIDIA (cudatools@nvidia.com).
(Answer: Abort installation)
ERROR: The installation was canceled due to the availability or presence of an alternate driver installation. Please see /var/log/nvidia-installer.log for more details.

依据提示方法，进行了尝试

1
sudo apt-get remove --purge nvidia-410 nvidia-modprobe nvidia-settings

结果发现此时变成了nivdia 440，继续执行cuda安装，报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
nvidia-installer log file '/var/log/nvidia-installer.log'
creation time: Sun Sep 27 01:22:16 2020
installer version: 450.51.05
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
nvidia-installer command line:
    ./nvidia-installer
    --ui=none
    --no-questions
    --accept-license
    --disable-nouveau
    --no-cc-version-check
    --install-libglvnd
Using built-in stream user interface
-> Detected 8 CPUs online; setting concurrency level to 8.
-> Installing NVIDIA driver version 450.51.05.
-> The NVIDIA driver appears to have been installed previously using a different installer. To prevent potential conflicts, it is recommended either to update the existing installation using the same mechanism by which it was originally installed, or to uninstall the existing installation before installing this driver.
Please review the message provided by the maintainer of this alternate installation method and decide how to proceed:
The package that is already installed is named nvidia-440.
You can upgrade the driver by running:
`apt-get install nvidia-440 nvidia-modprobe nvidia-settings`
You can remove nvidia-440, and all related packages, by running:
`apt-get remove --purge nvidia-440 nvidia-modprobe nvidia-settings`
This package is maintained by NVIDIA (cudatools@nvidia.com).
(Answer: Abort installation)
ERROR: The installation was canceled due to the availability or presence of an alternate driver installation. Please see /var/log/nvidia-installer.log for more details.

参照其指示，卸载440

1
sudo apt-get remove --purge nvidia-440 nvidia-modprobe nvidia-settings

再次安装，提醒是否更新cuda软链接，选择yes

1
2
3
4
5
6
7
8
9
10
11
12
13
===========
= Summary =
===========
Driver:   Installed
Toolkit:  Installed in /usr/local/cuda-11.0/
Samples:  Installed in /home/huqian/
Please make sure that
 -   PATH includes /usr/local/cuda-11.0/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-11.0/lib64, or, add /usr/local/cuda-11.0/lib64 to /etc/ld.so.conf and run ldconfig as root
To uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-11.0/bin
To uninstall the NVIDIA Driver, run nvidia-uninstall
Please see CUDA_Installation_Guide_Linux.pdf in /usr/local/cuda-11.0/doc/pdf for detailed information on setting up CUDA.
Logfile is /var/log/cuda-installer.log

此时安装过程似乎顺利完成，查看状态

1
nvidia-smi

尝试启动TensorFlow Serving试试，报错

1
2
2020-09-27 01:33:45.682888: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-09-27 01:33:45.688000: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

安装cudnn

从NVIDIA官网下载版本匹配的CUDNN安装文件，并执行命令

1
sudo dpkg -i libcudnn8_8.0.3.33-1+cuda11.0_amd64.deb

提示

1
2
3
4
5
(Reading database ... 168907 files and directories currently installed.)
Preparing to unpack libcudnn8_8.0.3.33-1+cuda11.0_amd64.deb ...
Unpacking libcudnn8 (8.0.3.33-1+cuda11.0) over (8.0.3.33-1+cuda11.0) ...
Setting up libcudnn8 (8.0.3.33-1+cuda11.0) ...
Processing triggers for libc-bin (2.23-0ubuntu11.2) ...

进入其中并make

1
2
cd NVIDIA_CUDA-11.0_Samples/
sudo make

经历了好久，终于装好了，不重启TensorFlow Serving仍然是同样错误，重启TensorFlow Serving后正常

故事结果

测试效果

使用原有基准图片进行测试，算法模型见配置信息，使用未优化前的P100机器，推理时间在2.7s，优化后则缩短到了0.6s

故事探究

从安装新版CUDA过程中，发现其输出log里包含很多模型相关名字，例如ResNet等，猜测是新版CUDA对模型做了额外优化，具体原因还待探究

赏

谢谢你请我吃糖果

扫一扫，分享到微信

微信分享二维码

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

Python后端开发，业余前端，自学CV中