我真的在使用GPU的tenstorflow吗？

Question

我正在学习神经网络，并尝试使用GPU来实现它。我正在使用：

• Python 3.8
• tensorflow gpu 2.6。0
• PyCharm
• PyCharm的Jupiter插件
• 视频卡NVIDIA 3080 TI-12 Gb

我已经安装了CUDA 11.4（和其他几个）和CudNN-v8.2.4.15

我的起始代码非常普通：

import os
os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'

import pandas as pd
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

from tensorflow import keras
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten, Conv1D
from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator

from sklearn.preprocessing import MinMaxScaler, OneHotEncoder

问题是，当我开始NN学习时，我在任务管理器中检查资源，它看起来很奇怪，好像它使用的是CPU而不是GPU。有截图（对不起，俄语）：

因此，当我启动这个过程时，CP使用率从5%增长到40%，显卡内存从2GB增长到6GB。

感觉就像它使用视频卡内存，但使用CPU进行计算。有可能吗？

UPD:Jupiter日志：

"E:\Program Files\JetBrains\PyCharm 2020.3\bin\runnerw.exe" C:\Users\levsh\AppData\Local\Programs\Python\Python38\python.exe -m jupyter notebook --no-browser --notebook-dir=C:/Users/levsh/PycharmProjects/ipynb
[W 2021-11-01 02:53:42.606 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-11-01 02:53:42.607 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-11-01 02:53:42.607 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2021-11-01 02:53:42.614 LabApp] JupyterLab extension loaded from c:\users\levsh\appdata\local\programs\python\python38\lib\site-packages\jupyterlab
[I 2021-11-01 02:53:42.614 LabApp] JupyterLab application directory is C:\Users\levsh\AppData\Local\Programs\Python\Python38\share\jupyter\lab
[I 02:53:42.620 NotebookApp] Serving notebooks from local directory: C:/Users/levsh/PycharmProjects/ipynb
[I 02:53:42.620 NotebookApp] Jupyter Notebook 6.4.3 is running at:
[I 02:53:42.620 NotebookApp] http://localhost:8888/?token=a34f13cb67f5cf89bff0a8b8242b69a8727197a98ddb298f
[I 02:53:42.620 NotebookApp]  or http://127.0.0.1:8888/?token=a34f13cb67f5cf89bff0a8b8242b69a8727197a98ddb298f
[I 02:53:42.620 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 02:53:42.625 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///C:/Users/levsh/AppData/Roaming/jupyter/runtime/nbserver-9860-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=a34f13cb67f5cf89bff0a8b8242b69a8727197a98ddb298f
     or http://127.0.0.1:8888/?token=a34f13cb67f5cf89bff0a8b8242b69a8727197a98ddb298f
[I 02:53:42.626 NotebookApp] 302 GET /api/kernelspecs/ (127.0.0.1) 0.000000ms
[I 02:53:42.694 NotebookApp] Kernel started: 036ed1a8-7e7e-49bb-8c86-3efcfb4b993c, name: python3
[W 02:53:42.703 NotebookApp] No session ID specified
2021-11-01 02:53:49.739123: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-01 02:53:50.367361: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-11-01 02:53:50.367428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9440 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3080 Ti, pci bus id: 0000:1f:00.0, compute capability: 8.6
2021-11-01 02:53:50.526516: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2021-11-01 02:53:52.226233: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8204
2021-11-01 02:53:55.133752: I tensorflow/stream_executor/cuda/cuda_blas.cc:1760] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.

UPD2：TYZ评论的更新-不，不是。

tf.debugging.set_log_device_placement(True)

print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU'))) 
# Num GPUs Available:  1

tf.config.list_physical_devices('GPU') 
# [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

sess = tf.compat.v1.Session(config=tf.compat.v1.ConfigProto(log_device_placement=True))
# Device mapping: 
# /job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: NVIDIA GeForce RTX 3080 Ti, pci bus id: 0000:1f:00.0, compute capability: 8.6

在日志中，它看起来也不错：

2021-11-01 04:34:49.606386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /device:GPU:0 with 9440 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3080 Ti, pci bus id: 0000:1f:00.0, compute capability: 8.6
2021-11-01 04:36:58.513741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9440 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3080 Ti, pci bus id: 0000:1f:00.0, compute capability: 8.6

Answer 1

在程序运行时检查nvidia smi（nvidia系统管理界面），查找易失性GPU util。任务管理器不能很好地指示GPU的使用情况（也不能非常准确地使用其他资源，如RAM和temps imo…）。3080 Ti的GPU温度为71度这一事实表明GPU确实在使用（除非其他进程正在使用它）

例如，我现在正在使用RTX 3090进行培训，命令行的smi输出如下（截取屏幕截图中的进程）：

但是我的任务管理器看起来像（注意gpu的使用）：

现在，如果你有某种I/O瓶颈，即从CPU加载张量花费的时间太长，因此GPU处于空闲状态，那么这是另一个问题，可以通过分析和其他工具来解决，以确保加载过程得到优化。