CPU LOAD AVRG + how to deal process with D state
0
votes
1
answer
1341
views
we can see from our
RHEL 7.6
server ( kernel version - 3.10.0-957.el7.x86_64
) that following process are with D
state ( they runs from HDFS
user )
Note - *D state code means that process is in uninterruptible sleep*
ps -eo s,user,cmd | grep ^[RD]
D hdfs du -sk /grid/sdj/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990
D hdfs du -sk /grid/sdm/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990
R root ps -eo s,user,cmd
note's - the disks sdj
and sdm
are 3T
byte size , also "du -sk
" happens on other disks as sdd , sdf
etc
and the disks are with ext4 file-system
we are suspect that the fact that we have high CPU load avrg is because the "du -sk" that actually run on the disks
so I was thinking what we can do regarding to below behavior
one option is maybe to disable the "du -sk
" verification from HDFS
, but no clue how to do that
second option is to think what actually cause the D
state ?
I don't sure ... but maybe upgrade the kernel version will help to avoid D state? or else? ( like disable the CPU Thread(s) ) , etc ?
more details
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
and CPU LOAD AVRG is around ~ 42-45 ( for 15min avrg )
Reference :
https://community.cloudera.com/t5/Support-Questions/Does-hadoop-run-dfs-du-automatically-when-a-new-job-starts/td-p/231297
https://community.cloudera.com/t5/Support-Questions/Can-hdfs-dfsadmin-and-hdfs-dsfs-du-be-taxing-on-my-cluster/m-p/182402
https://community.pivotal.io/s/article/Dealing-with-Processes-in-State-D---Uninterruptible-Sleep-Usually-IO?language=en_US
https://www.golinuxhub.com/2018/05/how-to-disable-or-enable-hyper/
Asked by yael
(13936 rep)
Nov 28, 2021, 02:16 PM
Last activity: Nov 28, 2021, 02:43 PM
Last activity: Nov 28, 2021, 02:43 PM