三三世界-百宝箱

文章 分类 标签
130 9 247

公告

暂无站点公告

Linux服务器卡死报错:INFO: task blocked for more than 120 seconds

三三 2022-05-18 2492 1条评论 Linux使用 Linux服务器卡死task blockedhung_task_timeout_secs

首页 / 正文

1、问题

服务器突然不正常,假死,无法连接ssh,端口都通,ip也通。进入kvm可以看到:

INFO: task blocked for more than 120 seconds 

或者

info : task kwoker/8:0:51 blocked for more than 120 seconds
echo 0 ? proc/sys/kernel/bung_task_timeout_secs disables this message

1818.png

强制重启以后机器立即恢复,查看日志和监控,发现服务器负载很高,但是CPU利用率不高。出问题的时候IO超级低,服务器经常夯住,网站打不开。

2、问题原因:

关键信息是“hung_task_timeout_secs”,通过搜索,发现这是linux kernel的一个bug。摘抄一段:

By default Linux uses up to 40% of the available memory for file system caching.

After this mark has been reached the file system flushes all outstanding data to disk causing all following IOs going synchronous.

For flushing out this data to disk this there is a time limit of 120 seconds by default.

In the case here the IO subsystem is not fast enough to flush the data withing 120 seconds.

This especially happens on systems with a lot of memory.

The problem is solved in later kernels。

一般情况下,Linux会把可用内存的40%的空间作为文件系统的缓存cache。

默认情况下, Linux会最多使用40%的可用内存作为文件系统缓存。当超过这个阈值后,文件系统会把将缓存中的内存全部写入磁盘, 导致后续的IO请求都是同步的。

将缓存写入磁盘时,有一个默认120秒的超时时间。 出现上面的问题的原因是IO子系统的处理速度不够快,不能在120秒将缓存中的数据全部写入磁盘。

IO系统响应缓慢,导致越来越多的请求堆积,最终系统内存全部被占用,导致系统失去响应。

3、解决方法:

缩小文件系统缓存大小

降低缓存占内存的比例,比如由40%降到10%,这样的话需要同步到磁盘上的数据量会变小,IO写时间缩短,会相对比较平稳。

文件系统缓存的大小是由内核参数vm.dirty_ratio 和 vm.dirty_backgroud_ratio控制决定的。

vm.dirty_background_ratio 指定当文件系统缓存脏页数量达到系统内存百分之多少时(如5%)就会触发pdflush/flush/kdmflush 等后台回写进程运行,将一定缓存的脏页异步地刷入外存。

vm.dirty_ratio则指定了当文件系统缓存脏页数量达到系统内存百分之多少时(如10%),系统不得不开始处理缓存脏页(因为此时脏页数量已经比较多,为了避免数据丢失需要将一定脏页刷入外存),在此过程中很多应用进程可能会因为系统转而处理文件IO而阻塞。

通常情况下,vm.dirty_ratio的值要大于vm.dirty_background_ratio的值。

根据应用程序情况,对vm.dirty_ratio,vm.dirty_background_ratio两个参数进行调优设置。 例如,推荐如下设置:

sysctl -w vm.dirty_ratio=10
sysctl -w vm.dirty_background_ratio=5
sysctl -p

如果系统永久生效,修改/etc/sysctl.conf文件。加入如下两行:

编辑#vi /etc/sysctl.conf

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

重启#sysctl -p

重启系统生效。问题解决。

评论(1)

  1. Penzu 游客 2024-03-02 21:15 回复

    Heello would you mind sharing which blog platform you're
    using? I'm planning to start my own blog in the near future
    but I'm having a tough time making a decision between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your layout seems different
    then moost blogs and I'm looking for something completely unique.
    P.S Apologies forr being off-topic butt I had to ask!


Copyright 2021 三三世界-百宝箱. All Rights Reserved.

最新评论

  • 三三

    @Doug Shume it's ok for me , you can post here.

  • refugiaguenther

    Saved as a favorite, I like your website!

  • josefa

    If some one wishes to be updated with hottest technologies after that he must be visit this site and be up to date daily.

  • Penzu

    Heello would you mind sharing which blog platform you're using? I'm planning to start my own blog in the near future but I'm having a tough time making a decision between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your layout seems different then moost blogs and I'm looking for something completely unique. P.S Apologies forr being off-topic butt I had to ask!

  • Yupoo Fendi

    Thanks to my father who shared with me regarding this webpage, this website is genuinely amazing.

  • Doug Shume

    Hi, I have an overflow of customers that I'd like to send to you but I want to make sure you can handle more leads, let me know if you'd like me to send you more info.

  • SuperWind

    zh.us.to 有效
    kms.03k.org 有效
    kms.chinancce.com
    kms.shuax.com 有效
    kms.dwhd.org 有效
    kms.luody.info 有效
    kms.digiboy.ir 有效
    kms.lotro.cc 有效
    www.zgbs.cc 有效
    cy2617.jios.org 有效

  • 三三

    @         权限问题,试试sudo 再加命令。

  •         

    你好提示Permission denied 怎么办啊

日历

2024年06月

      1
2345678
9101112131415
16171819202122
23242526272829
30      

文章目录

上一张 下一张