Debian: kernel Out of socket memory
Dec102014
[摘要] 一大早LVS突然挂掉，短信，邮件轮番轰炸，我等苦逼SA立马上线查看。。 发现是愚蠢的LVS在频繁的踢掉和加入后端的realserver，导致服务极不稳定，从日志看问题应该出在后端的realserver。PS一下：我们的操作系统同样为愚蠢大便（Debian6）。观察了实际的realserver的日志及状态，发现FIN_WAIT2状态的连接非常多：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
Dec 10 01:39:45 sudops.com kernel: [1696088.973658] Out of socket memory Dec 10 01:39:45 sudops.com kernel: [1696088.973666] Out of socket memory Dec 10 01:39:45 sudops.com kernel: [1696088.973675] Out of socket memory TCP连接状态 # netstat -an|awk '{print $NF}' | sort | uniq -c | sort -nr | head -10 234725 FIN_WAIT2 84975 ESTABLISHED 14376 TIME_WAIT 10515 FIN_WAIT1 256 SYN_RECV 187 LAST_ACK 186 SYN_SENT 173 LISTEN 172 CLOSE_WAIT 129 0.0.0.0:* # cat /proc/net/sockstat sockets: used 66022 TCP: inuse 189515 orphan 132272 tw 76580 alloc 190763 mem 59842 UDP: inuse 129 mem 125 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 # cat /proc/sys/net/ipv4/tcp_max_orphans 262144 |
怀疑是orphan太多导致了Out of socket memory。于是尝试增大了net.ipv4.tcp_max_orphans的值，同时缩短了net.ipv4.tcp_fin_timeout的大小，以减少FIN状态。
内核参数优化可以参考我的另外文章：linux下TCP/IP及内核参数优化调优
现在已经恢复正常，基本参数及状态值如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
# cat /etc/sysctl.conf net.ipv4.tcp_max_orphans = 3276800 vm.swappiness=0 fs.file-max = 1491124 net.ipv4.tcp_max_tw_buckets = 10000 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.tcp_fin_timeout = 5 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl = 15 net.ipv4.tcp_max_syn_backlog = 8388608 net.core.netdev_max_backlog = 8388608 net.ipv4.tcp_keepalive_time = 1200 net.ipv4.tcp_window_scaling = 0 net.ipv4.tcp_sack = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.ip_local_port_range = 1024 65000 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.tcp_rmem = 4096 87380 4194304 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.tcp_mem = 94500000 915000000 927000000 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 # netstat -an|awk '{print $NF}' | sort | uniq -c | sort -nr | head -10 465079 ESTABLISHED 25749 FIN_WAIT2 10886 FIN_WAIT1 1504 CLOSE_WAIT 606 TIME_WAIT 256 SYN_RECV 245 SYN_SENT 173 LISTEN 147 LAST_ACK 129 0.0.0.0:* # cat /proc/net/sockstat sockets: used 475830 TCP: inuse 495505 orphan 27302 tw 10000 alloc 495507 mem 252960 UDP: inuse 129 mem 125 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 # cat /proc/sys/net/ipv4/tcp_max_orphans 3276800 |
因为应用类型是典型的keepalive应用，目前后端每台服务器ESTABLISHED连接数在50w左右，共有几十台realserver，看来服务器很给力，用户数也还可以哈。。
本文固定链接: https://www.sudops.com/debian-kernel-socket-memory.html | 运维·速度
Debian: kernel Out of socket memory：目前有1 条留言
发表评论
欢迎关注 运维·速度 微信公众号
