zabbix-server添加了网络设备模板导致boot.log占满根目录不能ssh登陆
今天上午准备登陆下测试环境的zabbix-server服务器查个东西，发现ssh一直连接不上报错"No space left on device"。
1[C:\~]$ ssh 172.16.131.142
2Last login: Fri Nov 1 11:28:19 2019 from 10.16.75.35
3/root/.pyenv/libexec/pyenv-init: line 131: cannot create temp file for here-document: No space left on device
于是使用ansible跳过去，查看磁盘空间发现根目录已经100%了。
1[root@ansible ~]# ssh 172.16.131.142
2[root@zabbix1 ~]# df -h
3Filesystem Size Used Avail Use% Mounted on
4/dev/vda1 50G 50G 0 100% /
5/dev/mapper/datavg-home_lv
6 343G 178G 148G 55% /home
7/dev/mapper/datavg-swap_lv
8 976M 490M 436M 53% /swap
因为之前遇到过类似情况，所以我猜想还是boot.log满了，去看一下果然41个G。
1[root@zabbix1 ~]# cd /var/log/
2[root@zabbix1 log]# du -sh *
326M audit
441G boot.log
54.0K dmesg
64.0K dmesg.old
74.0K dracut.log
850M httpd
9824M messages
104.0K tallylog
11224K wtmp
124.0K yum.log
1321M zabbix
查看日志里面的内容，日志在疯狂的写入，只截取部分。
1[root@zabbix1 log]# tail -f boot.log
2Nov 1 11:33:22 172.16.32.2 date=2019-11-01 time=11:33:22 devname=BJ-YZ-CO-FW1 devid=FG5H0E5818903326 logid=0103020301 type=event subtype=router level=information vd=root logdesc="Routing log" msg="OSPF: RtrPriority 1"
3Nov 1 11:33:22 172.16.32.2 date=2019-11-01 time=11:33:22 devname=BJ-YZ-CO-FW1 devid=FG5H0E5818903326 logid=0103020301 type=event subtype=router level=information vd=root logdesc="Routing log" msg="OSPF: RtrDeadInterval 12"
4Nov 1 11:33:22 172.16.32.2 date=2019-11-01 time=11:33:22 devname=BJ-YZ-CO-FW1 devid=FG5H0E5818903326 logid=0103020301 type=event subtype=router level=information vd=root logdesc="Routing log" msg="OSPF: DRouter 0.0.0.0"
5Nov 1 11:33:22 172.16.32.2 date=2019-11-01 time=11:33:22 devname=BJ-YZ-CO-FW1 devid=FG5H0E5818903326 logid=0103020301 type=event subtype=router level=information vd=root logdesc="Routing log" msg="OSPF: BDRouter 0.0.0.0"
6Nov 1 11:33:22 172.16.32.2 date=2019-11-01 time=11:33:22 devname=BJ-YZ-CO-FW1 devid=FG5H0E5818903326 logid=0103020301 type=event subtype=router level=information vd=root logdesc="Routing log" msg="OSPF: # Neighbors 1"
7Nov 1 11:33:22 172.16.32.2 date=2019-11-01 time=11:33:22 devname=BJ-YZ-CO-FW1 devid=FG5H0E5818903326 logid=0103020301 type=event subtype=router level=information vd=root logdesc="Routing log" msg="OSPF: Neighbor 172.16.44.18"
8Nov 1 11:33:22 172.16.32.2 date=2019-11-01 time=11:33:22 devname=BJ-YZ-CO-FW1 devid=FG5H0E5818903326 logid=0103020301 type=event subtype=router level=information vd=root logdesc="Routing log" msg="OSPF: NFSM[Vlanif105:172.16.44.18-172.16.46.1]: Full (HelloReceived)"
9Nov 1 11:24:02 2019 BJ-YZ-DS-SW1&2 %%10DRVPLAT/4/DrvDebug: -DevIP=172.16.32.5-Slot=2; Many Parity Errors have been detected in last 10s.
10Nov 1 11:24:02 2019 BJ-YZ-DS-SW1&2 %%10DRVPLAT/4/Log Info: -DevIP=172.16.32.5-Slot=2; Slot 2,unit 0 DLB_HGT_FLOWSET_TIMESTAMP_PAGE_X entry 693 parity error.
11Nov 1 11:24:02 2019 BJ-YZ-DS-SW1&2 %%10DRVPLAT/4/Log Info: -DevIP=172.16.32.5-Slot=2; Slot 2,unit 0 DLB_HGT_FLOWSET_TIMESTAMP_PAGE_X entry 693 parity error.
12Nov 1 11:24:02 2019 BJ-YZ-DS-SW1&2 %%10DRVPLAT/4/Log Info: -DevIP=172.16.32.5-Slot=2; Slot 2,unit 0 DLB_HGT_FLOWSET_TIMESTAMP_PAGE_X entry 693 parity error.
13Nov 1 11:24:02 2019 BJ-YZ-DS-SW1&2 %%10DRVPLAT/4/Log Info: -DevIP=172.16.32.5-Slot=2; Slot 2,unit 0 DLB_HGT_FLOWSET_TIMESTAMP_PAGE_X entry 693 parity error.
很明显日志里面有172.16.32.5和172.16.32.2的信息，去zabbix上看了一下发现是网络设备，但是zabbix显示的是已经挂掉了，但是可以查看，应该是缓存。
先备份下这个日志，再清空释放空间，因为根下已经没空间了，所以压缩到/home下。
1[root@zabbix1 log]# tar zcvf /home/2019-11-1-boot.log.tar.gz boot.log
2[root@zabbix1 log]# cat /dev/null > boot.log
查看zabbix发现server的10051端口已经没有了，只有agent的10050端口
1[root@zabbix1 ~]# netstat -tnlp | grep zabbix
2Active Internet connections (only servers)
3Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
4tcp 0 0 :::10050 :::* LISTEN 15271/zabbix_agentd
重启zabbix-server
1[root@zabbix1 ~]# service zabbix-server restart
2Shutting down Zabbix server: [FAILED]
3Starting Zabbix server: [ OK ]
4[root@zabbix1 ~]# netstat -tnlp | grep zabbix
5Active Internet connections (only servers)
6Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
7tcp 0 0 :::10050 :::* LISTEN 15271/zabbix_agentd
8tcp 0 0 :::10051 :::* LISTEN 1283/zabbix_server
原因
解决办法
1[root@zabbix1 rsyslog.d]# vim /etc/rsyslog.conf
2#local7.* /var/log/boot.log