Back

ubuntu - 解决mysql莫名其妙重启的问题

发布时间: 2020-10-29 00:37:00

昨天和今天发现mysql 莫名其妙的重启了。两个不同项目的数据库服务器都重启了  (ubuntu 16) 

重启的特点是: 莫名其妙的重启,一个是10秒内连续重启5次,一个是6秒内连续重启3次(每次重启2秒钟)

时刻不一样。

过程:

1. 发现一些程序不运行了。 cannot connect to mysql ...

39916 2020-10-29 06:46:30 +0800: Rack app error handling request { POST /api/v2/xxxx }
39917 #
39918 /home/ubuntu/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0/gems/activerecord-6.0.0/lib/active_record/connection_adapters/mysql2_adapter.rb:94:in
39927 /home/ubuntu/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0/gems/activerecord-6.0.0/lib/active_record/dynamic_matchers.rb:66:in `find_by_public_k

2. 看数据库服务器的日志:   /var/log/mysql/  ,发现了mysql 重启。

2020-10-28T22:35:38.166279Z 0 [Note] Giving 287 client threads a chance to die gracefully
2020-10-28T22:35:38.166478Z 0 [Note] Shutting down slave threads
2020-10-28T22:35:48.167779Z 0 [Note] Forcefully disconnecting 168 remaining clients

3. 搜索。关键词: why mysql restart without ... 就会出现: mysql keep restarting ... 等等。其中一个人的提示有了线索: 看看系统日志。

4. 于是查看 /var/log下面的系统日志( syslog .. ) 发现:

Oct 29 06:35:22 xx-mysql-production-japan systemd[1]: Starting Daily apt upgrade and clean activities...
Oct 29 06:35:37 xx-mysql-production-japan systemd[1]: Reloading.
Oct 29 06:35:38 xx-mysql-production-japan systemd[1]: Stopping MySQL Community Server...
Oct 29 06:35:49 xx-mysql-production-japan systemd[1]: Stopped MySQL Community Server.
Oct 29 06:35:49 xx-mysql-production-japan systemd[1]: Reloading.
Oct 29 06:35:50 xx-mysql-production-japan systemd[1]: Starting Message of the Day...
Oct 29 06:35:50 xx-mysql-production-japan systemd[1]: Reloading.

5. 果然,线索在于 Starting Daily apt upgrade and clean activities...

这个 apt upgrade 不是普通的update,  upgrade 会直接引起系统的不稳定, 相关进程或者服务重启。特别麻烦。

参考:  https://www.gongzi.org/disable-daily.html  把它果断关掉。

sudo apt-get remove unattended-upgrades  (选择y)

sudo su  , 然后复制粘贴:

systemctl kill --kill-who=all apt-daily.service
systemctl stop apt-daily.timer
systemctl disable apt-daily.timer
systemctl stop apt-daily.service
systemctl disable apt-daily.service
systemctl mask apt-daily.service
systemctl daemon-reload

最后确认一下:

systemctl status apt-daily.timer

● apt-daily.timer - Daily apt download activities
Loaded: loaded (/lib/systemd/system/apt-daily.timer; disabled; vendor preset: enabled)
Active: inactive (dead) since Thu 2020-10-29 08:32:58 CST; 1min 54s ago
Trigger: n/a

May 19 13:53:39 xx-mysql-production-japan systemd[1]: Started Daily apt download activities.
Oct 29 08:32:58 xx-mysql-production-japan systemd[1]: Stopped Daily apt download activities.

发现下面的字样,就可以了

Active: inactive (dead)

6. 最后复盘: 可以看到, 是 今年5.19 开始的日常更新, 这天我们把ubuntu server进行了重启,所以生效的。

top - 08:47:49 up 162 days, 18:54, 1 user, load average: 4.44, 4.76, 4.81

5.19 具体今天(10.29) 刚好 162天(我没细算),时间对的上。

Back