jenkins 磁盘满问题处理

前言

jenkins 有 master 跟 slave，运行久了这些节点都有可能磁盘满了。

1. 查看系统总体存储情况

[root@ecs-bj4-eccjenkinsslave-androidcicd devops]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G   12K  7.8G   1% /dev/shm
tmpfs           7.8G   68M  7.7G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        69G   45G   22G  78% /
tmpfs           1.6G     0  1.6G   0% /run/user/0
tmpfs           1.6G     0  1.6G   0% /run/user/1001

2. 从根目录开始一层层查找哪个目录占用的磁盘异常

通过层层排查，我们来到了 /var/lib/jenkins/jobs/ 目录，这个目录一看就是 jenkins job 构建产物的目录

[root@ecs-bj4-eccjenkinsslave-androidcicd jobs]# du -h --max-depth=1
800K ./Arv_Datalink_Manage_Test2
16M ./OTP_Runner
40M ./Arv_Koala_Admin_Dev
1.5M ./Arv_Datalink_Work_Test
972K ./Arv_SOUL_Test1
55M ./iOS_JoymoClient_Test
23M ./Arv_Koala_Server_Test
812K ./Arv_Datalink_Work_Pre
71M ./IOT_Android_Pos_Test
156M ./IOT_Android_AdScreen_Test
812K ./Arv_Datalink_Manage_Prod
86M ./Android_joymolauncher_Test
896K ./Arv_Datax_Test2
2.8M ./Arv_Datax_Pre
24K ./OTP_RUN_BEE_PROD
888K ./Arv_Datax_Prod
812K ./Arv_Datalink_Work_Prod
216M ./Store_UniClient
4.4M ./PrintInfo
43M ./iOS_FlutterClient_Test
119M ./Arv_SOUL_Prod
16M ./TempTest
408K ./demo-ios
26M ./OTP_Runner_Dev
55M ./Arv_Datalink_Manage_Dev
53M ./iOS_YummyClient_Test
26M ./OTP_Runner_Debug
20K ./xxx
7.7G ./OTP_Runner_Test
18M ./Android_joymooperation_Test
3.9M ./SpringbootBuild_allEnv
1.4G ./Store_RosApp
5.5M ./Arv_Datax_Dev
14M ./Store_AppPackaging
6.1M ./Arv_Koala_Server_Prod
1.4G ./OTP_Runner_Pre
32K ./Test_Arv_allTemplat
384M ./Android_yummyclient_Test
122M ./Arv_SOUL_Dev
16M ./Arv_Datalink_Work_Dev
66M ./Arv_Koala_Server_Dev
896K ./Arv_Datax_Test
808K ./Arv_Datalink_Manage_Pre
7.9M ./Arv_Koala_Server_Pre
56M ./MacOs_ProjectBuild_Test
804K ./Arv_Datalink_Work_Test2
28M ./Arv_SOUL_Pre
77M ./Test_Android_joymoclient_Test2
992K ./Arv_SOUL_Test2
7.1M ./Arv_Kafka_Connect_Mirror_Test
836K ./Arv_Datalink_Manage_Test1
36M ./iOS_JoymoOperation_Test
135M ./Android_joymoclient_Test
492M ./IOT_Android_GatewayApp_Test
452K ./SpringProject_Ecs
13G .

3. 通过上面查找出占用比较多的 job，到 jenkins 控制台查看配置

在 job 的配置里面有个丢弃策略，这边可以根据自己的情况调小点。

扩容

扩容的话需要调整相对于的工作目录

master

配置文件路径：/etc/sysconfig,调整下 JENKINS_HOME 参数

## Path:        Development/Jenkins
## Description: Jenkins Automation Server
## Type:        string
## Default:     "/var/lib/jenkins"
## ServiceRestart: jenkins
#
# Directory where Jenkins store its configuration and working
# files (checkouts, build reports, artifacts, ...).
#
JENKINS_HOME="/var/lib/jenkins"

## Type:        string
## Default:     ""
## ServiceRestart: jenkins
#
# Java executable to run Jenkins
# When left empty, we'll try to find the suitable Java.
#

调整完重启下服务

service jenkins restart

slave

新加了一块盘挂载到 /workspace 目录，相对于的 slave 的工作目录需要调整。

[root@ecs-test-jenkins-slave jenkinsHome]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  137M  7.7G   2% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1       148G   42G  100G  30% /
tmpfs           1.6G     0  1.6G   0% /run/user/0
/dev/vdb1        50G   13G   35G  26% /workspace
tmpfs           1.6G     0  1.6G   0% /run/user/1001
tmpfs           1.6G     0  1.6G   0% /run/user/1000