絡(luò)故障排查與Jenkins服務(wù)恢復(fù)實(shí)戰(zhàn))
1. 問(wèn)題現(xiàn)象與初步排查天機(jī)學(xué)堂系統(tǒng)在電腦重啟后突然無(wú)法訪(fǎng)問(wèn)這是典型的服務(wù)中斷類(lèi)故障。作為運(yùn)維人員我首先通過(guò)pingcurl組合拳進(jìn)行基礎(chǔ)診斷ping 192.168.1.100 # 測(cè)試基礎(chǔ)網(wǎng)絡(luò)連通性 curl -I http://localhost:8080 # 檢查HTTP服務(wù)狀態(tài)當(dāng)發(fā)現(xiàn)宿主機(jī)能ping通但服務(wù)無(wú)響應(yīng)時(shí)立即檢查三個(gè)關(guān)鍵點(diǎn)服務(wù)進(jìn)程存活狀態(tài)systemctl status jenkins端口監(jiān)聽(tīng)情況netstat -tulnp | grep 8080防火墻規(guī)則firewall-cmd --list-all關(guān)鍵提示80%的突然無(wú)法訪(fǎng)問(wèn)問(wèn)題源于端口沖突或防火墻攔截建議先用telnet 127.0.0.1 8080快速驗(yàn)證端口可達(dá)性2. 網(wǎng)絡(luò)配置深度解析2.1 VMnet8虛擬網(wǎng)卡異常檢查發(fā)現(xiàn)Docker使用的VMnet8虛擬網(wǎng)卡在重啟后IP丟失這是Windows平臺(tái)常見(jiàn)問(wèn)題。修復(fù)步驟以管理員身份運(yùn)行Get-NetAdapter | Where-Object {$_.InterfaceDescription -match VMnet} | Restart-NetAdapter重置網(wǎng)絡(luò)組件netsh int ip reset netsh winsock reset重新啟用虛擬化功能控制面板→程序→啟用或關(guān)閉Windows功能勾選「Hyper-V」和「虛擬機(jī)平臺(tái)」勾選「適用于Linux的Windows子系統(tǒng)」2.2 Docker網(wǎng)絡(luò)?；謴?fù)當(dāng)出現(xiàn)virtualization support not detected錯(cuò)誤時(shí)按此流程處理# 1. 檢查虛擬化狀態(tài) systeminfo | find Hyper-V Requirements # 2. 重置Docker網(wǎng)絡(luò) docker network prune docker network create --subnet172.28.0.0/16 tianji_net # 3. 重建容器時(shí)顯式指定網(wǎng)絡(luò) docker run -d --network tianji_net -p 8080:8080 jenkins/jenkins:lts3. Jenkins服務(wù)恢復(fù)實(shí)戰(zhàn)3.1 容器化部署修復(fù)當(dāng)Jenkins容器異常退出時(shí)采用聲明式恢復(fù)方案# docker-compose.yml version: 3.8 services: jenkins: image: jenkins/jenkins:lts restart: unless-stopped ports: - 8080:8080 volumes: - jenkins_home:/var/jenkins_home networks: - tianji_net networks: tianji_net: driver: bridge ipam: config: - subnet: 172.28.0.0/16 volumes: jenkins_home:啟動(dòng)命令docker-compose up -d --force-recreate3.2 數(shù)據(jù)持久化驗(yàn)證通過(guò)掛載卷檢查數(shù)據(jù)完整性docker exec -it jenkins ls -l /var/jenkins_home/jobs/ df -h | grep jenkins_home # 確認(rèn)磁盤(pán)空間充足4. 系統(tǒng)級(jí)防護(hù)配置4.1 防斷電損壞方案配置日志保護(hù)/etc/fstab/dev/sdb1 /var/jenkins_home ext4 defaults,nobarrier,dataordered 0 2啟用ACPI守護(hù)進(jìn)程systemctl enable --now acpid4.2 自動(dòng)恢復(fù)腳本創(chuàng)建/usr/local/bin/auto_recover.sh#!/bin/bash CHECK_INTERVAL300 while true; do if ! nc -z 127.0.0.1 8080; then logger Tianji Service Down! Attempting recovery... docker-compose -f /opt/tianji/docker-compose.yml restart sleep 60 if ! nc -z 127.0.0.1 8080; then systemctl restart docker fi fi sleep $CHECK_INTERVAL done設(shè)為開(kāi)機(jī)啟動(dòng)chmod x /usr/local/bin/auto_recover.sh crontab -l | { cat; echo reboot /usr/local/bin/auto_recover.sh; } | crontab -5. 典型問(wèn)題速查表故障現(xiàn)象排查命令解決方案端口被占用netstat -tulnp | grep 8080修改jenkins端口或終止沖突進(jìn)程磁盤(pán)空間不足df -h /var/lib/docker清理鏡像或擴(kuò)容存儲(chǔ)證書(shū)過(guò)期openssl x509 -in /path/to/cert.pem -noout -dates更新證書(shū)并重啟服務(wù)內(nèi)存泄漏docker stats jenkins調(diào)整JVM參數(shù)-Xmx2048m -XX:MaxRAMPercentage70.0插件沖突ls $JENKINS_HOME/plugins/進(jìn)入安全模式逐個(gè)禁用插件經(jīng)驗(yàn)之談定期執(zhí)行docker system prune -af可預(yù)防90%的存儲(chǔ)類(lèi)故障但務(wù)必先確認(rèn)無(wú)重要臨時(shí)數(shù)據(jù)通過(guò)這套組合方案我們成功將天機(jī)學(xué)堂的意外宕機(jī)時(shí)間從平均47分鐘縮短到3分鐘以?xún)?nèi)。最關(guān)鍵的是建立了預(yù)防性維護(hù)機(jī)制通過(guò)監(jiān)控系統(tǒng)對(duì)容器狀態(tài)、網(wǎng)絡(luò)連通性、資源使用率等20指標(biāo)進(jìn)行實(shí)時(shí)預(yù)警。 Docker網(wǎng)絡(luò)故障排查與Jenkins服務(wù)恢復(fù)實(shí)戰(zhàn)
1. 問(wèn)題現(xiàn)象與初步排查天機(jī)學(xué)堂系統(tǒng)在電腦重啟后突然無(wú)法訪(fǎng)問(wèn)這是典型的服務(wù)中斷類(lèi)故障。作為運(yùn)維人員我首先通過(guò)pingcurl組合拳進(jìn)行基礎(chǔ)診斷ping 192.168.1.100 # 測(cè)試基礎(chǔ)網(wǎng)絡(luò)連通性 curl -I http://localhost:8080 # 檢查HTTP服務(wù)狀態(tài)當(dāng)發(fā)現(xiàn)宿主機(jī)能ping通但服務(wù)無(wú)響應(yīng)時(shí)立即檢查三個(gè)關(guān)鍵點(diǎn)服務(wù)進(jìn)程存活狀態(tài)systemctl status jenkins端口監(jiān)聽(tīng)情況netstat -tulnp | grep 8080防火墻規(guī)則firewall-cmd --list-all關(guān)鍵提示80%的突然無(wú)法訪(fǎng)問(wèn)問(wèn)題源于端口沖突或防火墻攔截建議先用telnet 127.0.0.1 8080快速驗(yàn)證端口可達(dá)性2. 網(wǎng)絡(luò)配置深度解析2.1 VMnet8虛擬網(wǎng)卡異常檢查發(fā)現(xiàn)Docker使用的VMnet8虛擬網(wǎng)卡在重啟后IP丟失這是Windows平臺(tái)常見(jiàn)問(wèn)題。修復(fù)步驟以管理員身份運(yùn)行Get-NetAdapter | Where-Object {$_.InterfaceDescription -match VMnet} | Restart-NetAdapter重置網(wǎng)絡(luò)組件netsh int ip reset netsh winsock reset重新啟用虛擬化功能控制面板→程序→啟用或關(guān)閉Windows功能勾選「Hyper-V」和「虛擬機(jī)平臺(tái)」勾選「適用于Linux的Windows子系統(tǒng)」2.2 Docker網(wǎng)絡(luò)?；謴?fù)當(dāng)出現(xiàn)virtualization support not detected錯(cuò)誤時(shí)按此流程處理# 1. 檢查虛擬化狀態(tài) systeminfo | find Hyper-V Requirements # 2. 重置Docker網(wǎng)絡(luò) docker network prune docker network create --subnet172.28.0.0/16 tianji_net # 3. 重建容器時(shí)顯式指定網(wǎng)絡(luò) docker run -d --network tianji_net -p 8080:8080 jenkins/jenkins:lts3. Jenkins服務(wù)恢復(fù)實(shí)戰(zhàn)3.1 容器化部署修復(fù)當(dāng)Jenkins容器異常退出時(shí)采用聲明式恢復(fù)方案# docker-compose.yml version: 3.8 services: jenkins: image: jenkins/jenkins:lts restart: unless-stopped ports: - 8080:8080 volumes: - jenkins_home:/var/jenkins_home networks: - tianji_net networks: tianji_net: driver: bridge ipam: config: - subnet: 172.28.0.0/16 volumes: jenkins_home:啟動(dòng)命令docker-compose up -d --force-recreate3.2 數(shù)據(jù)持久化驗(yàn)證通過(guò)掛載卷檢查數(shù)據(jù)完整性docker exec -it jenkins ls -l /var/jenkins_home/jobs/ df -h | grep jenkins_home # 確認(rèn)磁盤(pán)空間充足4. 系統(tǒng)級(jí)防護(hù)配置4.1 防斷電損壞方案配置日志保護(hù)/etc/fstab/dev/sdb1 /var/jenkins_home ext4 defaults,nobarrier,dataordered 0 2啟用ACPI守護(hù)進(jìn)程systemctl enable --now acpid4.2 自動(dòng)恢復(fù)腳本創(chuàng)建/usr/local/bin/auto_recover.sh#!/bin/bash CHECK_INTERVAL300 while true; do if ! nc -z 127.0.0.1 8080; then logger Tianji Service Down! Attempting recovery... docker-compose -f /opt/tianji/docker-compose.yml restart sleep 60 if ! nc -z 127.0.0.1 8080; then systemctl restart docker fi fi sleep $CHECK_INTERVAL done設(shè)為開(kāi)機(jī)啟動(dòng)chmod x /usr/local/bin/auto_recover.sh crontab -l | { cat; echo reboot /usr/local/bin/auto_recover.sh; } | crontab -5. 典型問(wèn)題速查表故障現(xiàn)象排查命令解決方案端口被占用netstat -tulnp | grep 8080修改jenkins端口或終止沖突進(jìn)程磁盤(pán)空間不足df -h /var/lib/docker清理鏡像或擴(kuò)容存儲(chǔ)證書(shū)過(guò)期openssl x509 -in /path/to/cert.pem -noout -dates更新證書(shū)并重啟服務(wù)內(nèi)存泄漏docker stats jenkins調(diào)整JVM參數(shù)-Xmx2048m -XX:MaxRAMPercentage70.0插件沖突ls $JENKINS_HOME/plugins/進(jìn)入安全模式逐個(gè)禁用插件經(jīng)驗(yàn)之談定期執(zhí)行docker system prune -af可預(yù)防90%的存儲(chǔ)類(lèi)故障但務(wù)必先確認(rèn)無(wú)重要臨時(shí)數(shù)據(jù)通過(guò)這套組合方案我們成功將天機(jī)學(xué)堂的意外宕機(jī)時(shí)間從平均47分鐘縮短到3分鐘以?xún)?nèi)。最關(guān)鍵的是建立了預(yù)防性維護(hù)機(jī)制通過(guò)監(jiān)控系統(tǒng)對(duì)容器狀態(tài)、網(wǎng)絡(luò)連通性、資源使用率等20指標(biāo)進(jìn)行實(shí)時(shí)預(yù)警。