架構(gòu)下的日志集中管理實踐與EFK方案)
1. 為什么需要多服務(wù)日志集中管理在微服務(wù)架構(gòu)中一個典型的生產(chǎn)環(huán)境往往包含多個后端服務(wù)和前端應(yīng)用。以我最近部署的一個電商平臺為例整套系統(tǒng)包含3個Node.js后端服務(wù)用戶服務(wù)/訂單服務(wù)/商品服務(wù)2個Vue前端應(yīng)用主站/管理后臺1個Nginx網(wǎng)關(guān)Redis和MySQL各1個實例當所有服務(wù)都通過Docker Compose部署后最頭疼的問題就是日志分散在各個容器中。上周排查一個支付失敗問題時我不得不同時docker logs order-service查看訂單服務(wù)日志docker logs payment-service查看支付服務(wù)日志docker logs nginx查看網(wǎng)關(guān)日志再到Kibana中篩選前端錯誤日志這種碎片化的排查方式效率極低特別是當多個服務(wù)間存在調(diào)用鏈時。更糟的是docker logs默認只保留最新日志歷史數(shù)據(jù)隨時可能丟失。2. 基礎(chǔ)版方案直接掛載日志卷2.1 目錄結(jié)構(gòu)設(shè)計我采用的日志目錄結(jié)構(gòu)如下/project /services /backend - docker-compose.yml /frontend - docker-compose.yml /logs /backend /user-service /order-service /product-service /frontend /main-site /admin-console /infra /nginx /mysql2.2 關(guān)鍵Compose配置以訂單服務(wù)為例的配置片段version: 3.8 services: order-service: image: registry.example.com/order:v1.2 volumes: - /project/logs/backend/order-service:/var/log/app logging: driver: json-file options: max-size: 10m max-file: 32.3 實際踩坑記錄權(quán)限問題首次啟動時出現(xiàn)Permission denied錯誤解決方法mkdir -p /project/logs/backend/order-service chmod -R 777 /project/logs # 生產(chǎn)環(huán)境應(yīng)配置精確權(quán)限日志輪轉(zhuǎn)失效發(fā)現(xiàn)日志文件超過10MB后未切割原因是必須同時設(shè)置max-size和max-file某些應(yīng)用如Nginx需要在其配置中也開啟日志輪轉(zhuǎn)磁盤空間預警建議添加監(jiān)控腳本#!/bin/bash LOG_DIR/project/logs THRESHOLD90 current_usage$(df -h $LOG_DIR | awk NR2 {print $5} | tr -d %) if [ $current_usage -gt $THRESHOLD ]; then echo WARNING: Log directory usage $current_usage% | mail -s Log Alert adminexample.com fi3. 進階版方案EFK全棧日志系統(tǒng)3.1 架構(gòu)設(shè)計graph TD A[Frontend] --|Log| B(Fluentd) C[Backend] --|Log| B D[Database] --|Log| B B -- E[Elasticsearch] E -- F[Kibana]3.2 關(guān)鍵組件配置3.2.1 Fluentd配置fluentd.conf核心部分source type forward port 24224 /source match docker.** type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix docker flush_interval 5s /match3.2.2 Elasticsearch調(diào)優(yōu)docker-compose.efk.yml內(nèi)存設(shè)置services: elasticsearch: image: elasticsearch:7.17.0 environment: - ES_JAVA_OPTS-Xms2g -Xmx2g # 根據(jù)服務(wù)器內(nèi)存調(diào)整 - discovery.typesingle-node ulimits: memlock: soft: -1 hard: -13.3 性能優(yōu)化技巧日志過濾在Fluentd中丟棄DEBUG級別日志filter docker.** type grep exclude key log pattern /DEBUG/ /exclude /filter索引策略每天自動創(chuàng)建新索引match docker.** type elasticsearch logstash_date_format %Y%m%d /match查詢加速為常用字段添加索引PUT /docker-*/_mapping { properties: { container_name: { type: keyword }, service: { type: keyword } } }4. 混合部署實戰(zhàn)案例4.1 場景需求開發(fā)環(huán)境使用基礎(chǔ)版日志卷生產(chǎn)環(huán)境使用EFK全棧方案兩種環(huán)境配置無縫切換4.2 條件化Compose配置x-logging: default-logging driver: ${LOG_DRIVER:-json-file} options: max-size: 10m max-file: 3 x-efk-logging: efk-logging driver: fluentd options: fluentd-address: fluentd:24224 tag: docker.{{.Name}} services: webapp: image: myapp:${TAG:-latest} logging: : *${LOG_CONFIG:-default-logging}啟動命令示例# 開發(fā)環(huán)境 LOG_DRIVERjson-file docker-compose up # 生產(chǎn)環(huán)境 LOG_CONFIGefk-logging TAGprod docker-compose -f docker-compose.yml -f docker-compose.efk.yml up4.3 監(jiān)控指標設(shè)計Grafana儀表盤建議包含日志量趨勢按服務(wù)/級別錯誤日志TOP 10日志存儲增長預測關(guān)鍵服務(wù)響應(yīng)時間關(guān)聯(lián)分析5. 異常處理手冊5.1 Fluentd常見問題癥狀日志未出現(xiàn)在Kibana中 排查步驟檢查Fluentd容器狀態(tài)docker-compose logs fluentd | grep -i error驗證Elasticsearch連接curl -XGET http://elasticsearch:9200/_cat/indices?v查看緩沖隊列docker exec fluentd ls -lh /var/log/fluentd/buffer5.2 日志丟失預防啟用本地備份雙寫策略match docker.** type copy store type elasticsearch ... /store store type file path /var/log/fluentd/backup /store /match設(shè)置磁盤水位警戒線docker run -it --rm --privileged --pidhost debian nsenter -t 1 -m -u -n -i sh -c echo 90 /proc/sys/vm/dirty_ratio6. 安全加固方案6.1 訪問控制Kibana基礎(chǔ)認證services: kibana: environment: - ELASTICSEARCH_USERNAMEadmin - ELASTICSEARCH_PASSWORD${ES_PASSWORD}Elasticsearch TLS加密# 生成證書 docker run --rm elasticsearch:7.17.0 \ bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass # 掛載到容器 volumes: - ./certs/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p126.2 日志脫敏Fluentd過濾器示例filter docker.** type record_transformer enable_ruby true record message ${record[message].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, [CREDIT_CARD])} /record /filter7. 成本優(yōu)化實踐7.1 冷熱數(shù)據(jù)分離PUT _ilm/policy/logs_policy { policy: { phases: { hot: { actions: { rollover: { max_size: 50GB, max_age: 7d } } }, warm: { min_age: 7d, actions: { allocate: { number_of_replicas: 1 } } } } } }7.2 日志采樣配置對DEBUG日志進行50%采樣filter docker.** type sample rate 50 sample_key level sample_value DEBUG /filter 微服務(wù)架構(gòu)下的日志集中管理實踐與EFK方案
1. 為什么需要多服務(wù)日志集中管理在微服務(wù)架構(gòu)中一個典型的生產(chǎn)環(huán)境往往包含多個后端服務(wù)和前端應(yīng)用。以我最近部署的一個電商平臺為例整套系統(tǒng)包含3個Node.js后端服務(wù)用戶服務(wù)/訂單服務(wù)/商品服務(wù)2個Vue前端應(yīng)用主站/管理后臺1個Nginx網(wǎng)關(guān)Redis和MySQL各1個實例當所有服務(wù)都通過Docker Compose部署后最頭疼的問題就是日志分散在各個容器中。上周排查一個支付失敗問題時我不得不同時docker logs order-service查看訂單服務(wù)日志docker logs payment-service查看支付服務(wù)日志docker logs nginx查看網(wǎng)關(guān)日志再到Kibana中篩選前端錯誤日志這種碎片化的排查方式效率極低特別是當多個服務(wù)間存在調(diào)用鏈時。更糟的是docker logs默認只保留最新日志歷史數(shù)據(jù)隨時可能丟失。2. 基礎(chǔ)版方案直接掛載日志卷2.1 目錄結(jié)構(gòu)設(shè)計我采用的日志目錄結(jié)構(gòu)如下/project /services /backend - docker-compose.yml /frontend - docker-compose.yml /logs /backend /user-service /order-service /product-service /frontend /main-site /admin-console /infra /nginx /mysql2.2 關(guān)鍵Compose配置以訂單服務(wù)為例的配置片段version: 3.8 services: order-service: image: registry.example.com/order:v1.2 volumes: - /project/logs/backend/order-service:/var/log/app logging: driver: json-file options: max-size: 10m max-file: 32.3 實際踩坑記錄權(quán)限問題首次啟動時出現(xiàn)Permission denied錯誤解決方法mkdir -p /project/logs/backend/order-service chmod -R 777 /project/logs # 生產(chǎn)環(huán)境應(yīng)配置精確權(quán)限日志輪轉(zhuǎn)失效發(fā)現(xiàn)日志文件超過10MB后未切割原因是必須同時設(shè)置max-size和max-file某些應(yīng)用如Nginx需要在其配置中也開啟日志輪轉(zhuǎn)磁盤空間預警建議添加監(jiān)控腳本#!/bin/bash LOG_DIR/project/logs THRESHOLD90 current_usage$(df -h $LOG_DIR | awk NR2 {print $5} | tr -d %) if [ $current_usage -gt $THRESHOLD ]; then echo WARNING: Log directory usage $current_usage% | mail -s Log Alert adminexample.com fi3. 進階版方案EFK全棧日志系統(tǒng)3.1 架構(gòu)設(shè)計graph TD A[Frontend] --|Log| B(Fluentd) C[Backend] --|Log| B D[Database] --|Log| B B -- E[Elasticsearch] E -- F[Kibana]3.2 關(guān)鍵組件配置3.2.1 Fluentd配置fluentd.conf核心部分source type forward port 24224 /source match docker.** type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix docker flush_interval 5s /match3.2.2 Elasticsearch調(diào)優(yōu)docker-compose.efk.yml內(nèi)存設(shè)置services: elasticsearch: image: elasticsearch:7.17.0 environment: - ES_JAVA_OPTS-Xms2g -Xmx2g # 根據(jù)服務(wù)器內(nèi)存調(diào)整 - discovery.typesingle-node ulimits: memlock: soft: -1 hard: -13.3 性能優(yōu)化技巧日志過濾在Fluentd中丟棄DEBUG級別日志filter docker.** type grep exclude key log pattern /DEBUG/ /exclude /filter索引策略每天自動創(chuàng)建新索引match docker.** type elasticsearch logstash_date_format %Y%m%d /match查詢加速為常用字段添加索引PUT /docker-*/_mapping { properties: { container_name: { type: keyword }, service: { type: keyword } } }4. 混合部署實戰(zhàn)案例4.1 場景需求開發(fā)環(huán)境使用基礎(chǔ)版日志卷生產(chǎn)環(huán)境使用EFK全棧方案兩種環(huán)境配置無縫切換4.2 條件化Compose配置x-logging: default-logging driver: ${LOG_DRIVER:-json-file} options: max-size: 10m max-file: 3 x-efk-logging: efk-logging driver: fluentd options: fluentd-address: fluentd:24224 tag: docker.{{.Name}} services: webapp: image: myapp:${TAG:-latest} logging: : *${LOG_CONFIG:-default-logging}啟動命令示例# 開發(fā)環(huán)境 LOG_DRIVERjson-file docker-compose up # 生產(chǎn)環(huán)境 LOG_CONFIGefk-logging TAGprod docker-compose -f docker-compose.yml -f docker-compose.efk.yml up4.3 監(jiān)控指標設(shè)計Grafana儀表盤建議包含日志量趨勢按服務(wù)/級別錯誤日志TOP 10日志存儲增長預測關(guān)鍵服務(wù)響應(yīng)時間關(guān)聯(lián)分析5. 異常處理手冊5.1 Fluentd常見問題癥狀日志未出現(xiàn)在Kibana中 排查步驟檢查Fluentd容器狀態(tài)docker-compose logs fluentd | grep -i error驗證Elasticsearch連接curl -XGET http://elasticsearch:9200/_cat/indices?v查看緩沖隊列docker exec fluentd ls -lh /var/log/fluentd/buffer5.2 日志丟失預防啟用本地備份雙寫策略match docker.** type copy store type elasticsearch ... /store store type file path /var/log/fluentd/backup /store /match設(shè)置磁盤水位警戒線docker run -it --rm --privileged --pidhost debian nsenter -t 1 -m -u -n -i sh -c echo 90 /proc/sys/vm/dirty_ratio6. 安全加固方案6.1 訪問控制Kibana基礎(chǔ)認證services: kibana: environment: - ELASTICSEARCH_USERNAMEadmin - ELASTICSEARCH_PASSWORD${ES_PASSWORD}Elasticsearch TLS加密# 生成證書 docker run --rm elasticsearch:7.17.0 \ bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass # 掛載到容器 volumes: - ./certs/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p126.2 日志脫敏Fluentd過濾器示例filter docker.** type record_transformer enable_ruby true record message ${record[message].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, [CREDIT_CARD])} /record /filter7. 成本優(yōu)化實踐7.1 冷熱數(shù)據(jù)分離PUT _ilm/policy/logs_policy { policy: { phases: { hot: { actions: { rollover: { max_size: 50GB, max_age: 7d } } }, warm: { min_age: 7d, actions: { allocate: { number_of_replicas: 1 } } } } } }7.2 日志采樣配置對DEBUG日志進行50%采樣filter docker.** type sample rate 50 sample_key level sample_value DEBUG /filter