很多人都知道nginx可以做反向代理和負(fù)載均衡,但是關(guān)于nginx的健康檢查(health_check)機(jī)制了解的不多。其實(shí)社區(qū)版nginx提供的health_check機(jī)制其實(shí)很薄弱,主要是通過(guò)在upstream中配置max_fails和fail_timeout來(lái)實(shí)現(xiàn),這邊文章主要是深入分析社區(qū)版的health_check機(jī)制,當(dāng)然還有更好的一些建議,比如商業(yè)版的nginx plus或者阿里的tengine,他們包含的健康檢查機(jī)制更加完善和高效,如果你堅(jiān)持使用nginx社區(qū)版,當(dāng)然還可以自己寫(xiě)或者找第三方模塊來(lái)編譯了。
首先說(shuō)下我的測(cè)試環(huán)境,CentOS release 6.4 (Final) + nginx_1.6.0 + 2臺(tái)tomcat8.0.15作為后端服務(wù)器。(聲明:以下所有配置僅僅為測(cè)試所用,不代表線上環(huán)境真實(shí)所用,真正的線上環(huán)境需要更多配置和優(yōu)化。)
nginx配置如下:
worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log logs/access.log main; sendfile on; keepalive_timeout 65; upstream backend { server localhost:9090 max_fails=1 fail_timeout=40s; server localhost:9191 max_fails=1 fail_timeout=40s; } server { listen 80; server_name localhost; location / { proxy_pass http://backend; proxy_connect_timeout 1; proxy_read_timeout 1; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } }
}
關(guān)于nginx和tomcat的配置的基本配置不再說(shuō)明,大家可以去看官方文檔。
我們可以看到我在upstream 指令中配置了兩臺(tái)server,每臺(tái)server都設(shè)置了max_fails和fail_timeout值。
現(xiàn)在開(kāi)始啟動(dòng)nginx,然后啟動(dòng)后臺(tái)的2臺(tái)server, 故意把在Tomcat Listener中Sleep 10分鐘,也就是tomcat啟動(dòng)要花費(fèi)10分鐘左右,端口已開(kāi),但是沒(méi)有接收請(qǐng)求,然后我們?cè)L問(wèn)http://localhost/response/ (response這個(gè)接口是我在tomcat中寫(xiě)的一個(gè)servlet接口,功能很簡(jiǎn)單,如果是9090的server接收請(qǐng)求則返回9090,如果是9191端口的server則返回9191.),現(xiàn)在觀察nginx的表現(xiàn)。
我們查看nginx中
access.log
192.168.42.254 - - [29/Dec/2014:11:24:23 +0800] "GET /response/ HTTP/1.1" 504 537 720 380 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36" 2.004 host:health.iflytek.com 192.168.42.254 - - [29/Dec/2014:11:24:24 +0800] "GET /favicon.ico HTTP/1.1" 502 537 715 311 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36" 0.000 host:health.iflytek.com
error.log
2014/12/29 11:24:22 [error] 6318#0: *4785892017 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.42.254, server: health.iflytek.com, request: "GET /response/ HTTP/1.1", upstream: "http://192.168.42.249:9090/response/", host: "health.iflytek.com" 2014/12/29 11:24:23 [error] 6318#0: *4785892017 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.42.254, server: health.iflytek.com, request: "GET /response/ HTTP/1.1", upstream: "http://192.168.42.249:9191/response/", host: "health.iflytek.com" 2014/12/29 11:24:24 [error] 6318#0: *4785892017 no live upstreams while connecting to upstream, client: 192.168.42.254, server: health.iflytek.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://health/favicon.ico", host: "health.iflytek.com"
(為什么要在listener中設(shè)置睡眠10分鐘,這是因?yàn)槲覀兊臉I(yè)務(wù)中需要做緩存預(yù)熱,所以這10分鐘就是模擬服務(wù)器啟動(dòng)過(guò)程中有10分鐘的不可用。)
觀察日志發(fā)現(xiàn)在兩臺(tái)tomcat啟動(dòng)過(guò)程中,發(fā)送一次請(qǐng)求,nginx會(huì)自動(dòng)幫我們進(jìn)行重試所有的后端服務(wù)器,最后會(huì)報(bào) no live upstreams while connecting to upstream錯(cuò)誤。這也算是nginx做health_check的一種方式。這里需要特別強(qiáng)調(diào)一點(diǎn),我們?cè)O(shè)置了proxy_read_timeout 為 1秒。后面再重點(diǎn)講解這個(gè)參數(shù),很重要。
等待40s,現(xiàn)在把9090這臺(tái)服務(wù)器啟動(dòng)完成,但是9191這臺(tái)服務(wù)器仍然是正在啟動(dòng),觀察nginx日志表現(xiàn)。
access.log
192.168.42.254 - - [29/Dec/2014:11:54:18 +0800] "GET /response/ HTTP/1.1" 200 19 194 423 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36" 0.210 host:health.iflytek.com 192.168.42.254 - - [29/Dec/2014:11:54:18 +0800] "GET /favicon.ico HTTP/1.1" 404 453 674 311 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36" 0.212 host:health.iflytek.com
error.log
沒(méi)有打印任何錯(cuò)誤
瀏覽器返回9090,說(shuō)明nginx正常接收請(qǐng)求。
我們?cè)俅握?qǐng)求一次。
access.log
192.168.42.254 - - [29/Dec/2014:13:43:13 +0800] "GET /response/ HTTP/1.1" 200 19 194 423 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36" 1.005 host:health.iflytek.com
說(shuō)明正常返回,同時(shí)返回9090
error.log
2014/12/29 13:43:13 [error] 6323#0: *4801368618 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.42.254, server: health.iflytek.com, request: "GET /response/ HTTP/1.1", upstream: "http://192.168.42.249:9191/response/", host: "health.iflytek.com"
發(fā)現(xiàn)nginx error.log 增加了一行upstream time out的錯(cuò)誤。但是客戶端仍然正常返回,upstream默認(rèn)是輪訓(xùn)的負(fù)載,所以這個(gè)請(qǐng)求默認(rèn)會(huì)轉(zhuǎn)發(fā)到9191這臺(tái)機(jī)器,但是因?yàn)?191正在啟動(dòng),所以這次請(qǐng)求失敗,然后有nginx重試轉(zhuǎn)發(fā)到9090機(jī)器上面。
OK,但是fail_timeout=40s是什么意思呢?我們要不要重現(xiàn)一下這個(gè)參數(shù)的重要性?Let's go !
現(xiàn)在你只需要靜靜的做個(gè)美男子,等待9191機(jī)器啟動(dòng)完畢!多發(fā)送幾次請(qǐng)求!然后咦,你發(fā)現(xiàn)9191機(jī)器返回9191響應(yīng)了噢!fail_timeout=40s其實(shí)就是如果上次請(qǐng)求發(fā)現(xiàn)9191無(wú)法正常返回,那么有40s的時(shí)間該server會(huì)不可用,但是一旦超過(guò)40s請(qǐng)求也會(huì)再次轉(zhuǎn)發(fā)到該server上的,不管該server到底有沒(méi)有真正的恢復(fù)。所以可見(jiàn)nginx社區(qū)版的health_check機(jī)制有多么的薄弱啊,也就是一個(gè)延時(shí)屏蔽而已,如此周而復(fù)始!如果你用過(guò)nginx plus其實(shí)你會(huì)發(fā)現(xiàn)nginx plus 提供的health_check機(jī)制更加強(qiáng)大,說(shuō)幾個(gè)關(guān)鍵詞,你們自己去查! zone slow_start health_check match ! 這個(gè)slow_start其實(shí)就很好的解決了緩存預(yù)熱的問(wèn)題,比如nginx發(fā)現(xiàn)一臺(tái)機(jī)器重啟了,那么會(huì)等待slow_starts設(shè)定的時(shí)間才會(huì)再次發(fā)送請(qǐng)求到該服務(wù)器上,這就給緩存預(yù)熱提供了時(shí)間。