https://thehftguy.com/2016/10/03/haproxy-vs-nginx-why-you-should-never-use-nginx-for-load-balancing/
Load balancers are the point of entrance to the datacenter. They are on the critical path to access anything and everything.
That give them some interesting characteristics. First, they are the most important thing to monitor in an infrastructure. Second, they are in a unique position to give insights not only about themselves but also about every service that they are backing.
There are two popular open-source software load balancers: HAProxy and nginx. Let’s see how they compare in this regard.
Enable monitoring on the load balancers
The title is self explanatory. It should be systematic for everything going to production.
- Install something new
- Enable stats and monitoring stuff
- Enable logs
Enabling nginx status page
Edit /etc/nginx/nginx.conf:
server { listen 0.0.0.0:6644; access_log off; allow 127.0.0.0/8; allow 10.0.0.0/8; deny all; location / { stub_status on; } }
Enabling HAProxy stats page
Edit /etc/haproxy/haproxy.cfg:
listen stats 0.0.0.0:6427 mode http maxconn 10 no log acl network_allowed src 127.0.0.0/8 acl network_allowed src 10.0.0.0/8 tcp-request connection reject if !network_allowed stats enable stats uri /
Collecting metrics from the load balancer
There are standard monitoring solutions: datadog, signalfx, prometheus, graphite… [2]
These tools gather metrics from applications, servers and infrastructure. They allow to explore the metrics, graph them and send alerts.
Integrating the load balancers into our monitoring system is critical. We need to know about active clients, requests/s, error rate, etc…
Needless to say, the monitoring capabilities will be limited by what information is measured and provided by the load balancer.
[2] Sorted by order of awesomeness. Leftmost is better.
Metrics available from nginx
nginx provide only 7 different metrics.
Nginx only gives the sum, over all sites. It is NOT possible to get any number per site nor per application.
Active connections: The current number of active client connections including Waiting connections. accepts: The total number of accepted client connections. handled: The total number of handled connections. Generally, the parameter value is the same as accepts unless some resource limits have been reached (for example, the worker_connections limit). requests: The total number of client requests. Reading: The current number of connections where nginx is reading the request header. Writing: The current number of connections where nginx is writing the response back to the client. Waiting: The current number of idle client connections waiting for a request.
Source: https://nginx.org/en/docs/http/ngx_http_stub_status_module.html
Metrics available from haproxy
HAProxy provide 61 different metrics.
The numbers are given globally, per frontend and per backend (whichever makes sense). They are available on a human readable web page and in a raw CSV format.
0. pxname [LFBS]: proxy name 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend, any name for server/listener) 2. qcur [..BS]: current queued requests. For the backend this reports the number queued without a server assigned. 3. qmax [..BS]: max value of qcur 4. scur [LFBS]: current sessions 5. smax [LFBS]: max sessions 6. slim [LFBS]: configured session limit 7. stot [LFBS]: cumulative number of connections 8. bin [LFBS]: bytes in 9. bout [LFBS]: bytes out […] 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener) 33. rate [.FBS]: number of sessions per second over last elapsed second 34. rate_lim [.F..]: configured limit on new sessions per second 35. rate_max [.FBS]: max number of new sessions per second 36. check_status […S]: status of last health check, one of: 37. check_code […S]: layer5-7 code, if available 38. check_duration […S]: time in ms took to finish last health check 39. hrsp_1xx [.FBS]: http responses with 1xx code 40. hrsp_2xx [.FBS]: http responses with 2xx code 41. hrsp_3xx [.FBS]: http responses with 3xx code 42. hrsp_4xx [.FBS]: http responses with 4xx code 43. hrsp_5xx [.FBS]: http responses with 5xx code 44. hrsp_other [.FBS]: http responses with other codes (protocol error) […]
Source: http://www.haproxy.org/download/1.5/doc/configuration.txt
Monitoring the load balancer
The aforementioned metrics are used to generate a status on the running systems.
First, we’ll see what kind of status page is provided out-of-the-box by each load balancer. Then we’ll dive into third-party monitoring solutions.
nginx status page
The 7 nginx metrics are displayed on a human readable web page, accessible at 127.0.0.1:6644/
Nginx Status Page
No kidding. This is what nginx considers a “status page“. WTF?!
It doesn’t display what applications are load balanced. It doesn’t display what servers are online (is there anything even running???). There is nothing to see on that page and it won’t help to debug any issue, ever.
HAProxy stats page
For comparison, let’s see the HAProxy monitoring page, accessible at 127.0.0.1:6427
HAProxy Stats Page
Here we can see which servers are up or down, how much bandwidth is used, how many clients are connected and much more. That’s what monitoring is meant to be.
As an experienced sysadmin once told me: “This page is the most important thing in the universe.” [1]
Whenever something goes wonky. First, you open http://www.yoursite.com in a browser to see how bad it’s broken. Second, you open the HAProxy stats page to find what is broken. At this point, you’ve spot the source of the issue 90% of the time.
[0] This is especially true in environments where there is limited monitoring available, or worse, no monitoring tools at all. The status page is always here ready to help (and if it’s not, it’s only a few config lines away).
Integrating nginx with monitoring systems
All we can get are the 7 metrics from the web status page, of which only the requests/s is noteworthy. It’s not exposed in an API friendly format and it’s impossible to get numbers per site. The only hack we can do is parse the raw text, hopping no spacing will change in future versions.
Given that nginx doesn’t expose any useful information, none of the existing monitoring tools can integrate with it. When there is nothing to get, there is nothing to display and nothing to alert on.
Note: Some monitoring tools actually pretend to support nginx integrations. It means that they parse the text and extract the request/s number. That’s all they can get.
Integrating HAProxy with monitoring systems
In additional to the nice human readable monitoring page, all the HAProxy metrics are available in a CSV format. Tools can (and do) take advantage of it.
For instance, this is the default HAProxy dashboard provided by Datadog:
Datadog pre-made dashboard for HAProxy
Source: http://docs.datadoghq.com/integrations/haproxy/
A Datadog agent installed on the host gathers the HAProxy metrics periodically. The metrics can be graphed, the graphs can be arranged into dashboards (this one is an example), and last but not least we can configure automatic alerts.
The HAProxy stats page gives the current status (at the time the page is generated) whereas the monitoring solution saves the history and allows for debugging back in time.
Why does nginx have no monitoring?
All monitoring capabilities are missing from nginx on purpose. They are not and will never be available for free. Period.
If you are already locked-in by nginx and you need a decent monitoring page and a JSON API for integrating, you will have to pay for the “Nginx Plus” edition. The price starts at $1900 per server per year.
See: https://www.nginx.com/products/pricing/
Conclusion: Avoid nginx at all costs
Load balancers are critical points of transit and the single most important things to monitor in an infrastructure.
Nginx stripped all monitoring features for the sake of money, while pretending to be open-source.
Being left entirely blind on our operations is not acceptable. Stay away from nginx. Use HAProxy instead.