I’m exaggerating, but only a little. If you run Zabbix and have never looked at the volume of DNS requests it makes every day, take a look. You might be surprised.
DNS = Do Not Store?
Zabbix is a great, open-source monitoring tool. It runs on Linux. It talks to an agent on your servers, or SMTP, or pings them, or runs web requests against them. But Zabbix will not remember their IP addresses.
Unless you enter IP addresses for all your host interfaces, Zabbix needs to look up their names in DNS. It does not cache the host IP, even for a little while, so it will perform a DNS lookup every time it needs to communicate with each host. And Linux, unlike Windows, does not cache DNS replies out of the box, so the request must go out to a resolver every time.
I have a small home environment that I monitor with about 60 hosts. My DNS servers were handling millions of requests per day, and more than 95% were from the Zabbix server. Over a five-minute period, my Zabbix server sent or received 47,898 DNS packets. These were mainly requests to tell Zabbix the same IPs over and over again.
If you use names instead of IPs for your monitored hosts in Zabbix, your DNS servers are being hammered, too.
This is not really a criticism of Zabbix, by the way. Zabbix is just doing its job the way I told it to do. (It could store those addresses though, at least for a little while. After all, our host IPs do not change that often — if ever — and there is no need for such profligate protocol abuse.) Alas, there is hope. Here are a couple of ways to reduce the stress on our DNS servers.
Use IPs
One way to solve the problem is to use IP addresses instead of domains when configuring the host. For example, instead of this:
we could do this:
That obviates the need for address resolution, so no DNS lookups for this host.
This is the easiest way to eliminate the torrent of DNS traffic that is Zabbix Server. If, like me, you dislike using IPs instead of domain names — because that is why we have domain names, right? — there is another option.
Use dnsmasq
The “lightweight DNS forwarder” dnsmasq lets us provide DNS caching on the Zabbix server. (It is available at its official repository, but is most likely also available from your distribution’s repository.) That way, Zabbix (and anything else resolving names on the Zabbix host) can query the local dnsmasq service for an address. If dnsmasq resolved the address via a real DNS server recently (in the interval before the record expires), dnsmasq can respond immediately from its own cache1This assumes all the records fit into its cache, which is 150 entries by default. instead of putting packets on the network.
For this to work, you must (1) install dnsmasq, (2) configure it to use your DNS servers and bind to the local loopback address, and then (3) change the host’s DNS configuration to use the local loopback address instead of the DNS servers it would normally use. Of course, to be safe you should back up any files you change or make a snapshot of your VM if that is your platform before you start any of this.
I have installed the Zabbix 6.0 appliance that runs on CentOS Stream release 8 and my procedure is based on that distribution. If yours is different, you will have to adapt these instructions to your Linux.
Install dnsmasq
This should be the easy part.
$ sudo dnf install dnsmasq -y
Configure dnsmasq
The configuration file is /etc/dnsmasq.conf. Edit it as follows, using your DNS server addresses instead.
# Set your list of DNS servers server=10.7.0.53 server=10.7.1.53 server=10.7.2.53 # Bind only the local loopback address listen-address=127.0.0.1 bind-interfaces # Prevent dnsmasq from reading any files (like resolv.conf) to get DNS servers no-resolv no-poll # Other settings automatically set on my distro port=53 user=dnsmasq group=dnsmasq conf-dir=/etc/dnsmasq.d,.rpmnew,.rpmsave,.rpmorig
Enable and start dnsmasq.
$ sudo systemctl enable --now dnsmasq
You can use nslookup to check if dnsmasq is working. (If nslookup is unavailable, you must install the bind-utils package.) The first argument is the name you want to resolve and the second is the DNS server IP address.
$ nslookup www.centos.org 127.0.0.1 Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: Name: www.centos.org Address: 81.171.33.201 ...
Your output should be comparable if dnsmasq is working. It means nslookup is asking the local dnsmasq daemon to resolve the name. It should return the result either from the DNS servers you configured or from its cache. If it fails, check the configuration file, and make sure the dnsmasq daemon is running.
Reconfigure DNS Resolution
Once dnsmasq is working, you can change the network configuration to use dnsmasq instead of the existing DNS servers to resolve names. Linux distributions often make this one of the most confusing and frustrating changes to make for your OS, even though it is a such a simple little thing. Sometimes you can do it using netplan. Sometimes you can do it using your DHCP configuration file. Sometimes you can edit resolv.conf directly, but usually your changes will disappear after a restart. Sometimes you can use the resolvconf daemon. Sometimes you can use systemd-resolve, maybe. And sometimes this changes from version to version. The CentOS 8 appliance I am using lets me do it by editing the interface configuration file the old-fashioned way.
The interface configuration file is located in /etc/sysconfig/network-scripts. If your interface is eth0, the configuration file will be /etc/sysconfig/network-scripts/ifcfg-eth0. That file should have entries that start with “DNS”, like DNS1=some-ip. Remove or put a “#” in front of all those lines and replace them with this:
DNS1=127.0.0.1
After you save the changes, you can reload the configuration by running:
$ sudo systemctl restart network
If you are remote, you may lose your connection for a moment. Now check that you can resolve a name without specifying the localhost explicitly.
$ nslookup www.centos.org Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: Name: www.centos.org Address: 81.171.33.201 ...
Note that the local loopback address is now selected automatically.
All this host’s DNS requests will go through dnsmasq now, which will respond directly from cache, or resolve the name, cache it, and then respond.
Summary
Using dnsmasq can help substantially reduce DNS utilization by your Zabbix server instance. You can use it for any Linux host — not just Zabbix — and it makes a real difference for those that talk to DNS a lot.
If the idea of running another daemon does not appeal to you, there is always the IP method instead.
And in case you are wondering, the Zabbix agent is just as chatty with DNS — except that it only asks to resolve the server name(s) configured in the zabbix-agent.conf file…over and over again, all day long. That you can fix that with a hosts file (/etc/hosts) entry.