Netdata flooding Pihole logs

4 minute de lecture
Par Stéphane
Netdata flooding Pihole logs

Netdata is a very useful tool. It probes several parameters of a Unix server, and display them on a nice dashboard. For more, see their website : https://www.netdata.com.

That being said, Netdata is also monitoring my DNS server, which runs the classic Pihole program.

I just installed a new app on my iPhone, and wanted to see to which servers it was trying to contact. To achieve this I usually just do a tail -f /var/log/pihole.log either from a shell, or directly from Pihole web interface.

I've notice a very large number of queries like those :

May 17 00:00:03 dnsmasq[1581870]: config cachesize.bind is <TXT>
May 17 00:00:03 dnsmasq[1581870]: config insertions.bind is <TXT>
May 17 00:00:03 dnsmasq[1581870]: config evictions.bind is <TXT>
May 17 00:00:03 dnsmasq[1581870]: config hits.bind is <TXT>
May 17 00:00:03 dnsmasq[1581870]: config misses.bind is <TXT>
May 17 00:00:03 dnsmasq[1581870]: config auth.bind is <TXT>
May 17 00:00:03 dnsmasq[1581870]: config servers.bind is <TXT>
May 17 00:00:04 dnsmasq[1581870]: config cachesize.bind is <TXT>
May 17 00:00:04 dnsmasq[1581870]: config insertions.bind is <TXT>
May 17 00:00:04 dnsmasq[1581870]: config evictions.bind is <TXT>
May 17 00:00:04 dnsmasq[1581870]: config hits.bind is <TXT>
May 17 00:00:04 dnsmasq[1581870]: config misses.bind is <TXT>
May 17 00:00:04 dnsmasq[1581870]: config auth.bind is <TXT>
May 17 00:00:04 dnsmasq[1581870]: config servers.bind is <TXT>
May 17 00:00:05 dnsmasq[1581870]: config cachesize.bind is <TXT>
May 17 00:00:05 dnsmasq[1581870]: config insertions.bind is <TXT>
May 17 00:00:05 dnsmasq[1581870]: config evictions.bind is <TXT>
May 17 00:00:05 dnsmasq[1581870]: config hits.bind is <TXT>
May 17 00:00:05 dnsmasq[1581870]: config misses.bind is <TXT>
May 17 00:00:05 dnsmasq[1581870]: config auth.bind is <TXT>
May 17 00:00:05 dnsmasq[1581870]: config servers.bind is <TXT>
May 17 00:00:06 dnsmasq[1581870]: config cachesize.bind is <TXT>
May 17 00:00:06 dnsmasq[1581870]: config insertions.bind is <TXT>
May 17 00:00:06 dnsmasq[1581870]: config evictions.bind is <TXT>
May 17 00:00:06 dnsmasq[1581870]: config hits.bind is <TXT>

As you can see they repeat every second, which has 2 bad side effects :

  1. filling the disk, which can be a problem is your DNS server is a Raspberry Pi ;
  2. augmenting the load average of the CPU by roughly 0.5 (was ~0.70, and it's now less than 0.20).

To correct this behavior, I did the following :

sudo systemctl stop netdata
cd /usr/lib/netdata/conf.d/go.d
sudo emacs dnsmasq.conf
: In the GLOBAL section at the end of the file :
: replace # update_every: 1
: by update_every: 120
: [remove the comment character #, the space, and add 20 at the end of the lne]
: save & restart netdata
sudo systemctl restart netdata

And now :

May 17 01:02:38 dnsmasq[1581870]: config cachesize.bind is <TXT>
May 17 01:02:38 dnsmasq[1581870]: config insertions.bind is <TXT>
May 17 01:02:38 dnsmasq[1581870]: config evictions.bind is <TXT>
May 17 01:02:38 dnsmasq[1581870]: config hits.bind is <TXT>
May 17 01:02:38 dnsmasq[1581870]: config misses.bind is <TXT>
May 17 01:02:38 dnsmasq[1581870]: config auth.bind is <TXT>
May 17 01:02:39 dnsmasq[1581870]: config servers.bind is <TXT>
May 17 01:04:39 dnsmasq[1581870]: config cachesize.bind is <TXT>
May 17 01:04:39 dnsmasq[1581870]: config insertions.bind is <TXT>
May 17 01:04:39 dnsmasq[1581870]: config evictions.bind is <TXT>
May 17 01:04:39 dnsmasq[1581870]: config hits.bind is <TXT>
May 17 01:04:39 dnsmasq[1581870]: config misses.bind is <TXT>
May 17 01:04:39 dnsmasq[1581870]: config auth.bind is <TXT>
May 17 01:04:39 dnsmasq[1581870]: config servers.bind is <TXT>
May 17 01:06:38 dnsmasq[1581870]: config cachesize.bind is <TXT>
May 17 01:06:38 dnsmasq[1581870]: config insertions.bind is <TXT>
May 17 01:06:38 dnsmasq[1581870]: config evictions.bind is <TXT>
May 17 01:06:38 dnsmasq[1581870]: config hits.bind is <TXT>
May 17 01:06:38 dnsmasq[1581870]: config misses.bind is <TXT>
May 17 01:06:38 dnsmasq[1581870]: config auth.bind is <TXT>
May 17 01:06:39 dnsmasq[1581870]: config servers.bind is <TXT>

The UDP traffic dropped significantly when I applied the new timeout :

Average packet per second with netdata querying pihole every second, and after reseting the timeout.

Which means less I/O then lest interrupts therefore a more responsive server :

The server is a very small machine, it's powered by an Intel N3700 @1,60Ghz CPU, but which won't use more than a few Watts (about 6W). It costs me the price of a Raspberry Pi, and is more powerful.

I'm looking for a replacement, with a newer CPU. For example a Pentium N5000 would be twice as fast and a Celeron 5100 3 times faster !

Compared speed of the Intel processors N3700 N500 and N5100 (Passmarks).

The N3700 is used in the Synology D916+ and maybe other NAS.

If you want to have more information about this machine compared to a i5-6500T based one and a Raspberry Pi, search for Lenovo on this blog :D