Ytdyklly

I'm having problems with SystemD's resolved service on Ubuntu 18.10 - on my wireless network we have an IPv6 subnet that is tunneled through some weird network setup (work stuff) and can't be used for DNS resolution.

For some reason, even though I have set up static DNS configuration in /etc/systemd/resolved.conf.d/dns.conf to only include IPv4 DNS records, and the DHCP never advertises IPv6 DNS servers, resolved always adds some funky IPv6 addresses to the connection-specific DNS configuration on my wireless connection, and then "fallsback" to them and stops resolving addresses.

This is my dns.conf (the primary resolved.conf is default - there's nothing there):

[Resolve]

DNS=172.18.1.254 8.8.8.8 8.8.4.4 1.1.1.1

DNSSEC=no

(the first DNS address is the local router)

The resolved global configuration (as can be gotten from resolvectl status):

Global

       LLMNR setting: no

MulticastDNS setting: no

  DNSOverTLS setting: no

      DNSSEC setting: no

    DNSSEC supported: no

  Current DNS Server: 172.18.1.254

         DNS Servers: 172.18.1.254

                      8.8.8.8

                      8.8.4.4

                      1.1.1.1

...



Link 2 (wlp2s0)

      Current Scopes: DNS

       LLMNR setting: yes

MulticastDNS setting: no

  DNSOverTLS setting: no

      DNSSEC setting: no

    DNSSEC supported: no

  Current DNS Server: 2001:4860:4860::4444

         DNS Servers: 172.18.1.254

                      8.8.8.8

                      8.8.4.4

                      2001:470:1f09:114c::1

                      2001:4860:4860::4444

                      2001:4860:4860::8888

          DNS Domain: ~.

The wlp2s0 status starts with the current DNS server set as 172.18.1.254, then after a short time (we're talking minutes here), it falls back through all the IPv4 servers and lands on one of the IPv6 servers - after which all DNS queries fail, and I'm basically get disconnected from any meaningful network operation. A systemctl restart systemd-resolved clears the issue for a few minutes, after which its back to "no DNS for you, come back when you have a non-IPv6 network".

I have no idea where those IPv6 addresses come from. I've enabled debug logging on resolved, and looking at the debug log the first mention of an IPv6 address is something like this:

Feb 12 10:36:04 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).

Feb 12 10:36:04 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Switching to DNS server 2001:470:1f09:114c::1 for interface wlp2s0.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:04 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Using DNS server 2001:470:1f09:114c::1 for transaction 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Sending query packet with id 51369.

It looks like if something on my system queries an address that cannot be resolved, and the upstream DNS server returns SERVFAIL (this is not just my local router's DNS resolver that is broken - 8.8.8.8 returns the same response), then resolved falls back through all the server until it finds a server that doesn't respond at all - on the way causing any query, not just the one that is broken, to use a broken server.

I can trace the path of such an escalation in the debug log (some.domain is a placeholder for another address that cannot be resolved):

Feb 12 10:36:01 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:01 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:01 vesho systemd-resolved[22371]: Using feature level UDP for transaction 51369.

Feb 12 10:36:01 vesho systemd-resolved[22371]: Using DNS server 172.18.1.254 for transaction 51369.

Feb 12 10:36:01 vesho systemd-resolved[22371]: Sending query packet with id 51369.

Feb 12 10:36:01 vesho systemd-resolved[22371]: Cache miss for some.domain IN AAAA

Feb 12 10:36:01 vesho systemd-resolved[22371]: Transaction 44627 for <some.domain IN AAAA> scope dns on wlp2s0/*.

...

Feb 12 10:36:03 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).

Feb 12 10:36:03 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Switching to DNS server 8.8.8.8 for interface wlp2s0.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:03 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Using DNS server 8.8.8.8 for transaction 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Sending query packet with id 51369.

...

Feb 12 10:36:03 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).

Feb 12 10:36:03 vesho systemd-resolved[22371]: Server returned error SERVFAIL, retrying transaction with reduced feature level UDP.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:03 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Using feature level UDP for transaction 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Sending query packet with id 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).

Feb 12 10:36:03 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Switching to DNS server 8.8.4.4 for interface wlp2s0.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:03 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Using DNS server 8.8.4.4 for transaction 51369.

Feb 12 10:36:03 vesho systemd-resolved[22371]: Sending query packet with id 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).

Feb 12 10:36:04 vesho systemd-resolved[22371]: Server returned error SERVFAIL, retrying transaction with reduced feature level UDP.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:04 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Using feature level UDP for transaction 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Sending query packet with id 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).

Feb 12 10:36:04 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Switching to DNS server 2001:470:1f09:114c::1 for interface wlp2s0.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:04 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Using DNS server 2001:470:1f09:114c::1 for transaction 51369.

Feb 12 10:36:04 vesho systemd-resolved[22371]: Sending query packet with id 51369.

...

Feb 12 10:36:08 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).

Feb 12 10:36:08 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:08 vesho systemd-resolved[22371]: Switching to DNS server 2001:4860:4860::8888 for interface wlp2s0.

Feb 12 10:36:08 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:08 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:08 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.

Feb 12 10:36:08 vesho systemd-resolved[22371]: Using DNS server 2001:4860:4860::8888 for transaction 51369.

Feb 12 10:36:08 vesho systemd-resolved[22371]: Sending query packet with id 51369.

...

Feb 12 10:36:13 vesho systemd-resolved[22371]: Timeout reached on transaction 51369.

Feb 12 10:36:13 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:13 vesho systemd-resolved[22371]: Switching to DNS server 2001:4860:4860::8888 for interface wlp2s0.

Feb 12 10:36:13 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:13 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:13 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.

Feb 12 10:36:13 vesho systemd-resolved[22371]: Sending query packet with id 51369.

...

Feb 12 10:36:18 vesho systemd-resolved[22371]: Timeout reached on transaction 51369.

Feb 12 10:36:18 vesho systemd-resolved[22371]: Retrying transaction 51369.

Feb 12 10:36:18 vesho systemd-resolved[22371]: Switching to DNS server 2001:4860:4860::8888 for interface wlp2s0.

Feb 12 10:36:18 vesho systemd-resolved[22371]: Cache miss for some.domain IN A

Feb 12 10:36:18 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> scope dns on wlp2s0/*.

Feb 12 10:36:18 vesho systemd-resolved[22371]: Using feature level UDP+EDNS0 for transaction 51369.

Feb 12 10:36:18 vesho systemd-resolved[22371]: Sending query packet with id 51369.

... 

goes like that for a while

...

Feb 12 10:36:28 vesho systemd-resolved[22371]: Processing incoming packet on transaction 51369 (rcode=SERVFAIL).

Feb 12 10:36:28 vesho systemd-resolved[22371]: Server returned error: SERVFAIL

Feb 12 10:36:28 vesho systemd-resolved[22371]: Added SERVFAIL cache entry for some.domain IN A 30s

Feb 12 10:36:28 vesho systemd-resolved[22371]: Transaction 51369 for <some.domain IN A> on scope dns on wlp2s0/* now complete with <rcode-failure> from network (unsigned)

Feb 12 10:36:28 vesho systemd-resolved[22371]: Freeing transaction 51369.

So to conclude, my question is: Where does resovled find those IPv6 server addresses and how can I stop it from using them at all (setting a hardcoded DNS or FallbackDNS in the configuration files doesn't seem to stop it)?

If anyone has any suggestion for another solution to this problem, I'd love to hear it. Except if its "disable resolved" - I'm actually doing that now so I can get some work done, but I'd like to get to the bottom of this resolved problem and fix it.