After any network interruption, all connectors stay offline with Nameserver has failed messages

Support / Troubleshooting
1

Getting the following errors from multiple connectors (3 debian, one docker) after rebooting the building’s router to the internet which caused an internet interruption of one minute:

systemctl status twingate-connector.service
● twingate-connector.service - Twingate Connector service
     Loaded: loaded (/lib/systemd/system/twingate-connector.service; enabled; preset: disabled)
     Active: active (running) since Mon 2023-10-23 08:00:03 MDT; 2h 59min ago
   Main PID: 3166235 (twingate-connec)
      Tasks: 10 (limit: 38284)
     Memory: 9.7M
        CPU: 15.114s
     CGroup: /system.slice/twingate-connector.service
             └─3166235 /usr/bin/twingate-connector --systemd-watchdog

Oct 23 10:59:43 jarvis twingate-connector[3166235]: [msg] All nameservers have failed
Oct 23 10:59:43 jarvis twingate-connector[3166235]: [msg] Nameserver 10.0.0.1:53 is back up
Oct 23 10:59:43 jarvis twingate-connector[3166235]: [msg] Nameserver 10.0.0.1:53 has failed: Bad response 4 (query not implemented)
Oct 23 10:59:43 jarvis twingate-connector[3166235]: [msg] All nameservers have failed
Oct 23 10:59:44 jarvis twingate-connector[3166235]: [msg] Nameserver 10.0.0.1:53 is back up
Oct 23 10:59:44 jarvis twingate-connector[3166235]: [msg] Nameserver 10.0.0.1:53 has failed: Bad response 4 (query not implemented)
Oct 23 10:59:44 jarvis twingate-connector[3166235]: [msg] All nameservers have failed
Oct 23 10:59:44 jarvis twingate-connector[3166235]: [msg] Nameserver 10.0.0.1:53 is back up
Oct 23 10:59:44 jarvis twingate-connector[3166235]: [msg] Nameserver 10.0.0.1:53 has failed: Bad response 4 (query not implemented)
Oct 23 10:59:44 jarvis twingate-connector[3166235]: [msg] All nameservers have failed

All other services are running on the network, DNS is running, it seems as though the connectors just zombie out from the transient interruption.

I added a crontab entry to resolve the issue for now but this feels like a twingate connector related problem that should be resolved. I’d like to see the connectors be a bit more resilient to transient events.

Crontab Entry:

*/5 * * * * /usr/bin/journalctl -n 5 -u twingate-connector.service | grep fail && /usr/bin/systemctl restart twingate-connector.service > /dev/null 2>&1
2

Hey Ske,

This is a new one to me, I’ve never seen a connector spit that out, but ultimately what it suggests is that there’s a type of DNS query going out that is unsupported by the server that’s being asked. Is 10.0.0.1 your router? What brand is it? And what are you using for your upstream DNS servers on that router?

I’ve also got a request out to my team to see if they have any other questions and will let you know.

Thanks,

-arthur

3

Thanks. It’s an Asus GT-AX11000

Here’s the output from another system I have not added the crontab fix to and included the dig output. The OS isn’t having DNS issues while the connector still is, and the interruption was hours ago.

root@alfred:/# systemctl status twingate-connector.service
● twingate-connector.service - Twingate Connector service
     Loaded: loaded (/lib/systemd/system/twingate-connector.service; enabled; preset: disabled)
     Active: active (running) since Mon 2023-10-23 10:00:35 MDT; 1h 26min ago
      Until: Mon 2023-10-23 16:00:35 MDT; 4h 33min left
   Main PID: 3885881 (twingate-connec)
      Tasks: 10 (limit: 38284)
     Memory: 7.8M
        CPU: 9.491s
     CGroup: /system.slice/twingate-connector.service
             └─3885881 /usr/bin/twingate-connector --systemd-watchdog

Oct 23 11:26:53 alfred twingate-connector[3885881]: [msg] All nameservers have failed
Oct 23 11:26:55 alfred twingate-connector[3885881]: [msg] Nameserver 10.0.0.1:53 is back up
Oct 23 11:26:55 alfred twingate-connector[3885881]: [msg] Nameserver 10.0.0.1:53 has failed: Bad response 4 (query not implemented)
Oct 23 11:26:55 alfred twingate-connector[3885881]: [msg] All nameservers have failed
Oct 23 11:26:55 alfred twingate-connector[3885881]: [msg] Nameserver 10.0.0.1:53 is back up
Oct 23 11:26:55 alfred twingate-connector[3885881]: [msg] Nameserver 10.0.0.1:53 has failed: Bad response 4 (query not implemented)
Oct 23 11:26:55 alfred twingate-connector[3885881]: [msg] All nameservers have failed
Oct 23 11:26:56 alfred twingate-connector[3885881]: [msg] Nameserver 10.0.0.1:53 is back up
Oct 23 11:26:56 alfred twingate-connector[3885881]: [msg] Nameserver 10.0.0.1:53 has failed: Bad response 4 (query not implemented)
Oct 23 11:26:56 alfred twingate-connector[3885881]: [msg] All nameservers have failed
root@alfred:/# dig twingate.com

; <<>> DiG 9.18.16-1~deb12u1-Debian <<>> twingate.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53280
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;twingate.com.                  IN      A

;; ANSWER SECTION:
twingate.com.           28      IN      A       104.198.14.52

;; Query time: 0 msec
;; SERVER: 10.0.0.1#53(10.0.0.1) (UDP)
;; WHEN: Mon Oct 23 11:27:01 MDT 2023
;; MSG SIZE  rcvd: 57

root@alfred:/#
4

What upstream DNS servers are you using? ISP? Google? Cloudflare? something else?

5

Here’s the configuration for 10.0.0.1 DNS:

image
image1510×1226 120 KB

6

As an additional data point, here’s my container instance, which also has the same issue, and it is not using 10.0.0.1, for reference. It’s an LXC Debian bullseye container with twingate installed. Hope that’s of use.

root@twingate:~# systemctl status twingate-connector.service 
* twingate-connector.service - Twingate Connector service
     Loaded: loaded (/lib/systemd/system/twingate-connector.service; enabled; vendor preset: dis>
     Active: active (running) since Mon 2023-10-23 18:00:06 UTC; 26min ago
   Main PID: 123 (twingate-connec)
      Tasks: 10 (limit: 38284)
     Memory: 8.0M
        CPU: 3.682s
     CGroup: /system.slice/twingate-connector.service
             `-123 /usr/bin/twingate-connector --systemd-watchdog

Oct 23 18:00:06 twingate systemd[1]: Started Twingate Connector service.
Oct 23 18:00:06 twingate twingate-connector[123]: State: Offline
Oct 23 18:00:06 twingate twingate-connector[123]: State: Authentication
Oct 23 18:00:07 twingate twingate-connector[123]: State: Online
root@twingate:~# journalctl -ru twingate-connector.service
-- Journal begins at Thu 2023-06-22 23:32:36 UTC, ends at Mon 2023-10-23 18:24:52 UTC. --
Oct 23 18:00:07 twingate twingate-connector[123]: State: Online
Oct 23 18:00:06 twingate twingate-connector[123]: State: Authentication
Oct 23 18:00:06 twingate twingate-connector[123]: State: Offline
Oct 23 18:00:06 twingate systemd[1]: Started Twingate Connector service.
-- Boot 8xxxd8ea9a --
Oct 23 18:00:03 twingate systemd[1]: twingate-connector.service: Consumed 46.636s CPU time.
Oct 23 18:00:03 twingate systemd[1]: Stopped Twingate Connector service.
Oct 23 18:00:03 twingate systemd[1]: twingate-connector.service: Succeeded.
Oct 23 18:00:03 twingate twingate-connector[123]: ../core/pubnub_netcore.c:572: pbpal_report_err>
Oct 23 18:00:02 twingate twingate-connector[123]: State: Offline
Oct 23 18:00:01 twingate systemd[1]: Stopping Twingate Connector service...
Oct 23 18:00:00 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 18:00:00 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 18:00:00 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:58 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:58 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 17:59:58 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:57 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:57 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 17:59:57 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:50 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:50 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 17:59:50 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:48 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:48 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 17:59:48 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:47 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:47 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 17:59:47 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:40 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:40 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 17:59:40 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up

And just to drive home the behavior - A systemctl restart fixes the issue. Here’s the example on the container:

root@twingate:~# systemctl restart twingate-connector.service 
root@twingate:~# systemctl status twingate-connector.service 
* twingate-connector.service - Twingate Connector service
     Loaded: loaded (/lib/systemd/system/twingate-connector.service; enabled; vendor preset: dis>
     Active: active (running) since Mon 2023-10-23 18:30:08 UTC; 5s ago
   Main PID: 372 (twingate-connec)
      Tasks: 10 (limit: 38284)
     Memory: 5.8M
        CPU: 254ms
     CGroup: /system.slice/twingate-connector.service
             `-372 /usr/bin/twingate-connector --systemd-watchdog

Oct 23 18:30:08 twingate systemd[1]: Started Twingate Connector service.
Oct 23 18:30:08 twingate twingate-connector[372]: State: Offline
Oct 23 18:30:08 twingate twingate-connector[372]: State: Authentication
Oct 23 18:30:09 twingate twingate-connector[372]: State: Online
root@twingate:~# journalctl -ru twingate-connector.service
-- Journal begins at Thu 2023-06-22 23:32:36 UTC, ends at Mon 2023-10-23 18:30:09 UTC. --
Oct 23 18:30:09 twingate twingate-connector[372]: State: Online
Oct 23 18:30:08 twingate twingate-connector[372]: State: Authentication
Oct 23 18:30:08 twingate twingate-connector[372]: State: Offline
Oct 23 18:30:08 twingate systemd[1]: Started Twingate Connector service.
Oct 23 18:30:08 twingate systemd[1]: twingate-connector.service: Consumed 4.276s CPU time.
Oct 23 18:30:08 twingate systemd[1]: Stopped Twingate Connector service.
Oct 23 18:30:08 twingate systemd[1]: twingate-connector.service: Succeeded.
Oct 23 18:30:08 twingate twingate-connector[123]: ../core/pubnub_netcore.c:572: pbpal_report_err>
Oct 23 18:30:08 twingate twingate-connector[123]: State: Offline
Oct 23 18:30:07 twingate systemd[1]: Stopping Twingate Connector service...
Oct 23 18:00:07 twingate twingate-connector[123]: State: Online
Oct 23 18:00:06 twingate twingate-connector[123]: State: Authentication
Oct 23 18:00:06 twingate twingate-connector[123]: State: Offline
Oct 23 18:00:06 twingate systemd[1]: Started Twingate Connector service.
-- Boot 87d1exxxxb6c6b8d8ea9a --
Oct 23 18:00:03 twingate systemd[1]: twingate-connector.service: Consumed 46.636s CPU time.
Oct 23 18:00:03 twingate systemd[1]: Stopped Twingate Connector service.
Oct 23 18:00:03 twingate systemd[1]: twingate-connector.service: Succeeded.
Oct 23 18:00:03 twingate twingate-connector[123]: ../core/pubnub_netcore.c:572: pbpal_report_err>
Oct 23 18:00:02 twingate twingate-connector[123]: State: Offline
Oct 23 18:00:01 twingate systemd[1]: Stopping Twingate Connector service...
Oct 23 18:00:00 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 18:00:00 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 18:00:00 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:58 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:58 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 17:59:58 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:57 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:57 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
Oct 23 17:59:57 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 is back up
Oct 23 17:59:50 twingate twingate-connector[123]: [msg] All nameservers have failed
Oct 23 17:59:50 twingate twingate-connector[123]: [msg] Nameserver 1.1.1.1:53 has failed: Bad re>
root@twingate:~#
7

Thank you for the additional data points. I’ve passed them off to my team.

As soon as I hear back I’ll let you know!

8

Any update on why the connectors are not able to come back online on their own and what we might do to fix the issue?

9

Adding my voice here, I just noticed the same thing in my logs. Same error, slightly different details:

Rocky Linux 8, fully updated.
Twingate Connector, 1.6.0

I saw there were 3 entries in the /etc/resolv.conf, the error was showing up on the last one, which turned out to be my Asus router. There are two other DNS servers listed, one a local PiHole and one an external server called NextDNS

The router was adding itself at the end, I’ve disabled that for now, but wondering about the messages myself. I have a long list ofmessages telling me that the connectors have been offline, then online, and I haven’t had any issues with networking