Connector not getting back online

Support / Troubleshooting
1

Hi. Since today there seems to be an issue with our Twingate connectors.

I have a single site with two connectors. They haven’t been touched for a week in terms of updates and basically anything else.
Today these connectors just stopped working. No idea why.

I’ve restartet both hosts, performed updates on these but still - they are not getting back online. I even deployed new connectors just to be sure.

I did a packet capture on the nodes and I can see the following:

  • the connector does a dns lookup for my XXX.twingate.com domain and is successful
  • the connector starts some https talking to a google IP 34.111.220.252 which a least hosts a twingate ssl cert on port 443. So seems to be a reasonable host.

Still - no connect possible.

I can’t see anything on the logs either.

Feb 07 23:17:18 some.host.name.local systemd[1]: Started Twingate Connector service.
Feb 07 23:17:18 some.host.name.local twingate-connector[2021]: State: Offline
Feb 07 23:17:18 some.host.name.local twingate-connector[2021]: State: Authentication
Feb 07 23:17:18 some.host.name.local twingate-connector[2021]: State: Authentication

Any idea? I tried to increase log-level on my connector but not a single different output line.

2

Try to set the Debug logs in your connector and see what the logs output actually, we should be able to tell more. Do you know of any network changes or config changes on the firewall perhaps?

3

Hi Bren,

now we’re talking …

twingate-connector[3303]: [DEBUG] [libsdwan] [controller] get_sd: getting SD
twingate-connector[3303]: [DEBUG] [libsdwan] [controller] require_access_token: dat.expired
twingate-connector[3303]: [DEBUG] [libsdwan] submit_request: sending HTTP request ******,
twingate-connector[3303]: [DEBUG] [libsdwan] http::request::send_request: POST “https://xxx.twingate.com/api/v5/connector/refresh” application/json
twingate-connector[3303]: [DEBUG] [libsdwan] http::response::from: certificate , issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
twingate-connector[3303]: [DEBUG] [libsdwan] http::request::handle_response: POST “https://xxx.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
twingate-connector[3303]: [DEBUG] [libsdwan] operator(): got HTTP request , successful response
twingate-connector[3303]: [DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:",“,“typ”:“DAT”} {“nt”:“AN”,“aid”:“228762”,“did”:“1226123”,“jti”:”,",“iss”:“twingate”,“aud”:“xxx”,“exp”:1707491309,“iat”:1707487709,“ver”:“4”,“tid”:“23268”,“rnw”:1707487954,“rnetid”:“30157”}
twingate-connector[3303]: [WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
twingate-connector[3303]: [WARN] [libsdwan] [controller] operator(): failed to get SD: Invalid token, code 0

I’ve just re-generated the keys in the web gui, deployed the keys to /etc/twingate/connector.conf and restarted the service … still no connection, still the same issues.

4

I’ve removed all connectors on our site once again and deployed a new connector with new keys, new VM and all that. Still the same issue:

systemd[1]: Started Twingate Connector service.
twingate-connector[3432]: [INFO] [connector] logging to stderr: always
twingate-connector[3432]: [INFO] [connector] starting crash reporting service
twingate-connector[3432]: [INFO] [connector] initializing sdwan
twingate-connector[3432]: [INFO] [connector] started server on /var/run/twingate/connector.sock
twingate-connector[3432]: State: Offline
twingate-connector[3432]: [INFO] [connector] State: Offline
twingate-connector[3432]: State: Authentication
twingate-connector[3432]: [INFO] [connector] State: Authentication
twingate-connector[3432]: State: Authentication
twingate-connector[3432]: [INFO] [connector] State: Authentication
twingate-connector[3432]: [WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
twingate-connector[3432]: [WARN] [libsdwan] [controller] operator(): failed to get SD: Invalid token, code 0
twingate-connector[3432]: [WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
twingate-connector[3432]: [WARN] [libsdwan] [controller] operator(): failed to get SD: Invalid token, code 0
twingate-connector[3432]: [WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
twingate-connector[3432]: [WARN] [libsdwan] [controller] operator(): failed to get SD: Invalid token, code 0
twingate-connector[3432]: [WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
twingate-connector[3432]: [WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 0
twingate-connector[3432]: [WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
twingate-connector[3432]: [WARN] [libsdwan] [controller] operator(): failed to get SD: Invalid token, code 0
twingate-connector[3432]: [INFO] [connector] ping watchdog, interval 15000000 us
twingate-connector[3432]: [WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
twingate-connector[3432]: [WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 0

5

Hi, I think I know what the issue is and the key to it is the following line:

The Relays / Controller / Connectors / Clients exchange tokens for security purposes and all components need to reference the same timeframe. The token expired message likely indicates a clock drift on your Connector(s). Take a look here for more: https://help.twingate.com/hc/en-us/articles/5933234470045-Connector-Flapping-Offline-Online-or-Unavailable

6

DAMN! Yes. You’re awesome. For what ever reason NTP sync on my local vms did not work. I fixed that so now we’re back online. Nice! Thank you so much!