Twingate connectors not working anymore

Installation
1

Hello everyone,

suddenly my Twingate connection to my server is not working anymore. Therefore, I deleted the Docker containers, as well as the network and the image, and recreated them using the following command:

docker run -d
–sysctl net.ipv4.ping_group_range=“0 2147483647”
–env TWINGATE_NETWORK=“thedodo”
–env TWINGATE_ACCESS_TOKEN=“”
–env TWINGATE_REFRESH_TOKEN=“”
–env TWINGATE_LABEL_HOSTNAME=“hostname”
–name “twingate-crimson-turkey”
–restart=unless-stopped
–pull=always
twingate/connector:1

(Of course with NEW tokens, etc.)

and the FM (Fault Message) is:

docker logs fdc
State: Offline
State: Authentication
State: Authentication

Subsequently, I deleted everything again and re-created it, but this time in debug mode (–env TWINGATE_LOG_LEVEL=7)

docker run -d
–sysctl net.ipv4.ping_group_range=“0 2147483647”
–env TWINGATE_NETWORK=“thedodo”
–env TWINGATE_LOG_LEVEL=7
–env TWINGATE_ACCESS_TOKEN=“”
–env TWINGATE_REFRESH_TOKEN=“”
–env TWINGATE_LABEL_HOSTNAME=“hostname”
–name “twingate-crimson-turkey”
–restart=unless-stopped
–pull=always
twingate/connector:1

2

Now we are receiving the FM :

docker logs 544 -f
[INFO] [connector] logging to stderr: always
[INFO] [connector] starting crash reporting service
[INFO] [connector] initializing sdwan
[DEBUG] [libsdwan] sdwan_new() config: {“controller”:{“mode”:“access”,“url”:“https://thedodo.twingate.com”,“type”:“enterprise”,“app_version”:“1.61.0”},“modules”:{“viper”:{“nat”:{“log-level”:“DEBUG”},“dns-proxy”:{“enable-netbios”:1,“proxy-rules”:[{“type”:“bypass”,“route_id”:0,“default”:1}]},“generic-proxy”:{“session-table-size”:“1024”,“proxy-rules”:[{“type”:“bypass”,“route_id”:0,“default”:1}],“plugin-chain”:[{“name”:“gnrprx”}]}},“tranceport”:{“ssl”:{“ciphers”:“ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384”,“curves”:“P-256:P-384:P-521”,“sigalgs”:“ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512”,“server_auth”:0,“client_auth”:0},“dequeue”:1,“dequeue_r”:0,“enqueue”:3,“enqueue_r”:0,“reorder_stats”:100,“vlink_idle_time”:10,“frontend”:{“flush”:1,“flush_type”:1,“max_queue”:65536,“pools”:[{“impl”:“local”,“id”:3,“link”:“bev_http”,“connect_str”:1},{“impl”:“local”,“id”:5,“link”:“bev_tcp”},{“impl”:“local”,“id”:6,“link”:“bev_udp”}]},“backend”:{“flush”:1,“flush_type”:1,“max_queue”:65536,“pools”:[{“impl”:“local”,“id”:6,“link”:“udp_proxy”},{“id”:10,“impl”:“local”,“link”:“tcp_proxy”,“https_skipencrypt”:1},{“impl”:“local”,“id”:8,“link”:“packet”}]}}},“sdwan”:{“loglevel”:7,“loglevel_console”:-1,“ca_path”:null,“timestamp_fmt”:0,“log_analytics”:“”},“afvpn”:{“dns”:null}}
[DEBUG] [libsdwan] crypto_module_load_ca: loaded 2 trusted certificates from “default”
[DEBUG] [libsdwan] crypto_module_new: available cipher suites:“ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384”, curves: “P-256:P-384:P-521”
[DEBUG] [libsdwan] forced expiration of the access tokens
[DEBUG] [libsdwan] operator(): twingate_label_hostname=kkjb-server
[DEBUG] [libsdwan] [quic] init: new instance
[DEBUG] [libsdwan] [quic] on_accept: enabling server
[DEBUG] [libsdwan] reconfigure_direct_connect_socket: set the size of UDP socket buffers to 4194304 bytes
[DEBUG] [libsdwan] reconfigure_direct_connect_socket: reconfigured direct connect socket
[INFO] [libsdwan] sdwan_new: libsdwan_version=0.153.0, app_version=1.61.0, platform=Linux-x86_64
[DEBUG] [connector] connector start
[INFO] [connector] started server on /connector.sock
[INFO] [libsdwan] sdwan_state: Offline None
[INFO] [connector] State: Offline
State: Offline
State: Authentication
[DEBUG] [libsdwan] [controller] run_state_machine: Offline
[DEBUG] [libsdwan] [controller] set_state: switching from “Offline” to “Getting public keys”
[INFO] [libsdwan] sdwan_state: Authenticating None
[INFO] [connector] State: Authentication
[DEBUG] [libsdwan] [controller] get_controller_keys: fetching controller public keys…
[DEBUG] [libsdwan] send: sending HTTP request 9236688459556240908
[DEBUG] [libsdwan] http::request::send_request: GET “https://thedodo.twingate.com/api/v1/public_keys” text/plain
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: GET “https://thedodo.twingate.com/api/v1/public_keys” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 9236688459556240908 successful response
[DEBUG] [libsdwan] [controller] set_state: switching from “Getting public keys” to “Got public keys”
[DEBUG] [libsdwan] [controller] run_state_machine: Got public keys
[DEBUG] [libsdwan] [controller] set_state: switching from “Got public keys” to “Authenticating”
State: Authentication
[DEBUG] [libsdwan] [controller] set_bootstrap_mode: bootstrap mode from false to false
[INFO] [libsdwan] sdwan_state: Authenticating User
…
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 1
[INFO] [connector] processing “health” request
[DEBUG] [libsdwan] [controller] get_sd: getting SD
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“131502da-1994-40fd-9193-52744d13c375”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538717,“iat”:1704535117,“ver”:“4”,“tid”:“19343”,“rnw”:1704535417,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get SD: Invalid token, code 1
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“18f703b4-3496-43e0-9ff8-f1cf7e46c55b”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538725,“iat”:1704535125,“ver”:“4”,“tid”:“19343”,“rnw”:1704535425,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 1
[INFO] [connector] processing “health” request
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“6183eceb-0ba6-4ce3-bbb5-d55cccdac4dc”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538735,“iat”:1704535135,“ver”:“4”,“tid”:“19343”,“rnw”:1704535435,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 1
[INFO] [connector] processing “health” request
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“8a9d8318-3f10-409e-883f-7924e567e559”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538745,“iat”:1704535145,“ver”:“4”,“tid”:“19343”,“rnw”:1704535445,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 1
[INFO] [connector] processing “health” request
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“9a46a543-59e5-4006-9424-197738b336bf”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538755,“iat”:1704535155,“ver”:“4”,“tid”:“19343”,“rnw”:1704535455,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 1
[INFO] [connector] processing “health” request
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“763fa0f6-e665-42ce-94ea-3762bcc8816b”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538765,“iat”:1704535165,“ver”:“4”,“tid”:“19343”,“rnw”:1704535465,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 1
[INFO] [connector] processing “health” request
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“d8efd988-5b78-4169-96bc-90ffcde8e6b8”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538775,“iat”:1704535175,“ver”:“4”,“tid”:“19343”,“rnw”:1704535475,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 1
[DEBUG] [libsdwan] [controller] get_sd: getting SD
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“084b46c7-06e5-41f1-9495-94b1be6f7c1d”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538777,“iat”:1704535177,“ver”:“4”,“tid”:“19343”,“rnw”:1704535477,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get SD: Invalid token, code 1
[INFO] [connector] processing “health” request
[DEBUG] [libsdwan] [controller] require_access_token: dat.expired
[DEBUG] [libsdwan] send: sending HTTP request 3855790652855851781
[DEBUG] [libsdwan] http::request::send_request: POST “https://thedodo.twingate.com/api/v5/connector/refresh” application/json
[DEBUG] [libsdwan] http::response::from: certificate e4fad85af5c39bf04c0650ba652932617d58189f2ce62b639df8d225c4718710, issuer: C=US, O=Let’s Encrypt, CN=R3, subject: CN=.twingate.com
[DEBUG] [libsdwan] http::request::handle_response: POST “https://thedodo.twingate.com/api/v5/connector/refresh” 200 OK (duration 0 sec)
[DEBUG] [libsdwan] operator(): got HTTP request 3855790652855851781 successful response
[DEBUG] [libsdwan] [controller] verify_token: {“alg”:“ES256”,“kid”:“Vjppq9OiZua5vd_VA7mqR3Ra2PIciNIsekzBlrQXkwg”,“typ”:“DAT”} {“nt”:“AN”,“aid”:“210815”,“did”:“1160676”,“jti”:“0d162262-75c4-41a0-8b85-60ab14d1931d”,“iss”:“twingate”,“aud”:“thedodo”,“exp”:1704538785,“iat”:1704535185,“ver”:“4”,“tid”:“19343”,“rnw”:1704535485,“rnetid”:“79534”}
[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired
[WARN] [libsdwan] [controller] operator(): failed to get an access token: Invalid token, code 1

3

I hope one of you can help me further.
It cannot be due to an expired token, as I have been recreating the token if i recreating the container.

[WARN] [libsdwan] [controller] parse_verify_token: token verification failed: token expired

4

I had everything setup and relatively recently, I noticed the connectors stopped working.
I happen to run mine via docker on my Synology NAS. I thought since I had done some network changes, including installing something else and having multiple connections to the same subnet, I might have cause the issue myself.

Today, I fixed that network connectivity and deployed brand new connectors, with new keys, and new containers on my Synology. Still not connecting.
Can someone please help, I depend on this for remote access to my lab environment.

5

This morning I made the same log level change and see the same error, this is reporting that my brand new token is invalid.

6

Based on the logs, this might be due to the hardware clock being out of sync (causing the connector to flap). I can’t see the system timestamp in the logs provided, but compare your system timestamp to that of the “issued at” timestamp by converting from epoch to human readable (“iat”:1704535165 => Saturday, January 6, 2024 2:59:25 AM GMT-07:00).

Further instructions are provided in this support article, Connector Flapping.