Could not connect to the Twingate Relay

Hi,

I am able to setup twingate without issue everywhere but in our data center. When I install the connector on docker there it gives me the following error on the connector

Could not connect

Could not connect to the Twingate Relay. Be sure to unblock outbound ports 30000-31000.

I have verified all outgoing ports are open, there are no issues or logs showing errors with docker. We have tried with 1 to 1 natting and without and regardless of what we do the connector never connects. If I setup anywhere else outside of the datacenter environment it works right away. There is very little documentation on what might be causing this so I wanted to ask if anyone knew what might be going on here?

Hi @Marvistec,

unfortunately, it does sound like a connectivity issue between the Docker host and Twingate’s Relay infrastructure (outbound port range 30000-31000 is indeed part of it but there is a bit more (see here).

One thing that is potentially worth a try is to deploy a Connector on a VM within the data center: if it also fails to come online, it’s probably an issue with the firewall, if it does not, that would point towards a docker host issue.

Sorry I should have clarified. The connector is running on docker on an Ubuntu VM on a VSphere / Esxi host.

got it, I meant to suggest deploying an additional Connector on a VM directly without docker (using systemd, Ubuntu is perfect for this) and see if it comes online.

Thank you will test and report back.

Same issue again. Bummer we would really like to use your platform but this has been pretty hard to get working without more documentation. Any other suggestions?

Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_remove_conntrack(0x555a14e4b180)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_free(0x555a14e4b180, :0->:0)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] get_needed_conns_count: relay 34.94.156.197:30019, needed_conns_count 1
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [token_fetcher] get_ct: requesting ct_key_t(network: any)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [token_fetcher] get_token: got a cached token for ct_key_t(network: any)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] get_needed_conns_count: relay 34.94.156.197:30019, needed_conns_count 1
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] connect: opening 1 connections to relay relays.twingate.com at 34.94.156.197:30019
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_connect(0x555a14f38090)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] dns_getaddrinfo(34.94.156.197:30019)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] resolve_cb: connecting to 34.94.156.197:30019
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_conntrack(0x555a14f38090)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_set_state(0x555a14ed76f0): switching state from 5 to 1
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_connect(0x555a14ed76f0) hostname: 34.94.156.197:30019, proto: 1, err: 0
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] get_all_listen_addrs: listeners
size 1
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_bev_event(0x555a14ea98d0) events: 20, state: 1
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_bev_event(0x555a14ea98d0): socket error 104 (Connection reset by peer)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_error(0x555a14ea98d0, :0->:0)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_set_state(0x555a14ea3e20): switching state from 1 to 5
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] relay_transport_event: id 1002 failed to connect to relay 34.94.156.197:30019: 104 (Connection reset by peer)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_remove_conntrack(0x555a14ea98d0)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] _relay_free(0x555a14ea98d0, :0->:0)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] get_needed_conns_count: relay 34.94.156.197:30019, needed_conns_count 1
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [token_fetcher] get_ct: requesting ct_key_t(network: any)
Jan 25 20:29:34 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [token_fetcher] get_token: got a cached token for ct_key_t(network: any)

@Marvistec, I got more stuff we can check out :slight_smile:

Could you activate debug level logs on your Connector installed via systemd in Ubuntu?

In order to do that, add TWINGATE_LOG_LEVEL=7 to the /etc/twingate/connector.conf file and restart the Connector by running the following command: sudo systemctl restart twingate-connector

Once you have restarted the Connector, take another look at the Connector log file and post it here, it should tell us a bit more! (you can use the same command as before for this: journalctl -u twingate-connector -n 1000 -f or something similar)

1 Like

Jan 25 20:31:07 mttwingate systemd[1]: twingate-connector.service: Succeeded.
Jan 25 20:31:07 mttwingate systemd[1]: Stopped Twingate Connector service.
Jan 25 20:31:07 mttwingate systemd[1]: Started Twingate Connector service.
Jan 25 20:31:08 mttwingate twingate-connector[397811]: State: Offline
Jan 25 20:31:08 mttwingate twingate-connector[397811]: State: Authentication
Jan 25 20:31:08 mttwingate twingate-connector[397811]: State: Authentication
Jan 25 20:31:08 mttwingate twingate-connector[397811]: State: Online
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_connect(0x555a14edd910)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] dns_getaddrinfo(34.102.42.240:30011)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] resolve_cb: connecting to 34.102.42.240:30011
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_conntrack(0x555a14edd910)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_set_state(0x555a14eb4360): switching state from 5 to 1
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_connect(0x555a14eb4360) hostname: 34.102.42.240:30011, proto: 1, err: 0
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] get_all_listen_addrs: listeners_ size 1
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_bev_event(0x555a14f07040) events: 20, state: 1
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_bev_event(0x555a14f07040): socket error 104 (Connection reset by peer)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_error(0x555a14f07040, :0->:0)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_set_state(0x555a14e8eb80): switching state from 1 to 5
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] relay_transport_event: id 1393 failed to connect to relay 34.102.42.240:30011: 104 (Connection reset by peer)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_remove_conntrack(0x555a14f07040)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_free(0x555a14f07040, :0->:0)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] get_needed_conns_count: relay 34.102.42.240:30011, needed_conns_count 1
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [token_fetcher] get_ct: requesting ct_key_t(network: any)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [token_fetcher] get_token: got a cached token for ct_key_t(network: any)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] get_needed_conns_count: relay 34.102.42.240:30011, needed_conns_count 1
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] connect: opening 1 connections to relay relays.twingate.com at 34.102.42.240:30011
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_connect(0x555a14ed6cf0)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] dns_getaddrinfo(34.102.42.240:30011)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] resolve_cb: connecting to 34.102.42.240:30011
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_conntrack(0x555a14ed6cf0)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_set_state(0x555a14edf650): switching state from 5 to 1
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_connect(0x555a14edf650) hostname: 34.102.42.240:30011, proto: 1, err: 0
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] get_all_listen_addrs: listeners
size 1
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_bev_event(0x555a14ed3e70) events: 20, state: 1
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_bev_event(0x555a14ed3e70): socket error 104 (Connection reset by peer)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_on_error(0x555a14ed3e70, :0->:0)
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] relay_set_state(0x555a14eedb40): switching state from 1 to 5
Jan 25 20:31:06 mttwingate twingate-connector[396944]: [DEBUG] [libsdwan] [relay] relay_transport_event: id 1395 failed to connect to relay 34.102.42.240:30011: 104

thank you! let me check with our support team and get their thoughts

Thank you! I really appreciate you taking the time for this.

of course!

In the meantime, one thing that strikes me as odd from the logs is that Connectors establish connections to relays.twingate.com, in this case the log shows that the Connector host does resolve our Relays properly (in your case the resolution returned is to 34.102.42.240).

The Connector seems to be unable to establish outbound connectivity to port 30011 for that public IP but why connectivity isn’t there, I cannot tell.

Here is a couple of things to try:

Run a telnet command to the public IP of the Relay (34.102.42.240) and port 30011 from various locations to determine where the issue is:

  1. Run telnet 34.102.42.240 30011 from the host machine where the Connector is hosted
  2. Run telnet 34.102.42.240 30011 from your own machine (Assuming you are on the same network)
  3. Run telnet 34.102.42.240 30011 from your own machine but from a different network (home wifi for instance)

(I ran the command myself from my home and it connected successfully)

test 1 will likely fail.

If test 2 fails as well but test 3 succeeds, it will likely be a network issue (firewall?)

if test 2 and test 3 both fail, perhaps there is something installed on machines preventing those outbound connections?

@Marvistec, Support got back to me: It really does look like the Connector cannot connect to our Relays. The tests I mentioned above should tell us more about your side but if you could also send us your Twingate tenant name along with the public egress IP of your Connector then we can check on our side if it was blocked by us. (feel free to send the info to onboarding@twingate.com btw if you don’t want to share it here, you can simply reference this thread in the email).

One more thing Support pointed out: it could be a clock drift between the clock on the host machine running the Connector and actual time. Check out the article here: https://help.twingate.com/hc/en-us/articles/5933234470045

You should be able to see if there is a drift from the Admin Console, if you open up your Connector page, you will see a “time Offset” field, let us know what value it has.

EDIT: it turns out I cannot post more than 3 responses in a row on our Forum so editing my last response @Marvistec! I have heard back from our support team: there is no blocking on our side however we now see a live and functioning Connector in your environment. I assume you found the root cause or found a workaround?

Ill have to get the same network different machine later today but here is from the connector machine

From host machine where the Connector is hosted

XXXXX@mttwingate:~$  telnet 34.102.42.240 30011
Trying 34.102.42.240...
Connected to 34.102.42.240.

Hi!

reposting this from my edit yesterday:

@Marvistec! I have heard back from our support team: there is no blocking on our side however we now see a live and functioning Connector in your environment. I assume you found the root cause or found a workaround?

I just checked again and still see a single Connector in your environment that appears to be live and healthy. Just to confirm: are you still experiencing the same issue on your Connector?

Yes still no relay connection. I then deployed another connector using systemd and still same issue. I’ll do another in docker as well but so far haven’t seen any that have come online or are working.

Hi,

Something does not add up between the issue you are experiencing and what we see on our side:

We see 2 Connectors in your environment (both under 1 Remote Network).

The first Connector was created 22 hours ago and appears fully functional (we see it connect to our Relays successfully). The Egress IP we see for this live Connector is indeed the one you have shared offline with us, indicating that it is actually live and communicating.

The second Connector is not live but was only created 17 min ago.

BTW, we see no Resource added to Twingate environment, is this expected?

Ok I think the one you are seeing is the one that was setup in docker a while ago. The network for that one has since been deleted but I can see that it is still showing onlien and connected.

319b845ca7fa twingate/connector:1 “/connectord” 2 days ago Up 23 hours (healthy) twingate-charcoal-corgi

The one I just deployed was using the systemd method for a network that we have in our account.

How should I proceed. Deleted all and setup a new network and try to re deploy the docker connector on the new network?