Docker Connector Issues

Hello,

I’m facing a connectivity issue with the Connector and it seems that I’m not able to find a logical explanation or a solution for it.

To give you a bit of background details…

I’m using Hetzner as my Cloud provider for a new project.

I’ve created a private Network with a couple of subnets. Within this private network I have provisioned a NAT “Gateway” (Debian 11 + IPTables). The NAT works as expected, all the servers within this private network can connect outside and do the regular stuff as expected, no issues.

Within the same private network I’ve deployed a Docker based Twingate Connector but I’m facing some issues with it and I am not sure what’s the problem.

From the host server where the Docker based Connector is running:

root@htz-use1-din-twn-01:~# curl -IL https://cloudresty.twingate.com
...
HTTP/2 403

From the Docker instance (same host):

root@htz-use1-din-twn-01:~# docker run --rm alpine/curl:3.14 -IL https://cloudresty.twingate.com
...
HTTP/2 403
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

I assume both responses above are “valid” as there’s no token passed, it’s just a “regular” request.

Same host but towards Google:

root@htz-use1-din-twn-01:~# curl -IL https://google.com
HTTP/2 301
...
HTTP/2 200
root@htz-use1-din-twn-01:~# docker run --rm alpine/curl:3.14 -IL https://google.com
...
HTTP/2 301
...
HTTP/2 200

So technically I can reach any external endpoint being from the host itself or from Docker.

The Twingate Connector instead is throwing some errors and I just don’t understand what could be the problem:

root@htz-use1-din-twn-01:~# docker logs -f htz-use1-din-twn-01
[ERROR] [libsdwan] HYD: afvpn_obj_config: failed to process "dns"
State: Offline
State: Authentication
State: Error
[ERROR] [libsdwan] operator(): failed HTTP request 2042861868087309992 403 Forbidden
[ERROR] [libsdwan] controller_t::operator(): failed to get public keys: Forbidden
State: Offline
State: Authentication
State: Error
[ERROR] [libsdwan] operator(): failed HTTP request 2042861868087309992 403 Forbidden
[ERROR] [libsdwan] controller_t::operator(): failed to get public keys: Forbidden
State: Offline
State: Authentication
State: Error
...
State: Authentication
[ERROR] [libsdwan] operator(): failed HTTP request 8173942489313068422 403 Forbidden
[ERROR] [libsdwan] controller_t::operator(): failed to get SD: Forbidden, err code 403
[ERROR] [libsdwan] operator(): failed HTTP request 8173942489313068422 403 Forbidden
[ERROR] [libsdwan] controller_t::operator(): failed to get SD: Forbidden, err code 403
[ERROR] [libsdwan] operator(): failed HTTP request 8173942489313068422 403 Forbidden
[ERROR] [libsdwan] controller_t::operator(): failed to get SD: Forbidden, err code 403
[ERROR] [libsdwan] operator(): failed HTTP request 8173942489313068422 403 Forbidden
[ERROR] [libsdwan] controller_t::operator(): failed to get SD: Forbidden, err code 403
[ERROR] [libsdwan] operator(): failed HTTP request 8173942489313068422 403 Forbidden
[ERROR] [libsdwan] controller_t::operator(): failed to get SD: Forbidden, err code 403
[ERROR] [libsdwan] Core::healthy: controller is not healthy
[ERROR] [libsdwan] Core::healthy: controller is not healthy
[ERROR] [libsdwan] Core::healthy: controller is not healthy
[ERROR] [libsdwan] operator(): failed HTTP request 8173942489313068422 403 Forbidden
[ERROR] [libsdwan] controller_t::operator(): failed to get SD: Forbidden, err code 403
[ERROR] [libsdwan] Core::healthy: controller is not healthy
[ERROR] [libsdwan] Core::healthy: controller is not healthy
[ERROR] [libsdwan] Core::healthy: controller is not healthy
[ERROR] [libsdwan] Core::healthy: controller is not healthy

I wonder if anyone has encountered this issue before or if anyone can help me understand what am I missing.

Thank you

1 Like

Hi Sebastian,

What we’re seeing in both your cURL requests and the connector logs is that connections to your Twingate endpoint URL (cloudresty.twingate.com) are returning 403 Forbidden.

This is a very odd thing - as you should get a valid 200 response back, even if there had been a spelling error in the url/tenant slug.

Typically, we’d only see 403 returned from the tenant slug if there was a third party IdP in place and you were not logged in, or if you were trying to access a very specific protected endpoint.Based on what I can see from your account, this is not the case.

Can you please try ping cloudresty.twingate.com from the host machine and provide the output you get back?

Thanks!

-arthur

Hello,

Have the same issue but deploying twingate into AWS EC2 Instance.

Nov 02 22:30:58 ip-172-31-7-74 systemd[1]: Started Twingate Connector service.
Nov 02 22:30:58 ip-172-31-7-74 twingate-connector[1693]: [ERROR] [libsdwan] HYD: afvpn_obj_config: failed to process "dns"
Nov 02 22:30:58 ip-172-31-7-74 twingate-connector[1693]: State: Offline
Nov 02 22:30:58 ip-172-31-7-74 twingate-connector[1693]: State: Authentication
Nov 02 22:31:00 ip-172-31-7-74 twingate-connector[1693]: [ERROR] [libsdwan] operator(): failed HTTP request 4125923095300265675 410 Gone
Nov 02 22:31:00 ip-172-31-7-74 twingate-connector[1693]: State: Error
Nov 02 22:31:00 ip-172-31-7-74 twingate-connector[1693]: [ERROR] [libsdwan] controller_t::operator(): failed to get SD: Gone, err code 410
Nov 02 22:31:00 ip-172-31-7-74 twingate-connector[1693]: [ERROR] [libsdwan] controller_t::operator(): failed to get SD: Gone, err code 410
Nov 02 22:31:30 ip-172-31-7-74 twingate-connector[1693]: State: Offline
Nov 02 22:31:30 ip-172-31-7-74 twingate-connector[1693]: State: Authentication
Nov 02 22:31:30 ip-172-31-7-74 twingate-connector[1693]: [ERROR] [libsdwan] operator(): failed HTTP request 4125923095300265675 410 Gone
Nov 02 22:31:30 ip-172-31-7-74 twingate-connector[1693]: State: Error

Instance in default VPC, with relaxed relaxed security group, that allow any traffic an and out.

Ping to twingate http is ok:

ubuntu@ip-172-31-7-74:~$ ping ven.twingate.com
PING ven.twingate.com (35.190.71.73) 56(84) bytes of data.
64 bytes from 73.71.190.35.bc.googleusercontent.com (35.190.71.73): icmp_seq=1 ttl=112 time=1.23 ms
64 bytes from 73.71.190.35.bc.googleusercontent.com (35.190.71.73): icmp_seq=2 ttl=112 time=1.31 ms
64 bytes from 73.71.190.35.bc.googleusercontent.com (35.190.71.73): icmp_seq=3 ttl=112 time=1.21 ms
64 bytes from 73.71.190.35.bc.googleusercontent.com (35.190.71.73): icmp_seq=4 ttl=112 time=1.47 ms

used ami: ami-0c0517c13ff45ee78
also tried with prev image ami-044ecab01f7b01803 - same error in logs

Hi @SebastianG - how did you run your docker container? I think this type of error can happen when the required environment variables are missing. Also, since you mentioned you’re on Hetzner - our CLI tool can deploy Hetzner VMs pre-configured with Twingate.There’s a YouTube video showing how here.

Hi @knok16 - I think in your case the issue may be different. Is it possible you regenerated your connector tokens after deploying the connector?

1 Like

Yes, you are right: look like I clicked refresh tokens button somewhere in the middle of setup. Right now I regenerated tokens and tried to setup connector one more time, and it worked: it shown as Connected on twingate site, I was able to create VM in private AWS network, expose it in my local client and ssh to it.

Thanks a lot!

1 Like

Hi @Emrul,

First of all thanks for getting back to me.

My setup in Hetzner is a bit different from that YouTube! video and I don’t think the errors that I am facing are related to Twingate at all as I discovered that the DNS resolution is flapping and Twingate’s report is actually genuine / expected.

I have provisioned a couple of server in US region, all Debian 11, a network 10.22.0.0/16 and two subsets, 10.22.0.0/24 and 10.22.1.0/22.

1 x NAT “Gateway” with a public IP on eth0 and a private IP like 10.22.0.2 on enp7s0, nothing complicated:

echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf && sysctl -p &&\
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE && \
iptables -A FORWARD -i eth0 -o enp7s0 -m state --state RELATED,ESTABLISHED -j
ACCEPT && \
iptables -A FORWARD -i enp7s0 -o eth0 -j ACCEPT

1 x Twingate server with private IP only (no public interface), 10.22.0.3 on enp7s0 with the following config:

root@htz-use1-din-kub-01:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5)

# Include files from /etc/network/interfaces.d:
# source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto enp7s0
iface enp7s0 inet dhcp
   post-up ip route add default via 10.22.0.1
   dns-nameservers 8.8.8.8 8.8.4.4

Again, a very basic setup with Docker:

#cloud-config
runcmd:
  - printf "auto enp7s0\niface enp7s0 inet dhcp\n    post-up ip route add default via 10.22.0.1\n    dns-nameservers 8.8.8.8 8.8.4.4\n" >> /etc/network/interfaces
  - systemctl enable systemd-resolved.service
  - systemctl start systemd-resolved.service
  - systemctl restart networking
  - apt update
  - apt upgrade -y
  - apt install ca-certificates curl gnupg lsb-release
  - mkdir -p /etc/apt/keyrings
  - curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  - apt update
  - apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
  - docker run -d --sysctl net.ipv4.ping_group_range="0 2147483647" --env TENANT_URL="https://cloudresty.twingate.com" --env ACCESS_TOKEN="[REMOVED]" --env REFRESH_TOKEN="[REMOVED]" --env TWINGATE_LABEL_HOSTNAME="`hostname`" --name "htz-use1-din-twn-01" --restart=unless-stopped $(docker run --help | grep -- --pull >/dev/null && echo "--pull=always") twingate/connector:1

This setup is throwing those errors that I’ve posted initially.

My second try was to actually get rid of the Docker and to use the Linux package (env vars and all the buzz), but I get the same type of errors.

Playing with it I discovered that the DNS resolution is flapping, meaning that 6-7 times out of 10 this fails. I discovered this when I tried to install K8s on a different host under the same private network where I 6-7 times out of 10 the apt-get was failing with the same 403 error.

So this isn’t really a Twingate issue at all.

You are right, if I install Twingate on a server with a Public IP attached it works perfectly, no issue at all, but it doesn’t behind a NAT and my feeling is that Hetzner’s network somehow contributes to these issues or I am doing something fundamentally wrong.

Hi @SebastianG - got it, I can suggest one thing to try: you can add echo ip route before the apt update and see if it differs between runs. I am wondering here if adding to /etc/network/interfaces like this will be deterministic (I suspect not).

If that is the case you can look at Networking Config Version 2 — cloud-init 22.3 documentation and setting network routes using cloud-config.

Hi @Emrul,
I’ve tried pretty much everything I could to be honest, the cloud-init bit it’s more for you to understand what’s behind that setup but I tried a couple of things on the box itself. I have even reached out eventually to Hetzner’s support team and I got a modest “sail away, we’re supporting just the hardware” and I gave up.

I mean this should have worked 100% or failed 100%, but instead 6 times out of 10 was failing. Don’t quite have that time to spend with this and they don’t seem to be bothered with such requests. I had more support from you even though it was crystal clear that it wasn’t a Twingate issue at all.

Returned to my 1st love, GCP and it works like a charm, in 30 minutes I was able to set up everything (VPC, Subnets, Cloud NAT, Twingate and a K8s cluster), so Hetzner it’s a bit of history now. Probably that would have been a nice to have leg in Hetzner, but I assume that fella didn’t quite had the best morning coffee.

Thank you very much for your help!

2 Likes