I’ve setup a twingate resource as the FQDN to a database in azure.
The twingate connector is depolyed and attached to a vnet, the database has a private endpoint to that vnet.
When I connect to the database using the azure portal, i.e a via the browser, or using Azure DataStudio, which is basically an electron app (a browser), the connection works. I can establish a connection. I can see the connection being relayed in the resource logs of the twingate portal.
However, attemtping to make the same connection using a cli tool in the terminal or running a development instance of an application locally fails. By development instance I mean, running an ASP.NET app locally that is dependent on the remote database.
Looking at the logs, I can see an initial connection is made to the connector which lasts for 1 second, everything is green.
However the resource returns a forbidden response. The error indicates that our local client ip is being blocked. As in, the connection isn’t being relayed through teh twingate connector, but we are connecting directly despite having twingate running and an initial connection to the connector being logged.
This is affecting all our developers on both windows and mac.
In this instance - when the connection “fails” - what kind of response do you get if you do nslookup [the hostname in question] - does it return the “real” public IP of the Azure DB server or does it return a CGNAT IP (ie 100.x.x.x)?
If you do a nslookup [hostname in question] while NOT connected to Twingate, and add that public IP (or multiple IPs) as Twingate resources, and then reconnect to Twingate, does the behaviour continue?
I’m not intimately familiar with ASP environments so I’m not too certain how things might be configured, but is there an “internal” DNS server that runs and may need to be disabled so that the system is the primary?
This is what’s so confusing. If I run nslookup then the ip is 100.x.x.x when twingate is running and when I attempt a connection I can also see the connection in the resource logs and everything is green, see screenshot 1.
However, the connection is actually rejected by the resource which responds
Request originated from IP [my client IP not the twingate connector ip] through public internet. This is blocked by your Cosmos DB account firewall settings.
If the connection is being relayed through twingate as the logs suggest and nslookup is showing a twingate IP, then why is my public ip being exposed to the resource and not the connector IP?
We get a similar issue when working with Azure SQL Servers, but the issue is more confusing and frustrating. When we connect to the sql server using Azure DataStudio or Datagrip Screenshot 2, the connection is successful and that is reflected in the twingate logs.
However running an ASP[dot]NET api locally and attempting the same connection fails with the message stating that the firewall has blocked our local client IP, yet the connection is showed as relayed in the admin panel, screenshot 3. Again, it begs the question, why is our local IP being exposed to the service that is part of the trust network and registered as a resource?
Cannot open server ‘[server name]’ requested by the login. Client with IP address ‘[local client ip]’ is not allowed to access the server.
If you do a nslookup [domain] while not connected to Twingate - you’re going to see a couple other records come back with it as CNAME entries - likely [same as your main host].privatelink.documents.azure.com and [something].cloudapp.net
If you add Twingate resources for both of those hostnames, log out, do an ipconfig /flushdns (if Windows machine) and then log back in, do you still see the same behaviour?
I think this might resolve it actually. It seems there is a redirect as you have pointed out. I am going to do some further updates and testing across our stack and will confirm . Thanks for the help thus far.
I followed this solution, created extra resources using the additional cname records revealed by running nslookup against the resource. It seems to have resolved for me running on Mac but our windows users are still getting the same error.
If you look at your Twingate traffic logs, are you seeing those new hostnames get intercepted by Twingate for the Windows users even though it’s still not working? Have they done an ipconfig /flushdns?
Digging into this and there seems to an issue with how Azure SQL server handles connections. It seems Azure proxies and redirects connections to SQL Server, which is shown when you do nslookup.
My challenge now is how to mitigate this, becuase the end node and the IP of the Server changes. I am not sure how to make it fixed or compensate.
Looking at the default conectivity setting and it says
Default - Uses Redirect policy for all client connections originating inside of Azure (except Private Endpoint connections) and Proxy for all client connections originating outside Azure
I don’t understand what it means with regards to private endpoints, twingate is connected via a private endpoint.
There are 3 connectivity options but I am not sure if they make a difference.
Go to the Azure SQL server just setup and go to networking tab:
Set public network to Selected Access
Setup a private endpoint to the vnet hosting the connector
Add the database url and private link url as a twingate resource
log in to twingate on your machine
Run the following .net test script:
using Dapper;
using Microsoft.Data.SqlClient;
var connectionString ="";
using IDbConnection db = new SqlConnection(connectionString);
var query = @"select s.name as schema_name,
s.schema_id,
u.name as schema_owner
from sys.schemas s
inner join sys.sysusers u
on u.uid = s.principal_id
order by s.name";
var schemas = db.Query(query);
Expected behaviour with default connectivity:
The database allows the connection. On mac this works, on windows it fails and your client IP is exposed instead of the twingate one.
Change Azure SQL server connectivity to Proxy:
On mac and windows this now works but results in greater latency so is not ideal.
Change Azure SQL server connectivity to redirect:
Mac works, windows fails to connect exposing the client IP.
In all cases, the resource logs shows a connection. When Mac connects successfully the connection is shown to last several seconds. When connected on windows the connection is successful but closes within 1 second and then the redirect kicks in which twingate is not picking up.
Unfortunatly this issue is more pronounced trying to connect to Azure cosmos databases when do local development.
The cosmos DB doesn’t allow you to change connectivity.
I have set this up as stated before, with a private endpoint and resource setup, I can view the DB explorer in the browser but trying to connect doing nodejs development gets blocked, I get a response saying my local IP is blocked.
For cosmos… I had to specify both the main endpoint and the regional endpoint before vscode would connect. Cosmos splits its operations (read/write) to diff endpoints. Crossref your privatelink zone file for your case, but I had to have both the main an the secondary region associated with our instance. In our case both: xxxx.documents.azure.com and xxxx-southcentralus.documents.azure.com
Anytime where you believe you’ve specified a native Azure resource name correctly but getting nowhere… sometimes you can get away with doing a resource entry with an asterisk in the name like in this case *.documents.azure.com would have logged hits for both endpoints – and allowed me to discover.
This isn’t a workable solution. We have contractors who use their own device and they have other Azure projects. The point of twingate is to secure our infrastructure, split tunneling by default and all that, we don’t want to route all traffic through our connectors, only traffic related to our resources. The solution you describe @sduys would result in us capturing traffic we don’t want.
. Thanks for the help thus far.