API-generated connector tokens are invalid

Support / Troubleshooting
1

I did an initial proof of concept with Twingate where I manually set everything up. As I’ve been moving forward towards rolling it out to our company, I’ve been re-configuring things using Pulumi.

I have three connectors deployed in Amazon ECS. A few weeks ago, one of them started failing to connect. I briefly looked and couldn’t figure out why, and then dismissed it for the time being. Slowly the other two have also failed, and gotten themselves in a state where they don’t function at all.

The log output from twingate is minimal, but indicates they can’t authenticate correctly:

State: Offline
State: Authentication
State: Error
State: Offline
State: Authentication
State: Error

What I’ve determined is that the token values Pulumi gives me from twingate.TwingateConnectorTokens are considered to be invalid by Twingate_com. If I log them from my Pulumi run and paste them into a direct run of the twingate/connector:1 container, it has an auth failure; if I manually generate tokens and substitute them in, that works. So it’s not a network firewall sort of problem, just a token one.

There isn’t really documentation of the API so it’s hard for me to try this through a method other than Pulumi. And I can’t view generated tokens on the twingate_com web interface to be able to compare and see if they match.

When I force a rotation of the tokens with the keepers parameter, the connector comes up successfully. However, after 10 minutes, it fails auth and goes offline. This makes me suspect there’s some sort of lifecycle problem, where the token is created but for some reason deleted by a cleanup process. It’s very consistently 10 minutes.

I don’t know how many tokens are associated with any of my connectors or any way to find out that information, but it seems like it would be useful information.

I may be able to create a minimal example, but it should theoretically happen with pulumi-twingate/__main__.py at master · Twingate-Labs/pulumi-twingate · GitHub if you specify keepers. Here’s my definition:

connector_tokens = twingate.TwingateConnectorTokens(
    resource_name=_environment_name,
    connector_id=connector.id,
    # When any of these values change, it will trigger a credential rotation.
    keepers={
        'year': datetime.now().year,
        'month': datetime.now().month,
        #'foo': 'bar',
    },
)

Edit: I also just tried it again without keepers specified at all, and the same “10 minutes online and then auth failure” cycle happened.

2

it looks like the issue might be caused by the connector token reuse (or regenerate), which void the previous generated connector token.

in essence, once you generate a connector token for a connector, you should:

  1. only use this token once to deploy 1 connector
  2. not generate token for this connector again, otherwise the previous token will be voided
3

Yes, that sounds about right. There’s no way (that I’ve found) to list generated tokens for a connector, so I can’t verify that’s what’s happening, but from a blackbox perspective it seems likely to be the problem.

However, that should be happening automatically. I’m using twingate.TwingateConnectorTokens | Pulumi Registry , and so when a new token is generated, I would expect the underlying Terraform provider to delete the older token; I think that isn’t happening. So maybe I need to file a bug against the terraform provider .

The API strongly suggests this should be supported. Not only by the lack of any mention of only being able to have a single token per connector, but because there’s a parameter (keepers) that auto-rotates your tokens (I had set it up to do so monthly). If that weren’t intended to be supported in the product, then it wouldn’t make sense to have added that feature to the SDK.