How to rotate Teleport join tokens to allow active nodes to auto-join

Overview

Teleport is an excellent tool to manage access of infrastructure in a secure and easy method without relying on long lived certs and keys.

In this article we will be dealing with a very prominent use case of Teleport where created machines need to be automatically joined using a join token. However, currently there is no internal method where Teleport itself can post short lived join tokens to common secret layers such as Google Secrets, Kubernetes Secrets, AWS Secrets, and Hashicorps Vault.

Getting Started

Follow instructions to install Teleport in HA mode. In our case at **@GluuFederation**, we used Teleport on GKE and we used a Kubernetes CronJob to push to a Google Secret. New nodes would pull from this Google Secret and setup Teleport with the token automatically.

The Wrong Solution

We started off thinking to use the Teleport image inside the [**Kubernetes CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) in order to avoid authenticating via an [identity generated file](https://goteleport.com/docs/server-access/guides/tsh/#identity-files). The reason behind that was that we thought since the ***CronJob*was living inside the same namespace as the Teleport cluster we could better secure and generate tokens without the possibility of forgetting about the expiration of the identity file which would lead to possible downtime if we somehow forgot about rotating it. This is how the ***CronJob*looked, the main issue is in using image: quay.io/gravitational/teleport:9.3.8**:

# Rotate join tokens by pushing to Google Secret
apiVersion: batch/v1
kind: CronJob
metadata:
  annotations:
    meta.helm.sh/release-name: teleport-join-token-rotater
    meta.helm.sh/release-namespace: teleport
  labels:
    app: teleport
    app.kubernetes.io/managed-by: Helm
  name: teleport-join-token-rotater
  namespace: teleport

spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
      labels:
        app: teleport-join-token-rotater
      name: teleport-join-token-rotater
    spec:
      template:
        metadata:
          creationTimestamp: null
          labels:
            app: teleport-join-token-rotater
          name: teleport-join-token-rotater
        spec:
          containers:
          - command:
            - /bin/sh
            - -c
            - |
              teleport start --diag-addr=0.0.0.0:3000 &
              sleep 30
              apt-get update
              apt-get install curl -y
              apt-get install apt-transport-https ca-certificates gnupg -y
              echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
              curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
              apt-get update && apt-get install google-cloud-sdk -y
              gcloud auth activate-service-account --key-file=/etc/teleport-secrets/gcp-credentials.json --project=<gcp-project>
              tctl tokens add --type=node --ttl=1h | grep "token:" | awk '{print $4}' | sed 's/..$//' | gcloud secrets versions add teleport_join_token --data-file=-
            image: quay.io/gravitational/teleport:9.3.8
            imagePullPolicy: IfNotPresent
            name: teleport
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /etc/teleport-secrets
              name: gcp-credentials
              readOnly: true
            - mountPath: /etc/teleport-tls
              name: teleport-tls
              readOnly: true
            - mountPath: /etc/teleport
              name: config
              readOnly: true
            - mountPath: /var/lib/teleport
              name: data
          dnsPolicy: ClusterFirst
          restartPolicy: OnFailure
          terminationGracePeriodSeconds: 30
          volumes:
          - name: gcp-credentials
            secret:
              defaultMode: 420
              secretName: teleport-gcp-credentials
          - name: teleport-tls
            secret:
              defaultMode: 420
              secretName: teleport-tls
          - configMap:
              defaultMode: 420
              name: teleport
            name: config
          - emptyDir: {}
            name: data
  # Every Hour
  schedule: "0 * * * *"
  successfulJobsHistoryLimit: 3
  suspend: false

This was automated with Terraform and we made a Helm chart out of this file so execution via Terraform and maintenance would be easier. However for a quick test you can save the above in a file called ***wrong-teleport-join-token-rotater.yaml, edit the `<gcp-project>`*** by placing your gcp project id and then apply it.

kubectl apply -f wrong-teleport-join token-rotater.yaml

Whats wrong with the above ?

What happens here is that the token is actually generated correctly and also posted to the secrets accurately. However, when the newly created vm machine pulls the token, the actual Teleport short lived cluster that the ***CronJob*above created and was used to generate the token would be already out of service. For some reason which I will leave for the @goteleport folks to comment on, that short lived cluster that generated the token cannot be systematically going down and up with a new UID. According to our `CronJob`** that would be 24 times a day.

The initial tokens that are produced may work initially but soon after you will see the following in the Teleport logs of the VM machines:

Mar 09 12:40:01 [my.testing.com](http://my.testing.com/) teleport[11115]: User Message: invalid character '<' looking for beginning of value] auth/register.go:171

The Right Solution

1. Login to Teleports UI and create a new role with the least privilege access model in mind. Head to Team → Roles→ Create New Role, and copy the below yaml, and save your changes. This will create a role called tokenwith the ability to generate join tokens.

kind: role
metadata:
  description: Generate join tokens
  name: token
spec:
  allow:
    rules:
    - resources:
      - token
      verbs:
      - create
  deny: {}
  options:
    cert_format: standard
    enhanced_recording:
    - command
    - network
    forward_agent: true
    max_session_ttl: 30h0m0s
    port_forwarding: true
version: v4

2. Create a new user called token-rotaterwith role token. Head to Team → Users→ Create New User, and follow with the instructions until the user is fully setup.

3. Generate an identity file for the user token-rotater inside one of the Teleport pods. Set the ttl value to something reasonable, and add a tracker to remind you to rotate this identity file or else your CronJobwill no longer post join tokens.

# ttl = 60 days
kubectl exec -ti $(kubectl get pods --selector=app=teleport --output=jsonpath={.items[0]..metadata.name} -n <teleport-namespace>) -n <teleport-namespace> -- tctl auth sign --ttl=1440h--user=token-rotater--out=token.pem

4. Save the output file inside your Kubernetes secrets and make sure to remove that file from inside the pod or restart Teleport pods.

kubectl create secret generic token-rotater-user-identity --from-file=token.pem -n <teleport-namespace>

5. Apply the below CronJob, that uses ubuntu:20.04 as its base image instead of quay.io/gravitational/teleport:9.3.8 .

apiVersion: batch/v1
kind: CronJob
metadata:
  annotations:
    meta.helm.sh/release-name: teleport-join-token-rotater
    meta.helm.sh/release-namespace: teleport
  labels:
    app: teleport
    app.kubernetes.io/managed-by: Helm
  name: teleport-join-token-rotater
  namespace: teleport
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
      labels:
        app: teleport-join-token-rotater
      name: teleport-join-token-rotater
    spec:
      template:
        metadata:
          creationTimestamp: null
          labels:
            app: teleport-join-token-rotater
          name: teleport-join-token-rotater
        spec:
          containers:
          - command:
            - /bin/sh
            - -c
            - |
              apt-get update
              apt-get install curl -y
              # Install teleport
              curl https://get.gravitational.com/teleport-v9.3.8-linux-amd64-bin.tar.gz.sha256
              # <checksum> <filename>
              curl -O https://get.gravitational.com/teleport-v9.3.8-linux-amd64-bin.tar.gz
              # Verify that the checksums match
              tar -xzf teleport-v9.3.8-linux-amd64-bin.tar.gz
              cd teleport
              ./install
              cd ..
              apt-get install apt-transport-https ca-certificates gnupg -y
              echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
              curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
              apt-get update && apt-get install google-cloud-sdk -y
              gcloud auth activate-service-account --key-file=/etc/teleport-secrets/gcp-credentials.json --project=<gcp-project>
              tctl tokens add --type=node --ttl=2h --identity=/etc/token.pem --auth-server=teleport-dev.gluu.org:443 | grep "token:" | awk '{print $4}' | tr -d "." > token
              gcloud secrets versions add teleport_join_token --data-file=token
              sleep 5
            image: ubuntu:20.04
            imagePullPolicy: IfNotPresent
            name: teleport
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /etc/token.pem
              name: token-rotater-user-identity
              subPath: token.pem
              readOnly: true
            - mountPath: /etc/teleport-secrets
              name: gcp-credentials
              readOnly: true
          dnsPolicy: ClusterFirst
          restartPolicy: OnFailure
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
          volumes:
          - name: token-rotater-user-identity
            secret:
              secretName: "token-rotater-user-identity"
          - name: gcp-credentials
            secret:
              defaultMode: 420
              secretName: teleport-gcp-credentials
  schedule: 0 * * * *
  successfulJobsHistoryLimit: 3
  suspend: false

In Conclusion

To post join tokens the user must avoid using short lived Teleport pods to generate the join token and instead should use an identity generate file for a user with least privilege and use that to generate the token. Original post is here.