| Follow @lancinimarco | Subscribe to CloudSecList

Reading time ~11 minutes

Remotely Access your Kubernetes Lab with Cloudflare Tunnel

Back in April, I saw a post on the Cloudflare tech blog which explained how Auditable Terminal gives you a fully featured SSH client in your browser: you authenticate using Cloudflare Access, and can log into a computer - and get a terminal - just using a browser.

The post made me curious, but at the time I didn’t have capacity to look into it. Until now, when I decided to give it a go for my home lab.


The Environment: Kubernetes Lab on Baremetal

In Kubernetes Lab on Baremetal, I detailed the steps I took to deploy my own Kubernetes Lab on baremetal, and on an Intel NUC in particular.

With this post, I want to take the setup a step further, using Cloudflare Tunnel to access the cluster remotely.

The end goal is to use Cloudflare Tunnel to connect my Intel NUC to the Cloudflare network, and to use Auditable Terminal to connect to it using nothing more than a browser.

The remainder of this post will assume you have a server running Fedora CoreOS (or Linux in general) as operating system, as outlined in Kubernetes Lab on Baremetal.


Access the Host

From the Cloudflare’s documentation:

Cloudflare Tunnel runs a lightweight daemon (cloudflared) in your infrastructure that establishes outbound connections (Tunnels) between your service and the Cloudflare edge. When Cloudflare receives a request for your chosen hostname, it proxies the request through those connections to cloudflared. In turn, cloudflared proxies the request to your applications.

Cloudflare Tunnel. Image courtesy of Cloudflare
Cloudflare Tunnel. Image courtesy of Cloudflare

Basically, Cloudflare Tunnel requires the installation of a lightweight (and open source) server-side daemon (called cloudflared), to connect your infrastructure to Cloudflare.

Let’s see how to use it to access the Intel NUC remotely.

Create a Cloudflare Tunnel

Let’s start by SSH-ing into the host and downloading cloudflared for Linux:

[[email protected] ~]$ curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o cloudflared
[[email protected] ~]$ chmod +x cloudflared
[[email protected] ~]$ sudo mv cloudflared /usr/local/bin/

Once downloaded, we need to authenticate the cloudflared daemon: issue the cloudflared tunnel login command and follow the instructions to authenticate against your Cloudflare account. After logging into your account, select the domain you want to use:

[[email protected] ~]$ cloudflared tunnel login
Please open the following URL and log in with your Cloudflare account:

https://dash.cloudflare.com/argotunnel?callback=<redacted>

Leave cloudflared running to download the cert automatically.
2021-08-25T19:46:03Z INF Waiting for login...
2021-08-25T19:46:56Z INF Waiting for login...
You have successfully logged in.
If you wish to copy your credentials to a server, they have been saved to:
/var/home/core/.cloudflared/cert.pem
Select a domain Authorize cloudflared

Selecting a domain will automatically generate a cert.pem file in the default .cloudflared directory, which contains account-wide credentials.

We are now ready to create a tunnel (named nuc in this case):

[[email protected] ~]$ cloudflared tunnel create nuc
Tunnel credentials written to /var/home/core/.cloudflared/1111-2222-3333.json.
cloudflared chose this file based on where your origin certificate was found.
Keep this file secret. To revoke these credentials, delete the tunnel.

Created tunnel nuc with id 1111-2222-3333

This command will generate a credentials file and a tunnel, by establishing a persistent relationship between the name we chose (i.e., nuc) and a UUID (i.e., 1111-2222-3333) for the tunnel. At this point, no connection is active within the tunnel yet.

The last component missing to instaurate a connection is a configuration file (.cloudflared/config.yml), which will instruct the tunnel to route traffic from a given origin to the chosen hostname:

# [[email protected] ~]$ cat .cloudflared/config.yml
tunnel: 1111-2222-3333.json                                       # the UUID of the tunnel
credentials-file: /var/home/core/.cloudflared/1111-2222-3333.json # the credentials file generated

ingress:
  - hostname: nuc.marcolancini.it                                 # the hostname to use
    service: ssh://localhost:22
  - service: http_status:404
  # Catch-all rule, which responds with 404 if traffic doesn't match any of
  # the earlier rules

To start routing traffic, assign a CNAME record that points traffic to the tunnel’s subdomain:

[[email protected] ~]$ cloudflared tunnel route dns 1111-2222-3333 nuc.marcolancini.it
2021-08-25T19:58:10Z INF Added CNAME nuc.marcolancini.it which will route to this tunnel tunnelID=1111-2222-3333

You can now finally run cloudflared to proxy incoming traffic from the tunnel to any number of services running locally:

[[email protected] ~]$ cloudflared tunnel --config ~/.cloudflared/config.yml run nuc
2021-08-25T20:01:14Z INF Starting tunnel tunnelID=1111-2222-3333
2021-08-25T20:01:14Z INF Version 2021.8.3
2021-08-25T20:01:14Z INF GOOS: linux, GOVersion: devel +11087322f8 Fri Nov 13 03:04:52 2020 +0100, GoArch: amd64
2021-08-25T20:01:14Z INF Settings: map[config:/var/home/core/.cloudflared/config.yml cred-file:/var/home/core/.cloudflared/1111-2222-3333.json credentials-file:/var/home/core/.cloudflared/1111-2222-3333.json]
2021-08-25T20:01:14Z INF Generated Connector ID: <redacted>
2021-08-25T20:01:14Z INF cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/argo-tunnel/reference/service/
2021-08-25T20:01:14Z INF Initial protocol http2
2021-08-25T20:01:14Z INF Starting metrics server on 127.0.0.1:36423/metrics
2021-08-25T20:01:16Z INF Connection <redacted> registered connIndex=0 location=AMS
2021-08-25T20:01:16Z INF Connection registered connIndex=1 location=LHR
2021-08-25T20:01:17Z INF Connection registered connIndex=2 location=AMS
2021-08-25T20:01:18Z INF Connection registered connIndex=3 location=LHR
[...]

You can also check the status of the tunnel:

[[email protected] ~]$ cloudflared tunnel list
ID                  NAME   CREATED                CONNECTIONS
1111-2222-3333      nuc    2021-08-25T19:51:49Z   2xAMS, 2xLHR

Create a Zero Trust Policy

We do have a connection instaurated, but how can we connect to the host remotely then? The answer is Cloudflare Access.

If you don’t already have a Cloudflare for Teams account, you can visit https://dash.teams.cloudflare.com/ and follow the setup guide. The free plan is sufficient.

Once you have an account, navigate to the Cloudflare for Teams dashboard to create a new application. Select the Applications page from the sidebar, and then Add application:

Add an Application
Add an Application

On the next page, choose Self-hosted as application type:

Self-hosted Application
Self-hosted Application

Give the application a name and specify the subdomain that will become the hostname where the application will be accessible via. Note that it will have to match the hostname specified in the .cloudflared/config.yml file:

Application Details
Application Details

As a next step, we need to create rules that control who should be allowed access to the application (in a Zero-Trust style). In this case, we will add an email filter, specifying that only people with a @marcolancini.it email address will be able to authenticate:

Application Rules
Application Rules

Finally, let’s enable “Browser Rendering”: Cloudflare can render an SSH client in your browser without the need for client software or end user configuration changes. Once enabled, when users authenticate and visit the URL of the application, Cloudflare will render a terminal in their browser.

To enable it, navigate to the application page of the Access section in the Cloudflare for Teams dashboard. Click Edit and select the Settings tab. In the cloudflared settings card, select SSH from the “Browser Rendering” drop-down menu:

Application Settings
Application Settings

If we try to connect now, we will see that password-based authentication is disabled (gladly!), in favour of key-based authentication. You’ll be presented with a text form where to enter the host’s private key:

Cloudflare Launcher
Cloudflare Launcher
Login - User Login - Key

If, like me, you don’t like to copy-paste keys around, it’s time to setup short-lived certificates.

Configure Short-Lived Certificates

Cloudflare Access can replace traditional SSH key models with short-lived certificates issued to users based on the token generated by their Access login. The SSH server can then use that certificate to start the session.

Let’s generate a short-lived certificate public key. On the Teams dashboard, navigate to Access > Service Auth, then, in the drop-down, choose your application (NUC in this case):

Create a Public Key
Create a Public Key

After clicking on Generate certificate, a row will appear with a public key scoped to the application. Save this key as a new .pub file in the host’s filesystem (e.g., /etc/ssh/ca.pub).

Next, Cloudflare Access requires two changes to the /etc/ssh/sshd_config file used on the target host:

  1. PubkeyAuthentication should not be commented out, and should be set to yes (e.g., PubkeyAuthentication yes).
  2. Add a new line which will trust the newly created key: TrustedUserCAKeys /etc/ssh/ca.pub.

After these changes, restart the SSH server:

[[email protected] ~]$  systemctl restart sshd

Ensure Unix usernames match user SSO identities

It is important to note that the Cloudflare Access user should match a Unix username: Cloudflare Access will take the identity from a token and, using short-lived certificates, authorize the user on the host. Access matches based on the identity that precedes an email domain. Hence, Unix usernames must match the identity preceding the email domain.

For example, if the user’s identity is [email protected], then Cloudflare Access will look to match that identity to the Unix user marco. If you don’t have such a user, create one (e.g., sudo adduser marco).

If you try to connect and authenticate now, Cloudflare will render a terminal in your browser (notice the user marco).

Cloudflare Auditable Terminal
Cloudflare Auditable Terminal

Run cloudflared as a Service

Finally, to avoid having to manually launch the agent, Cloudflare Tunnel can install itself as a system service:

[[email protected] ~]$ sudo cloudflared service install

The configuration files in ~/.cloudflared/ will then automatically be copied to /etc/cloudflared/.


Subscribe to CloudSecList

If you found this article interesting, you can join thousands of security professionals getting curated security-related news focused on the cloud native landscape by subscribing to CloudSecList.com.


Access Kubernetes Services

The interesting fact is that Cloudflare Tunnel can also be used to route traffic to Kubernetes clusters. Let’s see how.

First of all, let’s deploy a sample service which we want to access remotely:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-deployment
  namespace: test
spec:
  selector:
    matchLabels:
      app: httpbin
  replicas: 1
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
        - name: httpbin
          image: kennethreitz/httpbin:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web-service
  namespace: test
spec:
  selector:
    app: httpbin
  ports:
    - protocol: TCP
      port: 80
➜ kubectl apply -f app.yaml
deployment.apps/httpbin-deployment created
service/web-service created

Then, let’s create a new tunnel dedicated to the test deployment, similarly as we did before:

[[email protected] ~]$ cloudflared tunnel create test-k8s
Tunnel credentials written to /var/home/core/.cloudflared/4444-5555-6666.json.
cloudflared chose this file based on where your origin certificate was found.
Keep this file secret. To revoke these credentials, delete the tunnel.

Created tunnel test-k8s with id 4444-5555-6666

With the tunnel created, we now need to share with the Kubernetes cluster the file containing the credentials of the tunnel. We can do so by creating a Secret:

[[email protected] .cloudflared]$ kubectl -n test create secret generic tunnel-credentials --from-file=credentials.json=/var/home/core/.cloudflared/4444-5555-6666.json
secret/tunnel-credentials created

[[email protected] .cloudflared]$ kubectl -n test get secret
NAME                  TYPE                                  DATA   AGE
default-token-72zr2   kubernetes.io/service-account-token   3      5m51s
tunnel-credentials    Opaque                                1      10s

Next, we need to associate the tunnel with a DNS record:

[[email protected] .cloudflared]$ cloudflared tunnel route dns 4444-5555-6666 k8s.marcolancini.it
2021-08-26T20:27:06Z INF Added CNAME k8s.marcolancini.it which will route to this tunnel tunnelID=4444-5555-6666

Finally, we need to deploy cloudflared in the cluster. The manifest below will generate a Deployment for cloudflared, alongside with a ConfigMap for its config:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: test
spec:
  selector:
    matchLabels:
      app: cloudflared
  replicas: 1
  template:
    metadata:
      labels:
        app: cloudflared
    spec:
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:2021.8.3
          args:
            - tunnel
            - --config
            - /etc/cloudflared/config/config.yaml
            - run
          volumeMounts:
            - name: config
              mountPath: /etc/cloudflared/config
              readOnly: true
            - name: creds
              mountPath: /etc/cloudflared/creds
              readOnly: true
      volumes:
        - name: creds
          secret:
            secretName: tunnel-credentials
        - name: config
          configMap:
            name: cloudflared
            items:
              - key: config.yaml
                path: config.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared
  namespace: test
data:
  config.yaml: |
    tunnel: test-k8s
    credentials-file: /etc/cloudflared/creds/credentials.json
    # metrics: 0.0.0.0:2000
    no-autoupdate: true
    ingress:
      - hostname: k8s.marcolancini.it
        service: http://web-service:80
      - service: http_status:404
➜ kubectl apply -f cloudflared.yaml
deployment.apps/cloudflared created
configmap/cloudflared created

➜ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
cloudflared-6fd757bfcd-lwcgv          1/1     Running   0          3m16s
httpbin-deployment-67f749774f-l587w   1/1     Running   0          27m

➜ kubectl logs $(kubectl get pod -l app=cloudflared -o jsonpath="{.items[0].metadata.name}")
2021-08-26T20:49:01Z INF Starting tunnel tunnelID=4444-5555-6666
2021-08-26T20:49:01Z INF Version
2021-08-26T20:49:01Z INF GOOS: linux, GOVersion: go1.16.4, GoArch: amd64
2021-08-26T20:49:01Z INF Settings: map[config:/etc/cloudflared/config/config.yaml cred-file:/etc/cloudflared/creds/credentials.json credentials-file:/etc/cloudflared/creds/credentials.json metrics:0.0.0.0:2000 no-autoupdate:true]
2021-08-26T20:49:01Z INF Generated Connector ID: b47e1cc3-96df-4f9f-b0b4-2b491f69990a
2021-08-26T20:49:01Z INF Initial protocol http2
2021-08-26T20:49:01Z INF Starting metrics server on [::]:2000/metrics
2021-08-26T20:49:01Z INF Starting Hello World server at 127.0.0.1:33855
2021-08-26T20:49:03Z INF Connection 519adf31-8b54-4c58-9aba-956e35548003 registered connIndex=0 location=AMS
{"level":"debug","time":"2021-08-26T20:49:03Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
2021-08-26T20:49:03Z INF Connection 8964c25c-ab8e-41a8-ba2f-55d3eabd3035 registered connIndex=1 location=LHR
{"level":"debug","time":"2021-08-26T20:49:04Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
2021-08-26T20:49:04Z INF Connection 50d3d0f7-98fc-4174-b127-8fdf7bb428b7 registered connIndex=2 location=AMS
{"level":"debug","time":"2021-08-26T20:49:05Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
2021-08-26T20:49:06Z INF Connection b58d388f-f9d9-4162-a51c-d2104149cb35 registered connIndex=3 location=LHR

When Cloudflare receives traffic for the hostname you configured in the previous step, it will send that traffic to the cloudflared instances running in this deployment. Then, those cloudflared instances will proxy the request to the application’s Service.

Kubernetes Service exposed by Cloudflare Tunnel
Kubernetes Service exposed by Cloudflare Tunnel

Automate with Terraform

Let’s take this a step further, and let’s automate it with Terraform. Cloudflare has a decent Terraform provider which can be used to automate the creation of the application and DNS bindings:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# ==============================================================================
# VARIABLES
# ==============================================================================
variable "domain" {
  default = "example.com"
}

variable "zone_id" {
  default = <CLOUDFLARE_ZONE_ID>
}

# ==============================================================================
# APPLICATION
# ==============================================================================
resource "cloudflare_access_application" "cf_app_k8s" {
  zone_id          = var.zone_id
  name             = "K8s Example"
  domain           = "k8s.${var.domain}"
  session_duration = "24h"
}

resource "cloudflare_access_policy" "cf_policy" {
  application_id = cloudflare_access_application.cf_app.id
  zone_id        = var.zone_id
  name           = "Email Filter"
  precedence     = "1"
  decision       = "allow"

  include {
    email = ["[email protected]${var.domain}"]
  }
}

resource "cloudflare_record" "cf_app_k8s_cname" {
  zone_id = var.zone_id
  name    = "k8s"
  value   = "<redacted>.cfargotunnel.com"
  type    = "CNAME"
  proxied = "true"
  ttl     = 1
}
K8s Application created by Terraform
K8s Application created by Terraform

Conclusions

In this blog post, part of the “Kubernetes Primer for Security Professionals” series, I described how to use Cloudflare Tunnel to connect my Intel NUC to the Cloudflare network, and Auditable Terminal to connect to it using nothing more than a browser.

I hope you found this post useful and interesting, and I’m keen to get feedback on it! If you find the information shared was useful, if something is missing, or if you have ideas on how to improve it, please let me know on Twitter.

Subscribe to CloudSecList

If you found this article interesting, you can join thousands of security professionals getting curated security-related news focused on the cloud native landscape by subscribing to CloudSecList.com.

Marco Lancini

Marco Lancini
Hi, I'm Marco Lancini. I'm a Security Engineer, mainly interested in cloud native technologies and security...  [read more]