Deploying datastores for IoT & Big Data: mongoDB on K8s. Part 2

🎬 Introduction

This blog post series is intended to give an overview of how datastores capable of supporting high volumes of data from IoT devices and Big Data services can be deployed on Kubernetes. In the first part of this series, the StatefulSet primitive has been used to set up and deploy a mongoDB Replica Set (cluster). This part, part 2, demonstrates how other Kubernetes primitives such as Secret can be applied to secure our initial, dummy deployment. Part 3 of this series explains how to shard and further secure our mongoDB cluster.

Prerequisites

It is assumed that you already have an up and running K8s environment, such as minikube. All the examples have been developed using minikube on macOS Catalina with VirtualBox.

For part 2 we will be using the sec-datastores K8s namespace. The same headless Service and ConfigMap that we used on Part 1 need to be created under this namespace.

Security Requirements

The following requirements are under the scope of this series part:

  • The communication between replicas must be authenticated.
  • DB clients must be authenticated. A user/pass scheme is acceptable.
  • DB clients must connect to the DB through a secure and trusted channel (TLS).

The following requirements are not under scope but might be developed in future parts of this series:

  • The communication between replicas must take place through an authenticated channel based on TLS.
  • The communication between the DB and DB clients must be through mutual-TLS.

🏰 Securing mongoDB Replication

Generating a key for the Replica Set

The first step to ensure authentication of the replicas is to define a secret replica key to be presented to each other when submitting replication deltas. A new random key (of 1024 characters) can be generated and encoded in base64 as follows:

openssl rand 756 | base64
T3VTeVZHL1EJwllZVaCjhSqtqapViaFGK5vifxWyshnBXdDBP8SqHFz/ ....

Generating a password for the mongoDB root user

We can generate a cryptographically secure password of 16 chars for the root user as follows:

export LC_CTYPE=C
openssl rand 4096 | tr -cd '[:alnum:];@$#' | head -c 16; echo
Lqyr8CvuWsuoSFCN

Creating K8s Secret for mongoDB

The password of the root user (jmcf) and the replica key shall be stored on a K8s Secret.

apiVersion: v1
kind: Secret
metadata:
  name: mongo-secret
  namespace: sec-datastores
type: Opaque
stringData:
  root-user: jmcf
  root-pwd:  Lqyr8CvuWsuoSFCN
  replica.key: "T3VTeVZHL1EJwllZVaCjhSqtqapViaFGK5vifxWyshnBXdDBP8SqHFz/cG7OFdFLRi2l2se50ThfL1nnlm7UC4uTpl8kBMxI3wdbZdHD6he0x36ghNrYDVyTMbWPNqqH8qYfutojwEum0vHyyIaZadMfuVhPmsQlAv+TqnIVTHXY1vZx8GiKfGiUafyLGRhrPmh2w2+sZXjl1lj3+DLj730axtlDpEf1mTGkB0OKArip4Wply2FuHYFdXYGLuemRegLB+ShbyvqORqegfRou+by5dYflNj/7TnIOCPEzeunPgbsJBVbgVptVGk6pYNIrnPvtZOS/P6u0L/0c1J1EZyQ9wnXfu6xZpZSPg46WC0IznHMwe/3I8Z/YUiOY5+Oq/DFFLIRJaMblppNV6p+30z0YV7HRYW1eVHnkSwoi5m6o43B/cUidVhueUV/z0c1ZpojL5gfUSpu2YZNjWa6kVZ1+nXVmtDT7HqEZ1o5GjNMYkQHZ6ysJW7wbmqg77tP+KGvpEMvc0f0HLQLd1KPJ151URM5HvxDG3F7erPPI3dszw0k0OEGu/504h3sfYdreFEYZYa0vr8w2TuEJymbwWippNqNOvZQsrmLX6JXfVwMcxad27y+DwUXwj8jNoWy2lriak9xqaKbid0qitSp3+1WVgiffL/sZBOeoGsgdqop1dNdOCSsGwuzlVpZ5TqtklSdc0R+V0wuiROe086icbuVtz26OcSAkAE14WTvn3ID2/b6v7VyIrLpT2ifPGSEgyXNK0d0dFiCmOo9ewF+F6G1FX0yn0Ra5S8ZEG4BrAy8LCiCSxnNxehEW5aeEfD6Vsu08dl1I6/Khigu1DJnptKRpSR8vzNuOgtR1fi2CJFdKL2sZZI5K7wpQ5DuBgSCHpxA4qQf5e5vxgshFRXiVklRu11Ud/6Ra1CmdU5KOJc8M4t0tj3yCrFplBO8fxZ52CtRYTKpVTqcSrIbxlMbU/jmQGvbpSEjhBZ9iLszugIwbbsEq"

📌  The Secret data properties will be made available to the mongoDB containers through a mounted Volume.

Securing StatefulSet of Part 1

In part 1 we defined an initial version of our StatefulSet that can be extended as follows:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo-db-statefulset
  labels:
    mongoDB-replica: "true"
    mongoDB-secured: "true"
    mongoDB-sharding: "false"
  annotations: 
    author: JMCF
  namespace: sec-datastores
spec: 
  selector: 
    matchLabels:
      app: mongoDB-replica
  serviceName: mongo-db-replica
  replicas: 3
  template:
    metadata: 
      labels: 
        app: mongoDB-replica
    spec: 
      terminationGracePeriodSeconds: 10
      volumes: 
        - name: secret-volume
          secret:
            secretName: mongo-secret
      containers: 
        - name: mongo-db
          image: mongo:4.2.6
          ports: 
            - containerPort: 27017
              protocol: TCP
          volumeMounts: 
            - mountPath: /data/db
              name: mongo-volume-for-replica
            - mountPath: /var/secrets
              name: secret-volume
          args: 
            - --replSet
            - $(REPLICA_SET_NAME)
            - --auth
            - --keyFile
            - /var/secrets/replica.key
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          envFrom: 
            - configMapRef:
                name: mongo-config
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: root-user
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: root-pwd
  volumeClaimTemplates: 
    - metadata: 
        name: mongo-volume-for-replica
      spec: 
        accessModes: 
          - ReadWriteOnce
        resources: 
          requests: 
            storage: 100Mi

Fixing the key file permissions problem

However if you apply the K8s manifest above you will find out that unfortunately the Pods will not be running. We can debug what is happening by running:

kubectl logs mongo-db-statefulset-0 -n sec-datastores
2020-11-07T18:30:38.656+0000 I  ACCESS   [main] permissions on /var/secrets/replica.key are too open

Although, initially you could think that the permissions problem can be fixed by using the defaultMode field of the Secret Volume declaration, it cannot (as of K8s 1.18 and mongoDB 4.2.6). There is another solution which implies running a Pod’s initialization container that just copies the required files with the proper permissions to a new Volume that will be the one actually consumed by the mongoDB container.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo-db-statefulset
  labels:
    mongoDB-replica: "true"
    mongoDB-secured: "true"
    mongoDB-sharding: "false"
  annotations: 
    author: JMCF
  namespace: sec-datastores
spec: 
  selector: 
    matchLabels:
      app: mongoDB-replica
  serviceName: mongo-db-replica
  replicas: 3
  template:
    metadata: 
      labels: 
        app: mongoDB-replica
    spec: 
      terminationGracePeriodSeconds: 10
      volumes: 
        - name: initial-secret-volume
          secret:
            secretName: mongo-secret
        - name: secret-volume
          emptyDir: {}
      initContainers:
        - name: set-file-permissions
          image: busybox
          command:
            - sh
            - -c
          args:
            - >- 
                 cp /var/init-secrets/replica.key /var/secrets/replica.key &&
                 chmod 400 /var/secrets/replica.key &&
                 chown 999:999 /var/secrets/replica.key
          volumeMounts:
            - mountPath: /var/init-secrets
              name: initial-secret-volume
              readOnly: true
            - mountPath: /var/secrets
              name: secret-volume
              readOnly: false
      containers: 
        - name: mongo-db
          image: mongo:4.2.6
          ports: 
            - containerPort: 27017
              protocol: TCP
          volumeMounts: 
            - mountPath: /data/db
              name: mongo-volume-for-replica
            - mountPath: /var/secrets
              name: secret-volume
              readOnly: true
          args: 
            - --replSet
            - $(REPLICA_SET_NAME)
            - --auth
            - --keyFile
            - /var/secrets/replica.key
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          envFrom: 
            - configMapRef:
                name: mongo-config
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: root-user
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: root-pwd
  volumeClaimTemplates: 
    - metadata: 
        name: mongo-volume-for-replica
      spec: 
        accessModes: 
          - ReadWriteOnce
        resources: 
          requests: 
            storage: 100Mi

📌  The Volumes are shared by all containers pertaining to the Pod: the init container (based on busybox), named set-file-permissions and the mongoDB container, named mongo-db.

📌  The lifetime of the final volume containing secrets, secret-volume, will be the Pod’s lifetime.

📌  Once the initialization command completes, the init container will die. In case of failure, the logs of the init container can be obtained using the -container option of kubectl logs.

Configuring the mongoDB Replica Set

The next step is connecting to our cluster through the mongoDB shell and configure the Replica Set. Now we need to make use of the root user (jmcf) and pass previously configured as env vars.

kubectl run tm-mongo-pod --namespace=sec-datastores -it --image=mongo:4.2.6 --restart=Never --rm=true -- mongo -u jmcf -p Lqyr8CvuWsuoSFCN mongo-db-statefulset-0.mongo-db-replica/admin

📌  So far we have not configured TLS so our DB connection will be through an insecure channel.

After checking that we can make an authenticated connection we can execute the Replica Set configuration script and check that our replication is working properly using the replica key provided. Now we are ensuring that the members of our Replica Set can only receive data from parties that know their shared secret (the replica key).

🔒 Setting up the TLS layer

In order to meet our initial requirements, a TLS layer has to be set up. In this part DB only clients must connect through TLS to the DB. In future parts we will show how replicas of the mongoDB cluster could also use mutual TLS to authenticate.

Create a CSR to be signed by the K8s CA

The first step is to generate a new RSA private key (2048 bits) as follows:

openssl genrsa -out mongo.key.pem 2048

Once we have our private key we need to generate a new Certificate. An interesting approach is to generate a Certificate signed by the Kubernetes CA itself, as that is a CA known by all Pods through their default Service Account.

First of all we need to generate a new Certificate Signing Request (CSR) for the private key generated above:

openssl req -new -out mongo.csr -key mongo.key.pem -config ./openssl.conf

The openssl.conf file is necessary as it is convenient to generate the CSR using an extended X509 feature named SAN (Subject Alternative Names), that allows one certificate to be associated with more than one DNS name.

We can inspect the content of our CSR as follows:

openssl req -in mongo.csr -noout -text

Afterwards we can generate our certificate signed by the minikube CA (you can find the CA’s certificate at $HOME/.minikube/ca.crt). However, we can generate the final signed certificate through a standard K8s manifest for Certificate Signing Requests:

apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: mongo-csr
spec:    
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJRGlEQ0NBbkFDQVFBd0pqRUxNQWtHQTFVRUJoTUNSVk14RnpBVkJnTlZCQXNNRGtOaGJuUmxjbUZHYjI1egpaV05oTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUFzOVRuQWVNczFGWWk4ZkJxCnBiNjB2bUVCUFFzWlMzYnlXdmFHWlk0dHV6WndZYS81VXVhOFhTaFVZcnZhYWx6Q052U2ZlVm1jSm5UbTJ5ZlMKbHFrQ2hEb1BlR2tGZ2l4d0ovMy91bStDN1N5bnBtQVJ0UHFoYnI1MFFIaEkvaGlneTVKelpjeVo2ZCthWXdmMwpmTm9vNk1DNVplWVdyajJvKzAxTGdtV000L1pTcDRCRTl6dkYyUXEwN1hiUmRqVHh0ekVrS2FuU0ZQcDkrdlpPCjZFUUJ5ejRWTUo5U0ZReEpQVE5US3ZsRDBSdjRhRUhBVklVNHBEbXptNXQrUEdMeTdrc2FRU0VPbW5GVkpYZ3MKK0dzbENhYTJ4L1BnVUpRWlhtc1E1TkVFaFBuYTZWWFhLaEdiN1pnL0JDR3JFbzU0ZkxmcGdZM3R6SzNNakFmWAo4OEFrNndJREFRQUJvSUlCR3pDQ0FSY0dDU3FHU0liM0RRRUpEakdDQVFnd2dnRUVNQWtHQTFVZEV3UUNNQUF3CkN3WURWUjBQQkFRREFnWGdNSUhwQmdOVkhSRUVnZUV3Z2Q2Q1NHMXZibWR2TFdSaUxYTjBZWFJsWm5Wc2MyVjAKTFRBdWJXOXVaMjh0WkdJdGNtVndiR2xqWVM1elpXTXRaR0YwWVhOMGIzSmxjeTV6ZG1NdVkyeDFjM1JsY2k1cwpiMk5oYklKSWJXOXVaMjh0WkdJdGMzUmhkR1ZtZFd4elpYUXRNUzV0YjI1bmJ5MWtZaTF5WlhCc2FXTmhMbk5sCll5MWtZWFJoYzNSdmNtVnpMbk4yWXk1amJIVnpkR1Z5TG14dlkyRnNna2h0YjI1bmJ5MWtZaTF6ZEdGMFpXWjEKYkhObGRDMHlMbTF2Ym1kdkxXUmlMWEpsY0d4cFkyRXVjMlZqTFdSaGRHRnpkRzl5WlhNdWMzWmpMbU5zZFhOMApaWEl1Ykc5allXd3dEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBR1Q2NXY0WUh3MGRFS1lpRmZsZm9QUDJpVkxqCmRWUk9mS2QwdVBrazJUamo4eEdjbzdpVFBWUEdIbzNvTlJFL21jMVhpZzYwa3lHVWsxM3ZNdTI0YXVhblFPdWkKaFhZUmhlaXlnNGZZTEI1VWVPZzk5N1hycTJXZWhoSUZqRXRHbS9ZMWcwdEQ5S1pBSTF1U29PSTE0bWFlUDZHWgpLYmxRNExNdXlKdndsZ1pzZURiREdYd25zeXA0YVNYQlNCa3VqVFNmV0RBZ21JWHlpZk0wdWxvQjVLVzJZK2pvCjVzWVB4SE1yYmpHalhwaEt3NjlTQzlObDl6UEVZWXB0dmg5UXcyaXBib1hLSXN5SlhPSS9iaHJ0VVBwL1l0TTAKOG13azFzSXZ3K29SbnN1Vmxxdnd0UTArNjAwRmV6OW9QYm1NTXpvajRzTzI0K1hqTVRpVldOVnRVa0k9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
kubectl apply -f csr.yaml
kubectl get csr
NAME        AGE   SIGNERNAME                     REQUESTOR       CONDITION
mongo-csr   16s   kubernetes.io/legacy-unknown   minikube-user   Pending

We can approve the CSR as follows:

kubectl certificate approve mongo-csr

After the certificate has been approved we can download it as follows:

kubectl get csr/mongo-csr -o jsonpath='{.status.certificate}{"\n"}' | base64 -d > mongo.crt

We can inspect the contents of our brand new certificate as follows:

openssl x509 -in mongo.crt -noout -text

Now we have all we need to set up TLS for our mongoDB cluster!

Extending StatefulSet to support TLS

First of all we need to extend our Secret to include the private key and the certificate. mongoDB requires both to be concatenated on the same file:

cat mongo.key.pem mongo.crt >> mongo.keycert
cat mongo.keycert | base64

We add the keycert file content as a base64-encoded Secret property:

apiVersion: v1
kind: Secret
metadata:
  name: mongo-secret
  namespace: sec-datastores
type: Opaque
stringData:
  root-user: jmcf
  root-pwd:  Lqyr8CvuWsuoSFCN
  replica.key: "T3VTeVZHL1EJwllZVaCjhSqtqapViaFGK5vifxWyshnBXdDBP8SqHFz/cG7OFdFLRi2l2se50ThfL1nnlm7UC4uTpl8kBMxI3wdbZdHD6he0x36ghNrYDVyTMbWPNqqH8qYfutojwEum0vHyyIaZadMfuVhPmsQlAv+TqnIVTHXY1vZx8GiKfGiUafyLGRhrPmh2w2+sZXjl1lj3+DLj730axtlDpEf1mTGkB0OKArip4Wply2FuHYFdXYGLuemRegLB+ShbyvqORqegfRou+by5dYflNj/7TnIOCPEzeunPgbsJBVbgVptVGk6pYNIrnPvtZOS/P6u0L/0c1J1EZyQ9wnXfu6xZpZSPg46WC0IznHMwe/3I8Z/YUiOY5+Oq/DFFLIRJaMblppNV6p+30z0YV7HRYW1eVHnkSwoi5m6o43B/cUidVhueUV/z0c1ZpojL5gfUSpu2YZNjWa6kVZ1+nXVmtDT7HqEZ1o5GjNMYkQHZ6ysJW7wbmqg77tP+KGvpEMvc0f0HLQLd1KPJ151URM5HvxDG3F7erPPI3dszw0k0OEGu/504h3sfYdreFEYZYa0vr8w2TuEJymbwWippNqNOvZQsrmLX6JXfVwMcxad27y+DwUXwj8jNoWy2lriak9xqaKbid0qitSp3+1WVgiffL/sZBOeoGsgdqop1dNdOCSsGwuzlVpZ5TqtklSdc0R+V0wuiROe086icbuVtz26OcSAkAE14WTvn3ID2/b6v7VyIrLpT2ifPGSEgyXNK0d0dFiCmOo9ewF+F6G1FX0yn0Ra5S8ZEG4BrAy8LCiCSxnNxehEW5aeEfD6Vsu08dl1I6/Khigu1DJnptKRpSR8vzNuOgtR1fi2CJFdKL2sZZI5K7wpQ5DuBgSCHpxA4qQf5e5vxgshFRXiVklRu11Ud/6Ra1CmdU5KOJc8M4t0tj3yCrFplBO8fxZ52CtRYTKpVTqcSrIbxlMbU/jmQGvbpSEjhBZ9iLszugIwbbsEq"
data:
  tls.keycert: "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBczlUbkFlTXMxRllpOGZCcXBiNjB2bUVCUFFzWlMzYnlXdmFHWlk0dHV6WndZYS81ClV1YThYU2hVWXJ2YWFsekNOdlNmZVZtY0puVG0yeWZTbHFrQ2hEb1BlR2tGZ2l4d0ovMy91bStDN1N5bnBtQVIKdFBxaGJyNTBRSGhJL2hpZ3k1SnpaY3laNmQrYVl3ZjNmTm9vNk1DNVplWVdyajJvKzAxTGdtV000L1pTcDRCRQo5enZGMlFxMDdYYlJkalR4dHpFa0thblNGUHA5K3ZaTzZFUUJ5ejRWTUo5U0ZReEpQVE5US3ZsRDBSdjRhRUhBClZJVTRwRG16bTV0K1BHTHk3a3NhUVNFT21uRlZKWGdzK0dzbENhYTJ4L1BnVUpRWlhtc1E1TkVFaFBuYTZWWFgKS2hHYjdaZy9CQ0dyRW81NGZMZnBnWTN0ekszTWpBZlg4OEFrNndJREFRQUJBb0lCQUFxb1YzaDVLakw3Vjk3Vwp6THM5N2lCS1IzU0NyK0VWRDQvc2hGS0lkcFZkeUpOQmhwa3ZLNEtwM1Rac3c2a0NEbENMZHRnT1EzN1Q5c3FPCk5hRFFLMklDdnQvMytXZVprcHdNSlplNi9CdHFSaFZLSkVLQmlBZTBLZGVZU1ZzbHdvSE9FQ0M4VzNMTTJhMTgKejJJSzI5blBjaTFqMFh4S0V6V3hnd1FiV0pvNTZZTDkvd3VOaU5iNHlyc0lnSnlRNnBydFhUZGk3QUNEcnBGMwpDQW5DaGZWUGpYdllIRDZLSHdtTzVPRm1QNkZvL2c2L1M3Y0tVK0N1K1NCWkY3VmRRZzdjVERtV1kvMGJCOUFOCkJhd1lVc1dQWXdNZHhsb0t4WWt6S1NqelQ1K1Z1SldIWFkwNkp6alpLY25EcjJDaXVOcWNRYlpMWW5sdzN6L24KSzFIdFhHRUNnWUVBM0padEE2WmQwSTFuNUFQU0xxOUlUYTNhaThqMG1RczgyekxpQ2lLYTRic2xtcGxTWHZQNQpWZ1BUcVdnU0IvNmtkZTlwZktRcFIzN0xDQTNIK0R2dEUrVzBpeHJjUWtGbXhhWXBhSTVmUkhicmNPbzNsQ3VWCk83VllvSDJvcVgwWnhqQis2bVZwTy8vSC9oQ2ZWeDZRazJma0VUQ0d6UStOd1h5OXZzYk11M2NDZ1lFQTBMT0QKYnhhUUtDVk13VjU5SE1aTDN4cDZMWTFCUWsyRXY3VDl3bnFkenB0MjhZajFhZHZYeWgzVHhTbEJWU2VWVmpKYwp0MGFBcVBqenNTS2p6TUswSVROb044SXNzOXlxUlBJend3VzZ2OEhJSjJIRktpVnk1ZFhZaTQwWGNjYnVDd1ErCjNLV1JKUEFwTVRRQlZIV1JMTFNHREZtbjVxNnE4R0ZqTWlaUWx5MENnWUVBblhsK0JiY0ZEbGVFclZEVkFhTDgKbVFmZUZ5QzFEWFRxVXduMk9Jb1B5OHBReEJka3FJaS9uVnBLQm94WEl3SDA0b1N4NTl0QjVOcHRreWhUYzdGRQpSSW9kNDVpZldXVmJPd3F1VVFnaWxydld4TnRRSmlVZSttc3lCQkR4RVY1UHdoN28yK2pkd1ltT2VwL2ZRTkIvCklkeHZBcDZEL1d4M0pJUXE2VUQrODIwQ2dZQmZJVVNLOFh6Nlg2NHlBbEVET2J4QjBQN2FIcVlkZzlvN3pGTlQKUDdNSkIwSmJiTE40OGxYVmNtMGlWT0RMNFFRb1MwS0o1Q0FuWlhzblJnUG1CT1k0c0FjVVMzVTI4eUhGMWFVVQpnMXNScDFJRjZZSUc2UlVMZ01ONU1QSDcwZEtWd1BTcWZPbkJMWnVKS3d4a0pFYnRXc2d6ZnZhU3B4R1NDc0k5CmpNdDFiUUtCZ1FDeUpYdVdLRzE2M09RR1VRVjJYdytYQWpJZWRISGZ2S0lGT2x1WHAwb0FZTFVwSGhxRkk4NW0KaDh5dmQzY2tub1o1ajhWSG9OWVFWYlUvYlBtbUsxbU1RZTByQUZWd0VvSHhuSHBDWGFHM1pmUkRYZkpnMTJwVwpac1VJZ3lDNEJLSFpieDhNZDE1Q3k4dEN5TXE1UWJGbm1oblRpNEJRVnlyTEF0UGZqb3QwN3c9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRDFUQ0NBcjJnQXdJQkFnSVFXVHVoUlh3UEdGRGJqemR2d1l3TUZEQU5CZ2txaGtpRzl3MEJBUXNGQURBVgpNUk13RVFZRFZRUURFd3B0YVc1cGEzVmlaVU5CTUI0WERUSXdNVEV3T0RFd05UY3pNMW9YRFRJeE1URXdPREV3Ck5UY3pNMW93SmpFTE1Ba0dBMVVFQmhNQ1JWTXhGekFWQmdOVkJBc1REa05oYm5SbGNtRkdiMjV6WldOaE1JSUIKSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXM5VG5BZU1zMUZZaThmQnFwYjYwdm1FQgpQUXNaUzNieVd2YUdaWTR0dXpad1lhLzVVdWE4WFNoVVlydmFhbHpDTnZTZmVWbWNKblRtMnlmU2xxa0NoRG9QCmVHa0ZnaXh3Si8zL3VtK0M3U3lucG1BUnRQcWhicjUwUUhoSS9oaWd5NUp6WmN5WjZkK2FZd2YzZk5vbzZNQzUKWmVZV3JqMm8rMDFMZ21XTTQvWlNwNEJFOXp2RjJRcTA3WGJSZGpUeHR6RWtLYW5TRlBwOSt2Wk82RVFCeXo0VgpNSjlTRlF4SlBUTlRLdmxEMFJ2NGFFSEFWSVU0cERtem01dCtQR0x5N2tzYVFTRU9tbkZWSlhncytHc2xDYWEyCngvUGdVSlFaWG1zUTVORUVoUG5hNlZYWEtoR2I3WmcvQkNHckVvNTRmTGZwZ1kzdHpLM01qQWZYODhBazZ3SUQKQVFBQm80SUJEakNDQVFvd0RnWURWUjBQQVFIL0JBUURBZ1dnTUF3R0ExVWRFd0VCL3dRQ01BQXdnZWtHQTFVZApFUVNCNFRDQjNvSkliVzl1WjI4dFpHSXRjM1JoZEdWbWRXeHpaWFF0TUM1dGIyNW5ieTFrWWkxeVpYQnNhV05oCkxuTmxZeTFrWVhSaGMzUnZjbVZ6TG5OMll5NWpiSFZ6ZEdWeUxteHZZMkZzZ2todGIyNW5ieTFrWWkxemRHRjAKWldaMWJITmxkQzB4TG0xdmJtZHZMV1JpTFhKbGNHeHBZMkV1YzJWakxXUmhkR0Z6ZEc5eVpYTXVjM1pqTG1OcwpkWE4wWlhJdWJHOWpZV3lDU0cxdmJtZHZMV1JpTFhOMFlYUmxablZzYzJWMExUSXViVzl1WjI4dFpHSXRjbVZ3CmJHbGpZUzV6WldNdFpHRjBZWE4wYjNKbGN5NXpkbU11WTJ4MWMzUmxjaTVzYjJOaGJEQU5CZ2txaGtpRzl3MEIKQVFzRkFBT0NBUUVBV0Z0VXhIb3ZVTDZjclBMZnFCTXcxK2pRU1o0Y2VFd3M4OHBkTVdJR0tOVS9KNHVMVVBvTgpTSzlOakQxUFpSTDU4d3NVTC9TR2RsYWF4NHpTcmo4WUdCNE9BejhiWTN0NXJxRW1Jb1J2bEY0ekltKzZ5VWhFCnpXUGNwbVl1Rkx1ekUvMGVIM3lDVFUzM0U0enczcTFnQ2RyaFlDaVJ0NUZINThYTmNPSDJPdjNNeW54aDd4NlYKTFBBL2lONUJUaDIzeVJFRElCdFBEUVZnQ3lHUk5LSXNBL2dINmhUaFgwZVc4d09VUnMzbCtycHJ3ajlzemFjTgo4NFBNMFN5bjI1cnJ1NEFNNFREYVRzN1puNHdOQUNETmMwZUViWkVacEUybitjYXBrRmtVTm5zTHVvMUtCQy9yCmpQYkhUU1laYjZFcXBaeDZaKzRKaTJ0bTc4aXBOSHo3WEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="

We can double-check that our keycert has been properly stored as a K8s secret:

kubectl get secret mongo-secret -o jsonpath="{.data['tls\.keycert']}" -n sec-datastores | base64 -d

📌  We are assuming that Secrets are stored in Base64 format in the Secret store, which should not happen in a production environment!!.

And now we need to extend our StatefulSet definition to provide the different TLS parameters:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo-db-statefulset
  labels:
    mongoDB-replica: "true"
    mongoDB-secured: "true"
    mongoDB-sharding: "false"
  annotations: 
    author: JMCF
  namespace: sec-datastores
spec: 
  selector: 
    matchLabels:
      app: mongoDB-replica
  serviceName: mongo-db-replica
  replicas: 3
  template:
    metadata: 
      labels: 
        app: mongoDB-replica
    spec: 
      terminationGracePeriodSeconds: 10
      volumes: 
        - name: initial-secret-volume
          secret:
            secretName: mongo-secret
        - name: secret-volume
          emptyDir: {}
      initContainers:
        - name: set-file-permissions
          image: busybox
          command:
            - sh
            - -c
          args:
            - >- 
                 cp /var/init-secrets/replica.key /var/secrets/replica.key &&
                 chmod 400 /var/secrets/replica.key &&
                 chown 999:999 /var/secrets/replica.key;
                 cp /var/init-secrets/tls.keycert /var/secrets/tls.keycert && 
                 chmod 400 /var/secrets/tls.keycert &&
                 chown 999:999 /var/secrets/tls.keycert
          volumeMounts:
            - mountPath: /var/init-secrets
              name: initial-secret-volume
              readOnly: true
            - mountPath: /var/secrets
              name: secret-volume
              readOnly: false
      containers: 
        - name: mongo-db
          image: mongo:4.2.6
          ports: 
            - containerPort: 27017
              protocol: TCP
          volumeMounts: 
            - mountPath: /data/db
              name: mongo-volume-for-replica
            - mountPath: /var/secrets
              name: secret-volume
              readOnly: true
          args: 
            - --replSet
            - $(REPLICA_SET_NAME)
            - --auth
            - --keyFile
            - /var/secrets/replica.key
            - --tlsMode
            - requireTLS
            - --tlsCertificateKeyFile
            - /var/secrets/tls.keycert
            - --tlsCAFile
            - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            - --tlsAllowConnectionsWithoutCertificates
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          envFrom: 
            - configMapRef:
                name: mongo-config
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: root-user
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongo-secret
                  key: root-pwd
  volumeClaimTemplates: 
    - metadata: 
        name: mongo-volume-for-replica
      spec: 
        accessModes: 
          - ReadWriteOnce
        resources: 
          requests: 
            storage: 100Mi

📌  The permissions of the keycert file have to be properly set by the init container.

📌  It is very easy and convenient to point to the CA file as it is always available as part of the Service Account of our Pods.

📌  --tlsAllowConnectionsWithoutCertificates allows clients to connect to the DB using just a user/pass.

Connecting to the Cluster through TLS

We can connect to the cluster through TLS as follows:

 kubectl run tm-mongo-pod --namespace=sec-datastores -it --image=mongo:4.2.6 --restart=Never --rm=true -- mongo --verbose --tls --tlsCAFile /var/run/secrets/kubernetes.io/serviceaccount/ca.crt  -u jmcf -p Lqyr8CvuWsuoSFCN mongo-db-statefulset-0.mongo-db-replica.sec-datastores.svc.cluster.local/admin

📌  We are using the FQDN of the mongoDB Pod so that there is a host match at the TLS layer.

🖊️ Conclusions

Kubernetes provides powerful primitives to deploy a secured, clustered mongoDB datastore service, so that we can give production-grade support to IoT and Big Data Applications which demand higher scalability.

🗒️ Feedback