This is a guest post, originally posted on Angelos Kolaitispersonal blog, reproduced here with his permission. Angelos is a MicroK8s software engineer at Canonical.

How we brought a universal, cluster-wide storage solution to MicroK8s.

Kubernetes is well-known as an open-source system for automating deployment, scaling and management of containerised applications. As such, it uses concepts such as Pods and Deployments to abstract away details regarding the underlying compute, networking and storage infrastructure.

Storage is abstracted as PersistentVolumes (a volume that is provisioned in the underlying infrastructure) and PersistentVolumeClaims (a claim for a pod that uses a persistent volume) resources. Typically, running a Kubernetes cluster also involves running a CSI provisioner, which watches for PersistentVolumeClaims and automatically provisions the requested volumes.

There are a large number of CSI drivers available. For example, if running Kubernetes on top of AWS, one would most likely use the aws-ebs-csi-driver. If running Kubernetes on-premises or in an OpenStack cloud, cinder-csi-driver would be more appropriate. If running on-premises and a Ceph cluster is available, ceph-csi could also be used.

While this solves the storage problems for Kubernetes users, it moves the hurdle of managing everything to the administrators of the Kubernetes cluster, as well as the Kubernetes distribution.

So, the question is, how can storage be made simpler for the administrator of the Kubernetes cluster?

Mayastor is currently under-development as a sub project of the Open Source CNCF project OpenEBS.

Mayastor can be deployed on any Kubernetes cluster and provide Kubernetes-native storage. It can also use existing block devices on the Kubernetes cluster nodes (if available), as well as image files.

Further, the Mayastor data plane supports replication. You can replicate your volume data across multiple pools, to ensure redundancy in case of a single node failure.

In general, Mayastor is a very interesting solution for Kubernetes-native, fast, redundant storage that works universally across any Kubernetes cluster.

Deploy Mayastor

NOTE: For the latest instructions, refer to the mayastor addon page.

The mayastor addon will be available with MicroK8s 1.24. You can already try it out by installing MicroK8s from latest/edge:

  1. Enable nvme_tcp and HugePages:sudo apt-get install linux-modules-extra-$(uname -r)
    sudo modprobe nvme_tcp
    sudo sysctl vm.nr_hugepages=1024
  2. Install MicroK8s from latest/edge:sudo snap install microk8s --classic --channel latest/edge
  3. Enable Mayastor:sudo microk8s enable core/mayastor

Deploy a test workload

Now that mayastor is deployed, it is time to test it using an example pod:

# pod-with-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: name: my-pvc
spec: storageClassName: mayastor accessModes: [ReadWriteOnce] resources: { requests: { storage: 5Gi } }
---
apiVersion: v1
kind: Pod
metadata: name: nginx
spec: volumes: - name: pvc persistentVolumeClaim: claimName: my-pvc containers: - name: nginx image: nginx ports: - containerPort: 80 volumeMounts: - name: pvc mountPath: /usr/share/nginx/html

Create the pod with:

sudo microk8s.kubectl create -f pod-with-pvc.yaml

Then, we can check to see that Mayastor has created our persistent volume, and the pod has come up:

sudo microk8s.kubectl get pod,pvc

The output should look like this:

NAME READY STATUS RESTARTS AGE
pod/nginx 1/1 Running 0 4m NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/my-pvc Bound pvc-e280b734-3224-4af3-af0b-e7ad3c4e6d79 5Gi RWO mayastor 4m

The new mayastor addon in MicroK8s is a very simple way to provision fast, replicated and redundant (though not highly-available) persistent volumes in Kubernetes.

The mayastor addon is considered beta as of MicroK8s 1.24 release. You can already try it by installing from the latest/edge track.

Future steps include:

  • Scale and stress testing.
  • Graduate addon from beta to stable status.
  • Refine the mayastor addon. Right now it automatically provisions image-file backed pools. but we want to make the addon easier to configure for MicroK8s clusters with designated block devices for storage.
  • Compare Mayastor with Rook (Ceph) and other Kubernetes native storage solutions.
  • Document and test backup and restore of mayastor volumes.
  • Support arm64 architectures. This is pending support in mayastor

Photo by Darrin Moore on Unsplash