The Kubernetes Storage Eco-System For SQL Server Professionals

There seems to a great deal of interest in containers and Kubernetes at present, fueled by Microsoft hinting that Kubernetes has a big part to play in the future of the Microsoft data platform:

Of the Kubernetes eco-system, storage appears to be one of the least well understood parts of it, more so when running Kubernetes on premises. My hope is that this blog post goes someway in demystifying storage and Kubernetes.

Stating Off With Pods and Mount Points . . .

One of the most fundamental concepts to grasp in Kubernetes is that of the pod. A pod is a collection of one or more containers that are always co-scheduled on the same host. Pods are immutable, if a pod dies it is never resurrected. To make a VMware comparison, in an ESXi cluster, the unit of scheduling is a virtual machine, in a Kubernetes cluster the unit of scheduling is a pod. The “Touch point” for storage in a pod is a volume which always has an associated mount point.

Pod Scheduling

Another core concepts behind Kubernetes is that it is designed to turn the data center into one big server. One of the algorithms Kubernetes uses to schedule what pod goes on which node is bin packing, which Kelsey Hightower illustrates brilliantly in this presentation using Tetris.

I strongly urge anyone with an interest in learning Kubernetes to watch this presentation, as it makes a great job of explaining Kubernetes from the ground up.

“Out of the box”, Kubernetes will look after scheduling. If there is a requirement to ensure that pods only run on a specific set of nodes, there is a means of doing this via label selectors, as documented here. A label selector is a directive of influencing resource utilization related decisions made by the cluster.

Disclaimer

I work for Pure Storage, this is my personal blog. Any reference to Pure in the yaml or in the graphic below is a hundred percent down to the fact that this is material I have readily available. If you follow Microsoft’s own documentation on configuring a SQL Server container in Kubernetes for high availability, which I encourage you to do, you will see that it outlines the exact same principles I have covered in this blog post.

. . . Moving On To Persistent Volume Claims

Kubernetes is a platform for developers, as such, it is designed to abstract details of the underlying storage away from the developer. This is the developer’s view of the storage world in this world:

Persistent volume claims have three attributes:

• name
• storage class
• size

This YAML excerpt illustrates how to create one of these:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mssql-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: pure-block

A persistent volume (often abbreviated to PV) is the entity that maps directly onto the storage platform. If a PV exists that the persistent volume claim can be satisfied from, Kubernetes will use it to satisfy the volume claim. Otherwise, a persistent volume will be created if the storage class is configured for dynamic storage provisioning. Below is an example of the YAML for block and file storage classes:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: pure
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
labels:
kubernetes.io/cluster-service: "true"
provisioner: pure-provisioner

parameters:
backend: block

---

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: pure-file
labels:
kubernetes.io/cluster-service: "true"
provisioner: pure-provisioner
parameters:
backend: file

The association of a persistent volume claim with a persistent volume is known as “Persistent volume claim binding”.

The Storage Provision-er

The provision-er abstracts out the mechanism of creating/deleting volumes across the different storage types used in the Kubernetes cluster. One provision-er instance usually exists per storage type. This is not to be confused with the FlexVolume driver which mounts the volume. FlexVolume is the mechanism by which storage vendors integrate their wares with Kubernetes, however the container storage interface is waiting in the wings to replace this. Under the covers, provision-ers usually run inside a “Daemon set”. Think of a daemon set as the Kubernetes version of a Linux / Unix daemon or Microsoft windows service.

Storage Classes

A storage class in conjunction with a provision-er allows storage to be provisioned dynamically. The exact storage classes available will vary from one storage platform / cloud provider to another. For the purposes of this blog post, we will focus on two types of storage class:

• A block storage class
• A file / unstructured data storage class

Storage classes are also developer visible objects that abstract out the actual provision-er implementation.

Static Versus Dynamic Provisioning 

Most storage provisioning mechanisms can be either static or dynamic. Under dynamic schemes, the Kubernetes administrator creates persistent volumes for developers to consume via persistent volume claims. With dynamic provisioning schemes, persistent volumes are dynamically created in response to requests for persistent volume claims as and when required. Rules associated with the storage class determine if and how persistent volumes are created for dynamic storage provisioning.

Statefulsets

Statefulsets were first introduced with Kubernetes 1.9 and are a nuanced topic worthy of an entire blog post in its own right. Therefore, I will attempt to summarize what statefulsets provide at a high level, in short they provide an elegant means of addressing the challenges of scaling stateful applications:

• Scaling out applications via replica sets will result in the storage being replicated
• Some stateful multi pod applications have a requirement for pods to be started and stopped in a specific order, namely applications with controller / worker pod architectures.
• Assigning a unique identifier to a collection of pods that form an application. For example if your application is comprised of a cluster, each cluster node will require its own identity.

Can I Safely Run Databases On Kubernetes ?

This is another topic that has surfaced in the SQL Server community. For the purpose of this blog, the focus will be on database engines that guarantee crash consistent recovery and support ACID transactions. Assuming that the underlying storage platform is stable and that the provision-er ensures that:

• A persistent volume claim is always bound to a persistent volume
• If a pod dies, a new instance of that pod, and note this is completely new and not a resurrected version used mount points underpinned by the same persistent volume claims as before

There is no reason why Kubernetes cannot be used as a platform to run SQL Server on. Note that when it comes to high availability, availability groups and as far as I am aware failover clusters are not supported on Kubernetes.

Another topic that has surfaced which has muddied the waters is that of replication, if you have any type of asynchronous replication and the publisher dies, there is the risk of data loss irrespective of whether the platform hosting the database is:

• On bare metal
• Virtualized
• Uses containers
• Uses containers via any orchestration framework

Final Words, What Should I Do ?

If you are using any form of Kubernetes-as-a-service via a public cloud provider, storage provisioning should be baked into the platform and it should just work. On premises users should look for storage platforms that provide automatic storage provisioning and a good Kubernetes-storage-as-a-service experience. Unless your day job involves standing up open source storage platforms or cutting code for storage platforms, I would avoid going down the rabbit hole of seeking out exotic open source solutions. Open source has served the SQL Server community tremendously well, dbatools is but one great example of this. However, my best advise is to focus on the core skills that your day job as a DBA / developer / architect entails and not on re-inventing the the storage platform wheel.