Enforce and Automate Policies for Kubernetes Data Protection
Achieving data protection for developers is easy, right? Let's “whip up” a daily cron job backup script and call it “production-ready.”
However, battle-hardened IT experts will know that it’s much more challenging than that.
The CIO wants to reduce the RPO (acceptable data loss) to minutes.
The General Manager (GM) wants to limit financial impact due to downtime of a revenue-generating app by reducing RTO (acceptable recovery time) to 1 hour.
The CISO demands immutable backups (backups that cannot be deleted or modified), a necessary defense against ransomware.
Data protection policies exist to mitigate these risks, but organizations struggle to enforce them, relying on tightly coordinated runbooks, after-the-fact reporting, or specialized operations teams whose sole job is to ensure backup compliance.
The reality is that the cloud-native organization is deploying applications to multiple clouds using CI/CD pipelines or infrastructure as code at a rapid pace – faster than IT can provision data protection policies. By the time the data protection team is done provisioning one policy, the developer team is already getting ready to deploy its next release.
Enter Data Protection Cloud Native Posture Management. Cloud-native policies enforce good security hygiene that a developer would not necessarily be expected to know. Think of security policies that reject weak RBAC, identify container images with known CVEs, recommend network policies that aren’t too open, or deny authorization for requests that are not least privileged.
The same strategy can apply for enforcing data protection objectives.
In a world in which deployments are happening frequently, a small misconfiguration such as not having immutable backups can have disastrous real-world results, such as backups being susceptible to data destruction, exposing a vulnerability to ransomware. This, in turn, could put the pressure on, forcing organizations to consider paying multi-million dollar ransoms.
Here at Kasten by Veeam, we’ve authored Kubernetes-native policies that can be integrated into Infrastructure as Code to provide “guardrails” for these types of problems. These policies are enforced by essentially preventing misconfigured apps from going to production.
The approach works by leveraging Kubernetes Admission Controllers that prevent pods, deployments, or StatefulSets from being scheduled onto a Kubernetes node, should a policy be violated.
Our implementation was developed using Kyverno (a Cloud Native Computing Foundation Sandbox project), which implements a custom Kubernetes admission controller.
How It Works
Like Kasten K10, Kyverno is Kubernetes-native. It’s a cloud-native policy engine donated to the CNCF by Nirmata, and was designed with Kubernetes professionals in mind. Kyverno is simple to author policies with and only requires an understanding of Kubernetes APIs (by means of declarative YAMLs).
If you’re familiar with Kubernetes YAML, see below. For everyone else, here’s a synopsis:
Enforce this policy on deployments and StatefulSets that are being deployed.
Look in the metadata header for a label called “purpose” with value “production.”
If that matches, look for a “dataprotection” label that starts with the value “k10” (regexp wildcards are supported).
Also, look for the existence of an immutable backup target.
apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: prod-backup-enforce-policy spec: validationFailureAction: enforce rules: - name: cd-prod-backup-policy match: resources: kinds: - Deployment - StatefulSet validate: message: "Production Deployments and StatefulSets must have Data Protection Policies with Immutability Enabled (use labels: dataprotection: k10-<policyname> and immutable: enabled)" pattern: metadata: labels: =(purpose): production dataprotection: k10-?* immutable: enabled
If a developer attempts to deploy a StatefulSet or deployment without these labels, they will get immediate feedback with a customizable message:
Error from server: error when creating "nginx-deployment-invalid.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: resource Deployment/nginx/nginx-deployment was blocked due to the following policies prod-backup-enforce-policy: cd-prod-backup-policy: 'validation error: Production Deployments and StatefulSets must have Data Protection Policies with Immutability Enabled (use labels: dataprotection: k10-<policyname> and immutable: enabled). Rule cd-prod-backup-policy failed at path /metadata/labels/immutable/'
Kyverno also has an ability to “generate” custom resources, which can automate scheduling of this application into a Kasten K10 App-Scoped Policy – a policy visible only by a sysadmin or admin of the app’s namespace.
In this example, the Generate policy is assigning the highest level of data protection (the “Gold” backup policy) for any apps matching the label “k10-goldpolicy.” It also looks for the assignment of an immutable Location Profile (a Kasten K10 backup target).
Combining enforcement and automation is very powerful. You can ensure compliance according to your organization’s data protection policies and schedule the operational jobs, completing both Day 1 and Day 2 operations, just like that. Best of all, you can combine other standard benchmarks or community policies to harden your security posture across the network – and your data management processes, as well.
Joey is a Principal Product Manager for Kasten by Veeam. Joey comes to Kasten with over a decade and a half of experience managing product and services portfolios worth $100M-$500M annually. Previously Joey was a Lead Product Manager for the Data Protection and Availability products at Dell EMC. He was also Director of Product Management at Synoptek. Joey excels at operating up and down the technology stack and forecasting a vision for emerging markets which has resulted in significant revenue growth for his organizations.
Kasten, Inc. 8800 Lyra Drive, Suite 450 Columbus, Ohio 43240
We value the critical role that the security community plays in helping us protect the confidentiality, integrity, and availability of our software, services, and information. If you have information about security vulnerabilities that affect Kasten software, services, or information, please report it to us via our HackerOne Vulnerability Disclosure Program, or anonymously via this form.