Local testing of Kubernetes policies

[jpreese]

Edit: running on OPA 0.12 , Windows 10

Version: 0.12.0
Build Commit: 64052c1
Build Timestamp: 2019-06-11T22:03:42Z
Build Hostname: 74e59ffe492b

Expected Behavior

opa test -v . runs the tests and reports back successful when policies are abided by

Actual Behavior

OPA seems to always return false

Steps to Reproduce the Problem

Using https://play.openpolicyagent.org/p/MinNMtDjC4 as reference

The above snippet which was written by @ashutosh-narkar in #1529 does run due to

var deny is unsafe

Using any of the other examples online, I'm not able to pass in a simple fake Kubernetes JSON and have it be validated successfully by OPA

Additional Info

Taking inspiration from the playground..

services.rego

package kubernetes.admission

import data.kubernetes.admission

required_annotations = {"alb.ingress.kubernetes.io/tags": {"some_value", "some_other_value"}, "alb.ingress.kubernetes.io/scheme": {"internal"}}

deny[msg] {
  input.request.kind.kind == "Service"
  required_annotations[key]
  not input.request.object.metadata.annotations[key]
  msg := "error..."
}

services_test.rego

package kubernetes.test_admission

import data.kubernetes.admission

test_deny_service_without_loadbalancer_annotation {  
  missing_service_annotation := {"kind": "Service"}

  count(admission.deny) == 1 with input as missing_service_annotation
}

results in

  Enter data.kubernetes.test_admission.test_deny_service_without_loadbalancer_annotation = _
  | Eval data.kubernetes.test_admission.test_deny_service_without_loadbalancer_annotation = _
  | Index data.kubernetes.test_admission.test_deny_service_without_loadbalancer_annotation = _ (matched 1 rule)
  | Enter data.kubernetes.test_admission.test_deny_service_without_loadbalancer_annotation
  | | Eval __local0__ = {"kind": "Service"}
  | | Eval __local2__ = data.kubernetes.admission.deny with input as __local0__
  | | Index __local2__ = data.kubernetes.admission.deny with input as __local0__ (matched 0 rules)
  | | Eval count(__local2__, __local1__) with input as __local0__
  | | Eval __local1__ = 1 with input as __local0__
  | | Fail __local1__ = 1 with input as __local0__
  | | Redo count(__local2__, __local1__) with input as __local0__
  | | Redo __local2__ = data.kubernetes.admission.deny with input as __local0__
  | | Redo __local0__ = {"kind": "Service"}
  | Fail data.kubernetes.test_admission.test_deny_service_without_loadbalancer_annotation = _

[ashutosh-narkar]

Would something like the example below work ?

service.rego

package kubernetes.admission

required_annotations = {"alb.ingress.kubernetes.io/tags": {"some_value", "some_other_value"}, "alb.ingress.kubernetes.io/scheme": {"internal"}}

deny[msg] {
  input.request.kind.kind == "Service"
  required_annotations[key]
  not input.request.object.metadata.annotations[key]
  msg := sprintf("Compliance check failed: ELB_009: All load balancers must be tagged as per the tagging policy. Missing annotation %v required", [key])
}

service_test.rego

test_deny_service_without_loadbalancer_annotation {
     in := {
        "kind": "AdmissionReview",
        "apiVersion":"admission.k8s.io/v1beta1",
        "request":{
            "uid": "66c738ea-1b4c-11e9-a7d2-080027f75b4a",
            "kind": {
                "group": "extensions",
                "version": "v1beta1",
                "kind": "Service"
            },
            "namespace": "production",
            "operation": "CREATE",
            "object": {
                "metadata": {
                    "name": "ingress-ok",
                    "namespace": "production",
                    "uid": "66c73498-1b4c-11e9-a7d2-080027f75b4a",
                    "generation": 1,
                    "annotations": {
                        "kubernetes.io/ingress.class": "alb"
                    },
                    "creationTimestamp": "2019-01-18T18:10:56Z"
                }
            }
        }
    }

    violations := deny with input as in
    count(violations) == 2
}

$ opa test -v service.rego service_test.rego
data.kubernetes.admission.test_deny_service_without_loadbalancer_annotation: PASS (784.305µs)
--------------------------------------------------------------------------------
PASS: 1/1

[jpreese]

As I dig through it a little more, there does seem to be an introduction of wrapping the YML in an AdmissionReview Kind.. but the first step I'm looking for is CI validation where I just want to validate purely the YML. Then the same policies would be on OPA in the cluster.

[jpreese]

@ashutosh-narkar which version of OPA is that? I'm on 0.12, which seems to be the latest and that gives me

var deny is unsafe

I have not been able to use allow or deny in any of my statements.

[ashutosh-narkar]

$ ./opa_darwin_amd64 test -v service.rego service_test.rego
data.kubernetes.admission.test_deny_service_without_loadbalancer_annotation: PASS (1.377908ms)
--------------------------------------------------------------------------------
PASS: 1/1

$ ./opa_darwin_amd64 version
Version: 0.12.0
Build Commit: 64052c1c
Build Timestamp: 2019-06-11T22:03:26Z
Build Hostname: 74e59ffe492b

[jpreese]

Ok, at its most basic form, I was able to get it to work. I can build from there.

services_test.rego

package kubernetes.test_admission

import data.kubernetes.admission

test_deny_services {

  request := {
    "request": 
      { 
        "kind": 
        {
          "kind": "Service"
        }
      }
    }

    count(admission.deny) == 1 with input as request
}

services.rego

package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Service"
  msg := "error"
}

I have some concerns around "admission" policies and kubernetes yaml policies. At first glance, they seem to differ, though they really shouldn't.

In other words, if you submit a PR with service.yaml that contains valid kubernetes that contains Kind: Service, I'd want this policy to fail in my pipeline with opa test .Though I haven't played around with it enough to get a stronger sense of how everything is implemented.

Thanks for the help!

[ashutosh-narkar]

The input that you see above is what OPA gets from the Kube API server when it's implemented as an admission controller. Are you looking for a policy that validates the Kube manifest itself and is not about admission control. Is that accurate ?

[jpreese]

That is exactly correct. My concern is that you'd have to have two policies that check the exact same thing.

In other words, if I have an OPA policy that says "no services allowed", I'd want that policy to validate against my kubernetes manifest in my CI pipeline.

But I'd also want that same policy running in my kubernetes cluster with the admission controller.

Can OPA use the exact same policy to validate Kubernetes manifest itself and the result from an admission controller? I'd like to avoid duplicating every policy, one for CI/Kubernetes manifests, and one for the admission controller.

[ashutosh-narkar]

One option would be to have multiple rules with the same name in your package. For ex,

deny[msg] {
    input.kind == "Service"
    msg := "No service allowed"
}

deny[msg] {
    input.kind == "AdmissionReview"
    input.request.kind.kind = "Ingress"
    <some_logic_here>
    msg := ...
}

So based on how the input looks (manifest validation vs admission control), the appropriate rule gets evaluated and returns the result. Whatever is the common logic can be factored out into rules/functions.

[jpreese]

Ok! This was similar to what I was theorizing.

I know you can also do say..

package somepackage
is_service {
  input.kind = "Service"
  // but would need some OR here
}

deny[msg] {
    somepackage.is_service
    ...
    msg := "No service allowed"
}