Verifying linkerd images with keyless cosign through kyverno

Emil Snorre Alnæs

If you're running linkerd and kyverno in a cluster, getting kyverno to use cosign verification for the linkerd images is pretty easy. There are some things to watch:

  1. Finding the config for linkerd's own cosign signatures involves a little digging in their github workflows. Until this makes its way into their own public user docs, we can likely assume that some part of their signing process can change without much or any publicity.

    Open source means we can discover this on our own, but there's also a caveat emptor when you do stuff like that, especially as going digging through code means they're not even aware we're an emptor for this feature.

  2. proxy-init is a bit of an odd duck out with a different workflow subject. As far as I can tell, this also means that matching just cr.l5d.io/linkerd/* is out:

    • The attestor entries in the kyverno policy is an AND list, not OR.
    • To my knowledge the matching in kyverno doesn't have negative lookahead, so we can't do something like …/linkerd/(?!proxy-injector).+
  3. The kyverno verify images policy by default mutates the digest. This confuses linkerd check, as well as argocd sync status, and probably more things I'm not even aware of.

With that out of the way, the policy itself is pretty simple and seems to work as expected, though we've kept it in audit mode rather than enforce to begin with. We also use linkerd-stable, so we match the tags for that. Beyond that you'll probably recognize most of this from the kyverno keyless signing with Github Workflows documentation.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: cosign-verify
spec:
  validationFailureAction: Audit
  webhookTimeoutSeconds: 30
  rules:
    - name: check-image-linkerd
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "cr.l5d.io/linkerd/proxy-init:*"
          # This is default true, but confuses both argocd and linkerd check and probably more
          # things
          mutateDigest: false
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/linkerd/linkerd2-proxy-init/.github/workflows/release-proxy-init.yml@refs/tags/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    additionalExtensions:
                      githubWorkflowTrigger: push
                      githubWorkflowName: "Proxy-init release"
                      githubWorkflowRepository: "linkerd/linkerd2-proxy-init"
                    rekor:
                      url: https://rekor.sigstore.dev
        - imageReferences:
            - "cr.l5d.io/linkerd/destination:*"
            - "cr.l5d.io/linkerd/identity:*"
            - "cr.l5d.io/linkerd/metrics-api:*"
            - "cr.l5d.io/linkerd/proxy-injector:*"
            - "cr.l5d.io/linkerd/proxy:*"
            - "cr.l5d.io/linkerd/tap:*"
            - "cr.l5d.io/linkerd/web:*"
          # This is default true, but confuses both argocd and linkerd check and probably more
          # things
          mutateDigest: false
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/linkerd/linkerd2/.github/workflows/release.yml@refs/tags/stable-*"
                    issuer: "https://token.actions.githubusercontent.com"
                    additionalExtensions:
                      githubWorkflowTrigger: push
                      githubWorkflowName: "Release"
                      githubWorkflowRepository: "linkerd/linkerd2"
                    rekor:
                      url: https://rekor.sigstore.dev
...

edit

Since writing this it turned out the proxy-init check in-cluster isn't entirely happy. The kyverno test however is happy. And I forgot to include the kyverno tests!

Kyverno tests

You'll need three files: The policy above, a pod, and a test spec.

Any meshed pod will do. So you can do something like kubectl -n linkerd get pod linkerd-destination-<TAB> -o yaml > linkerd-destination.yaml

And then you'll need a file called "kyverno-test.yaml" (the filename is magic):

---
name: cosign-verify
policies:
  - /path/to/policy/cosign-verify.yaml
resources:
  - linkerd-destination.yaml
results:
  - policy: cosign-verify
    rule: check-image-linkerd
    resource: linkerd-destination-<TAB>
    kind: Pod
    result: pass
...

Run kyverno test . and you should see something like this:

Executing cosign-verify...
applying 1 policy to 1 resource... 

│───│───────────────│─────────────────────│──────────────────────────│────────│
│ # │ POLICY        │ RULE                │ RESOURCE                 │ RESULT │
│───│───────────────│─────────────────────│──────────────────────────│────────│
│ 1 │ cosign-verify │ check-image-linkerd │ /Pod/linkerd-destination │ Pass   │
│───│───────────────│─────────────────────│──────────────────────────│────────│

Test Summary: 1 tests passed and 0 tests failed

You can add more tests, but adding more in this blog post would just make it longer without adding new information. Adding checks for the other images you want to check signatures for is left as an exercise for the reader.