how to write a k8s admission webhook

置顶 | 发表于
什么是admission webhook

Kubernetes Ａdmission　Webhook是一种HTTP回调机制，它允许Kubernetes调用外部Web服务，以便在某些事件发生时执行自定义代码。

Admission Webhook是Kubernetes提供的一种扩展机制，用于在资源被持久化到etcd之前，对资源进行验证或修改。Admission Webhook可以分为两种类型：Validation Webhook和Mutating Webhook。

Validation Webhook用于验证资源是否符合预期的规则，如果资源不符合规则，则会拒绝资源被持久化到etcd。

Mutating Webhook则可以对资源进行修改，在资源被持久化到etcd之前，将资源修改为期望的状态。允许对请求进行更改，例如对Pod进行注入，以添加一些特定于应用程序的设置，如日志记录、密钥管理、监视等。

Webhook机制使得用户可以根据自己的需求编写和部署自己的代码，以扩展和定制Kubernetes平台的行为。

Webhook何时调用,

图片来源于网络

如何编写一个webhook
Webhook请求与响应
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
// 结构体定义了请求和响应的字段
// staging/src/k8s.io/api/admission/v1/types.go
type AdmissionReview struct {
	metav1.TypeMeta `json:",inline"`
	// Request describes the attributes for the admission request.
	// +optional
	Request *AdmissionRequest `json:"request,omitempty" protobuf:"bytes,1,opt,name=request"`
	// Response describes the attributes for the admission response.
	// +optional
	Response *AdmissionResponse `json:"response,omitempty" protobuf:"bytes,2,opt,name=response"`
}

type AdmissionRequest struct {
	UID types.UID `json:"uid" protobuf:"bytes,1,opt,name=uid"`
	// Kind is the fully-qualified type of object being submitted (for example, v1.Pod or autoscaling.v1.Scale)
	Kind metav1.GroupVersionKind `json:"kind" protobuf:"bytes,2,opt,name=kind"`
	// Resource is the fully-qualified resource being requested (for example, v1.pods)
	Resource metav1.GroupVersionResource `json:"resource" protobuf:"bytes,3,opt,name=resource"`
	// SubResource is the subresource being requested, if any (for example, "status" or "scale")
	// +optional
	SubResource string `json:"subResource,omitempty" protobuf:"bytes,4,opt,name=subResource"`

	// +optional
	RequestKind *metav1.GroupVersionKind `json:"requestKind,omitempty" protobuf:"bytes,13,opt,name=requestKind"`

	// +optional
	RequestResource *metav1.GroupVersionResource `json:"requestResource,omitempty" protobuf:"bytes,14,opt,name=requestResource"`

	// +optional
	RequestSubResource string `json:"requestSubResource,omitempty" protobuf:"bytes,15,opt,name=requestSubResource"`

	// Name is the name of the object as presented in the request.  On a CREATE operation, the client may omit name and
	// rely on the server to generate the name.  If that is the case, this field will contain an empty string.
	// +optional
	Name string `json:"name,omitempty" protobuf:"bytes,5,opt,name=name"`
	// Namespace is the namespace associated with the request (if any).
	// +optional
	Namespace string `json:"namespace,omitempty" protobuf:"bytes,6,opt,name=namespace"`
	// Operation is the operation being performed. This may be different than the operation
	// requested. e.g. a patch can result in either a CREATE or UPDATE Operation.
	Operation Operation `json:"operation" protobuf:"bytes,7,opt,name=operation"`
	// UserInfo is information about the requesting user
	UserInfo authenticationv1.UserInfo `json:"userInfo" protobuf:"bytes,8,opt,name=userInfo"`
	// Object is the object from the incoming request.
	// +optional
	Object runtime.RawExtension `json:"object,omitempty" protobuf:"bytes,9,opt,name=object"`
	// OldObject is the existing object. Only populated for DELETE and UPDATE requests.
	// +optional
	OldObject runtime.RawExtension `json:"oldObject,omitempty" protobuf:"bytes,10,opt,name=oldObject"`

	// +optional
	DryRun *bool `json:"dryRun,omitempty" protobuf:"varint,11,opt,name=dryRun"`

	// +optional
	Options runtime.RawExtension `json:"options,omitempty" protobuf:"bytes,12,opt,name=options"`
}

type AdmissionResponse struct {
	// UID is an identifier for the individual request/response.
	// This must be copied over from the corresponding AdmissionRequest.
	UID types.UID `json:"uid" protobuf:"bytes,1,opt,name=uid"`

	// Allowed indicates whether or not the admission request was permitted.
	Allowed bool `json:"allowed" protobuf:"varint,2,opt,name=allowed"`

	// Result contains extra details into why an admission request was denied.
	// This field IS NOT consulted in any way if "Allowed" is "true".
	// +optional
	Result *metav1.Status `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`

	// The patch body. Currently we only support "JSONPatch" which implements RFC 6902.
	// +optional
	Patch []byte `json:"patch,omitempty" protobuf:"bytes,4,opt,name=patch"`

	// The type of Patch. Currently we only allow "JSONPatch".
	// +optional
	PatchType *PatchType `json:"patchType,omitempty" protobuf:"bytes,5,opt,name=patchType"`

	// +optional
	AuditAnnotations map[string]string `json:"auditAnnotations,omitempty" protobuf:"bytes,6,opt,name=auditAnnotations"`

	// +optional
	Warnings []string `json:"warnings,omitempty" protobuf:"bytes,7,rep,name=warnings"`
}
  • 请求: Webhook 发送 POST 请求时，请设置 Content-Type: application/json 并对 admission.k8s.io API 组中的 AdmissionReview 对象进行序列化，将所得到的 JSON 作为请求的主体。
  • 响应: Webhook 使用 HTTP 200 状态码、Content-Type: application/json 和一个包含 AdmissionReview 对象的 JSON 序列化格式来发送响应。该 AdmissionReview 对象与发送的版本相同，且其中包含的 response 字段已被有效填充。
    响应示例:
    1
    2
    3
    4
    5
    6
    7
    8
    {
      "apiVersion": "admission.k8s.io/v1",
      "kind": "AdmissionReview",
      "response": {
        "uid": "<value from request.uid>",
        "allowed": true
      }
    }
开始编写Webhook

  1. 配置ValidatingAdmissionWebhook,此准入控制器调用与请求匹配的所有验证性 Webhook。 匹配的 Webhook 将被并行调用。如果其中任何一个拒绝请求，则整个请求将失败。 该准入控制器仅在验证（Validating）阶段运行

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingWebhookConfiguration
    metadata:
      name: pod-webhook
    webhooks:
      - name:  pod.webhook.hysyeah.com
        # 定义访问的服务,如果是外部服务则指定对应的URL
        clientConfig:
          service:
            name: pod-webhook
            namespace: default
            path: "/configmaps"
          caBundle: "update <>"
        # 匹配规则,如果传入请求与rules的指定operations,groups,version,resources匹配
        # 则该请求将发送到webhook
        rules:
          - operations: [ "CREATE", "UPDATE", "DELETE"]
            apiGroups: ["apps", ""]
            apiVersions: ["v1"]
            resources: ["configmaps"]
        failurePolicy: Ignore
        sideEffects: None
        admissionReviewVersions:
          - v1

  2. 配置MutatingWebhookConfiguration

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    apiVersion: admissionregistration.k8s.io/v1
    kind: MutatingWebhookConfiguration
    metadata:
      name: pod-webhook
    webhooks:
      - name: pod.webhook.hysyeah.com
        rules:
          - apiGroups:
              - ""
            apiVersions:
              - v1
            operations:
              - CREATE
              - UPDATE
            resources:
              - pods
        failurePolicy: Ignore
        sideEffects: None
        admissionReviewVersions:
          - v1
        clientConfig:
          # base64 -w 0 ca.crt
          caBundle: "update <>"
          service:
            name: pod-webhook
            namespace: default
            path: "/add-label"

  3. 构建镜像
    nerdctl build -t hysyeah/pod-webhook:v3 .

1
2
3
4
5
6
7
8
9
10
11
12
13
FROM golang:1.18-alpine

WORKDIR /app

COPY . .
RUN go env -w GO111MODULE=on
RUN go env -w GOPROXY=https://goproxy.cn,direct
RUN go build -o main

RUN chmod +x main

EXPOSE 443
CMD ["./main"]
  1. 编写对应的deployment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pod-webhook
  template:
    metadata:
      labels:
        app: pod-webhook
    spec:
      containers:
      - name: pod-webhook
        image: hysyeah/pod-webhook:v3
        imagePullPolicy: IfNotPresent
        command: ["/app/main"]
        args: ["-tls-cert-file=/keys/tls.crt","-tls-private-key-file=/keys/tls.key"]
        ports:
        - containerPort: 443
        volumeMounts:
        - name: tls-keys
          mountPath: /keys
      volumes:
        - name: tls-keys
          secret:
            secretName: pod-webhook-tls
            items:
            - key: tls.crt
              path: tls.crt
            - key: tls.key
              path: tls.key

---
apiVersion: v1
kind: Service
metadata:
  name: pod-webhook
spec:
  selector:
    app: pod-webhook
  ports:
    - name: tcp
      port: 443
      targetPort: 443

  1. 部署webhook
    执行kubectl apply -f deployment.yaml

  2. 验证webhook

  • 新建一个pod,发现pod添加了label: added-label=yes

    1
    2
    NAME                         READY   STATUS    RESTARTS   AGE     LABELS
    my-curl                      1/1     Running   0          170m    added-label=yes,app=my-curl

  • 新建一个configmap,包含webhook-e2e-test: webhook-disallow
    结果如下:

    1
    Error from server (the configmap contains unwanted key and value): error when creating "configmap.yaml": admission webhook "pod.webhook.hysyeah.com" denied the request: the configmap contains unwanted key and value

完整的代码可查看完整代码

注意事项
  • 确保启用 MutatingAdmissionWebhook 和ValidatingAdmissionWebhook控制器
  • 确保启用了 admissionregistration.k8s.io/v1 API
  • 确保您的代码不会影响Kubernetes集群的稳定性和安全性
  • 当用户尝试创建的对象与返回的对象不同时，用户可能会感到困惑。
  • 与覆盖原始请求中设置的字段相比，使用原始请求未设置的字段会引起问题的可能性较小。 应尽量避免覆盖原始请求中的字段设置

REF:
1.https://github.com/jpeeler/podpreset-crd/tree/master/webhook
2.https://github.com/kubernetes/kubernetes/tree/release-1.21/test/images/agnhost/webhook
3.https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/extensible-admission-controllers/
4.https://banzaicloud.com/blog/k8s-admission-webhooks/