k8s-crd

发表于

CRD是k8s提供对现有资源进行扩展的一种方式，当现有k8s资源都无法满足你的需求时就可以考虑使用CRD对现有资源进行扩展。

环境信息：golang 1.16.2,kubernetes v1.19.0

如何自定义一个k8s CRD(Custom Resource Definition)

1.新建一个项目，项目名称为k8s-crd

1
2
3
4
➜  gopro mkdir k8s-crd
➜  gopro cd k8s-crd 
➜  k8s-crd mkdir -p pkg/apis/crd/v1
➜  k8s-crd go mod init hysyeah.top/k8s-crd

2.定义资源注册所需的字段信息，pkg/apis/crd/register.go

1
2
3
4
5
6
package  crd

const (
	GroupName = "crd.alpha.io"
	Version = "v1"
)

3.定义全局标签，pkg/apis/crd/v1/doc.go.k8s中许多代码都是通过代码生成器生成的，代码生成器通过Tags(标签)来判断一个包如何进行代码生成。k8s中存在如下两种Tag:

  • 全局Tags: 定义在每个包的doc.文件中，对整个包中的类型自动生成代码
  • 局部Tags: 定义在Go语言的类型声明上方，只对指定的类型自动生成代码

可参考deepcopy-gen

全局Tags告诉deepcopy-gen代码生成器为该包中的每个类型自动生成DeepCopy函数。其中//+groupNmae定义了资源组名称，一般使用域名形式命名

1
2
3
4
// +k8s:deepcopy-gen=package

// +groupName=crd.hysyeah.top
package v1

4.定义资源类型，pkg/apis/crd/v1/types.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// +genclient
// +genclient:noStatus
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// Machine describes a Machine resource
type Dog struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec DogSepc `json:"spec"`
}

// DogSpec定义资源拥有的属性
type DogSpec struct {
	//在这里可以自定义Dog所拥有的属性,我们这种定义了Kg和Age两个属性
	Kg int `json:"kg"`
	Age int `json:"age"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// DogList定义列表资源
type DogList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata"`
	Items []Dog `json:"items"`
}

Dog上方的Tags为局部Tags。它定义了两个代码生成器genclient和deepcopy-gen。其中genclient代码生 成器为这个资源类型自动生成对应的客户端代码,deepcopy-gen代码生成器为这个资源类型自动生成DeepCopy函数，并为该类型生成返回值为runtime.Object类型的DeepCopyObject函数。

5.定义资源注册方法,pkg/apis/crd/v1/register.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"hysyeah.top/k8s-crd/pkg/apis/crd"
)

// 定义注册资源的资源组和版本
var SchemeGroupVersion = schema.GroupVersion{
	Group: crd.GroupName,
	Version: crd.Version,
}

var (
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	AddToScheme = SchemeBuilder.AddToScheme
)

func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

func Kind(kind string) schema.GroupKind {
	return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// addKnownTypes 定义注册方法
func addKnownTypes(scheme *runtime.Schema) error  {
	scheme.AddKnownType(
		SchemeGroupVersion,
		&Dog{},
		&DogList{},
		)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}

6.配置code-generator,在项目根目录下新建目录hack并新建文件tools.go。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// +build tools

/*
   Copyright 2019 The Kubernetes Authors.

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
*/

// This package imports things required by build scripts, to force `go mod` to see them as dependencies
package tools

import _ "k8s.io/code-generator"

新建文件hack/boilerplate.go.txt,生成代码需要添加这个license

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/*
Copyright The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

新建hack/update-codegen.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/usr/bin/env bash

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -o errexit
set -o nounset
set -o pipefail

# generate the code with:
# --output-base    because this script should also be able to run inside the vendor dir of
#                  k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir
#                  instead of the $GOPATH directly. For normal projects this can be dropped.
ROOT_PACKAGE="hysyeah.top/k8s-crd"
CUSTOM_RESOURCE_NAME="crd"
CUSTOM_RESOURCE_VERSION="v1"

chmod +x ../vendor/k8s.io/code-generator/generate-groups.sh
../vendor/k8s.io/code-generator/generate-groups.sh all "$ROOT_PACKAGE/pkg/client" "$ROOT_PACKAGE/pkg/apis" "$CUSTOM_RESOURCE_NAME:$CUSTOM_RESOURCE_VERSION" \
  --go-header-file $(pwd)/boilerplate.go.txt

7.生成代码

1
2
3
4
5
6
7
8
9
//在项目根目录下
➜  k8s-crd go mod vendor
➜  k8s-crd chmod +x hack/update-codegen.sh
➜  k8s-crd cd hack 
➜  hack./update-codegen.sh
Generating deepcopy funcs
Generating clientset for crd:v1 at hysyeah.top/k8s-crd/pkg/client/clientset
Generating listers for crd:v1 at hysyeah.top/k8s-crd/pkg/client/listers
Generating informers for crd:v1 at hysyeah.top/k8s-crd/pkg/client/informers

执行完代码生成命令后，根据日志可能看出代码已经生成出来了，但是你却发现生成的文件并不在项目目录中，这是什么原因呢？
其实代码生成的路径是$GOPATH/src/hysyeah.top/k8s-crd，将生成代码拷贝至项目中。此时项目结构如下,其中红框中的代码是自动生成的：
image

8.有了自定义资源，我们还需要定义如何使用它，不然这个资源也无用之地。而对于资源的如何使用k8s是通过controller来进行资源控制的。

  • 新建目录pkg/signals,接收系统信号，直接使用社区代码
  • controller.go,业务逻辑代码，可以针对资源不同的事件注册对应的函数
  • main.go, 入口函数

9.编译，生成一个二进制文件

1
2
go mod vendor
go build -o dogcontroller

10.创建CRD

1
2
3
4
5
➜  k8s-crd kubectl apply -f Dog.yaml
customresourcedefinition.apiextensions.k8s.io/dogs.crd.hysyeah.top created
➜  k8s-crd kubectl get crd
NAME                   CREATED AT
dogs.crd.hysyeah.top   2021-05-15T12:25:01Z

11.创建Dog实例

1
2
➜  k8s-crd kubectl apply -f example-dog.yaml
dog.crd.hysyeah.top/alpha-dog created

12.运行dogcontroller

1
./dogcontroller -kubeconfig=$HOME/.kube/config

13.分别执行相应的操作，controller会解发相应的动作。以删除dog实例为例。
image

小结：
以上便是手动构建CRD的过程，可以使我们更好的了解CRD和Controller的工作原理。其实工作中如果要自定义一个CRD和Controller大可不必那么麻烦，可以借助一些框架使整个过程更方便，如kubebuilder,operator-sdk。

参考：
1.https://github.com/kangxiaoning/learn-kubernetes-crd/blob/main/README.md
2.https://github.com/kubernetes/code-generator