k8s之kube-proxy[上]

kube-proxy 是集群中每个节点(node)上所运行的网络代理, 是实现 Kubernetes 服务(Service) 概念的一部分。

kube-proxy 维护节点上的一些网络规则, 这些网络规则会允许从集群内部或外部的网络会话与 Pod 进行网络通信。


一次请求过程中需要kube-proxy参与吗?

做个实验:在kube-proxy正常工作的情况下使用上面的yaml文件创建Service,发现服务能正常访问。关闭kube-proxy发现服务也能正常访问。
关闭kube-proxy重新创建Service然后你会发现服务不能访问。这里可以先得出一个结论: kube-proxy只会在Service创建时创建一些规则(iptables/ipvs),
然后在Service在被访问的过程,kube-proxy并不参与。


使用下面的yaml文件创建Service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-curl
spec:
replicas: 1
selector:
matchLabels:
app: my-curl
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: my-curl
spec:
containers:
- name: my-curl
image: hysyeah/my-curl:v1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
name: my-curl
spec:
selector:
app: my-curl
ports:
- name: http
port: 8080
targetPort: 8080
type: NodePort

iptables的每条链下面的规则处理顺序都是从上往下逐条遍历,除非遇到DROP,REJECT,RETURN。如果链下面是自定义链,则跳转到
对应的自定义链执行链下的所有规则,然后跳转回来执行原来那条链后面的规则。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
# 
# ServiceIP: 10.98.75.173
# PodIP: 10.244.0.221
➜ ✗ sudo iptables -L -n -t nat

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
KUBE-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
MASQUERADE all -- 192.168.250.0/24 !192.168.250.0/24 /* managed by anbox-bridge */
FLANNEL-POSTRTG all -- 0.0.0.0/0 0.0.0.0/0 /* flanneld masq */

Chain FLANNEL-POSTRTG (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0 mark match 0x4000/0x4000 /* flanneld masq */
RETURN all -- 10.244.0.0/24 10.244.0.0/16 /* flanneld masq */
RETURN all -- 10.244.0.0/16 10.244.0.0/24 /* flanneld masq */
RETURN all -- !10.244.0.0/16 10.244.0.0/24 /* flanneld masq */
MASQUERADE all -- 10.244.0.0/16 !224.0.0.0/4 /* flanneld masq */ random-fully
MASQUERADE all -- !10.244.0.0/16 10.244.0.0/16 /* flanneld masq */ random-fully

Chain KUBE-EXT-W7DCC3F5ZN5NE3KW (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 0.0.0.0/0 0.0.0.0/0 /* masquerade traffic for default/my-curl:http external destinations */
KUBE-SVC-W7DCC3F5ZN5NE3KW all -- 0.0.0.0/0 0.0.0.0/0

Chain KUBE-KUBELET-CANARY (0 references)
target prot opt source destination

Chain KUBE-MARK-DROP (0 references)
target prot opt source destination
MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000

Chain KUBE-MARK-MASQ (14 references)
target prot opt source destination
MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
KUBE-EXT-W7DCC3F5ZN5NE3KW tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/my-curl:http */ tcp dpt:31122

Chain KUBE-POSTROUTING (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000/0x4000
MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK xor 0x4000
MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ random-fully

Chain KUBE-PROXY-CANARY (0 references)
target prot opt source destination

Chain KUBE-SEP-2KOR5GS2OWGIGVHG (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.220 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.0.220:53

Chain KUBE-SEP-E5ZPFNRTFC3O2YGB (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 192.168.2.123 0.0.0.0/0 /* default/kubernetes:https */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ tcp to:192.168.2.123:6443

Chain KUBE-SEP-ELL5SOECSE25BPHX (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.220 0.0.0.0/0 /* kube-system/kube-dns:dns */
DNAT udp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.244.0.220:53

Chain KUBE-SEP-G3W5CCH2EXTJTWNT (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.219 0.0.0.0/0 /* kube-system/kube-dns:metrics */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.244.0.219:9153

Chain KUBE-SEP-G5TPU3TRPGLSSBUB (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.220 0.0.0.0/0 /* kube-system/kube-dns:metrics */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.244.0.220:9153

Chain KUBE-SEP-IJMQAJ4HGPA7P6ZX (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.219 0.0.0.0/0 /* kube-system/kube-dns:dns */
DNAT udp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.244.0.219:53

Chain KUBE-SEP-SZ5GI3C7S6AQLVNR (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.221 0.0.0.0/0 /* default/my-curl:http */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/my-curl:http */ tcp to:10.244.0.221:8080

Chain KUBE-SEP-ZXBKTOI5KHHUPKIG (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.244.0.219 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.0.219:53

Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-SVC-JD5MR3NA4I4DYORP tcp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
KUBE-SVC-W7DCC3F5ZN5NE3KW tcp -- 0.0.0.0/0 10.98.75.173 /* default/my-curl:http cluster IP */ tcp dpt:8080
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- 0.0.0.0/0 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-NODEPORTS all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-SEP-ZXBKTOI5KHHUPKIG all -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp -> 10.244.0.219:53 */ statistic mode random probability 0.50000000000
KUBE-SEP-2KOR5GS2OWGIGVHG all -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp -> 10.244.0.220:53 */

Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.96.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
KUBE-SEP-G3W5CCH2EXTJTWNT all -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics -> 10.244.0.219:9153 */ statistic mode random probability 0.50000000000
KUBE-SEP-G5TPU3TRPGLSSBUB all -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics -> 10.244.0.220:9153 */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SEP-E5ZPFNRTFC3O2YGB all -- 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https -> 192.168.2.123:6443 */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
target prot opt source destination
KUBE-MARK-MASQ udp -- !10.244.0.0/16 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-SEP-IJMQAJ4HGPA7P6ZX all -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns -> 10.244.0.219:53 */ statistic mode random probability 0.50000000000
KUBE-SEP-ELL5SOECSE25BPHX all -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns -> 10.244.0.220:53 */

Chain KUBE-SVC-W7DCC3F5ZN5NE3KW (2 references)
target prot opt source destination
KUBE-MARK-MASQ tcp -- !10.244.0.0/16 10.98.75.173 /* default/my-curl:http cluster IP */ tcp dpt:8080
KUBE-SEP-SZ5GI3C7S6AQLVNR all -- 0.0.0.0/0 0.0.0.0/0 /* default/my-curl:http -> 10.244.0.221:8080 */

Chain PREROUTING中的规则执行顺序如下:
KUBE-SERVICES我们只关注我们创建的Service这条链KUBE-SVC-W7DCC3F5ZN5NE3KW

KUBE-SERVICES —> KUBE-SERVICES —> KUBE-SVC-W7DCC3F5ZN5NE3KW —> KUBE-MARK-MASQ —> KUBE-SEP-SZ5GI3C7S6AQLVNR
—> KUBE-MARK-MASQ —> DNAT —> KUBE-NODEPORTS —> KUBE-EXT-W7DCC3F5ZN5NE3KW —> KUBE-MARK-MASQ —>
KUBE-SVC-W7DCC3F5ZN5NE3KW —> KUBE-MARK-MASQ —> KUBE-SEP-SZ5GI3C7S6AQLVNR
—> KUBE-MARK-MASQ —> DNAT(10.244.0.221:8080)
最终会把请求转发到pod内。

在链KUBE-SERVICES下每一个Service都会有一条自定义链,在一次请求过程中需要一个一个遍历执行,如果Service过多则有很大的性能消耗。


这些iptables规则是怎么生成的呢?

kube-proxy会监听以下资源:

Endpointskube-proxy会监听ServiceEndpoints之间的变化。当ServiceEndpoint发生变化时,kube-proxy会更新相应的iptables规则或IPVS规则,以确保流量正确地路由到新的Endpoints

Servicekube-proxy会监听Service的创建、更新和删除。当Service发生变化时,kube-proxy会相应地更新相应的iptables规则或IPVS规则。

Nodekube-proxy会监听Node的变化,例如节点的加入或离开集群。当节点发生变化时,kube-proxy会相应地更新iptables规则或IPVS规则,以确保流量正确地路由到可用的节点。

通过监听这些资源的变化,kube-proxy能够动态地更新网络规则,以确保流量能够正确地路由到集群中的Pod

下篇文章我们将讲下kube-proxy的具体实现。