k8s部署中间件
目前于超老师的java程序部署，数据库中间件（mysql/redis）都是部署在另一套台机器的纯docker中。
[root@docker01 ~]#docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ea36dacd7456 redis:3.2 "docker-entrypoint.s…" 19 hours ago Up 19 hours 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp redis
6e538d8769c7 mysql:5.7 "docker-entrypoint.s…" 19 hours ago Up 19 hours 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql
c5e7dd973e6e registry "/entrypoint.sh /etc…" 19 hours ago Up 19 hours 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp registry
[root@docker01 ~]#
现需求，将数据库也运行在pod里，该如何改造？
Redis-pod改造
pod-redis.yml
apiVersion: v1
kind: Pod
metadata:
name: redis
namespace: yuchao
labels:
app: redis
spec:
# hostNetwork: true
containers:
- name: redis
image: redis:3.2
ports:
- containerPort: 6379
创建
[root@k8s-master ~/k8s-all]#vim pod-redis.yaml
[root@k8s-master ~/k8s-all]#
[root@k8s-master ~/k8s-all]#kubectl create -f pod-redis.yaml
pod/redis created
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get po
NAME READY STATUS RESTARTS AGE
eladmin-api 1/1 Running 0 22h
redis 0/1 ContainerCreating 0 5s
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
eladmin-api 1/1 Running 0 22h 10.244.2.7 k8s-slave1 <none> <none>
redis 1/1 Running 0 22s 10.244.1.4 k8s-slave2 <none> <none>
问题，eladmin后台怎么连接redis？
[root@k8s-master ~/k8s-all]#telnet 10.244.1.4 6379
Trying 10.244.1.4...
Connected to 10.244.1.4.
Escape character is '^]'.
^]
telnet> Connection closed.
#走ip链接是可以，但是pod重建，pod-ip动态变化，下一次再改配置文件？
#那不行，得走dns，走域名，或者负载均衡。这就是后续的知识点。
问题演示，pod重建，pod-ip变化
# 建议用这个
[root@k8s-master ~/k8s-all]#kubectl -n yuchao delete pod redis
pod "redis" deleted
# 或者
[root@k8s-master ~/k8s-all]#kubectl delete -f pod-redis.yaml
# 重建redis，ip变化了，pod。
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
eladmin-api 1/1 Running 0 22h 10.244.2.7 k8s-slave1 <none> <none>
redis 1/1 Running 0 4s 10.244.1.5 k8s-slave2 <none> <none>
我们没法使用一个可变的ip在业务后端配置文件中写死，因此需要使用k8s提供的service资源。
hostNetwork了解
这里只作为了解，让pod使用宿主机ip，好比docker的host模式
[root@k8s-master ~/k8s-all]#kubectl explain pod.spec|grep -i network
DNSPolicy. To have DNS options set along with hostNetwork, you have to
unless hostNetwork is true, if it is available, then fall back on the
the pod's hosts file if specified. This is only valid for non-hostNetwork
hostNetwork <boolean>
Host networking requested for this pod. Use the host's network namespace.
https://git.k8s.io/enhancements/keps/sig-network/580-pod-readiness-gates
[root@k8s-master ~/k8s-all]#
[root@k8s-master ~/k8s-all]#cat pod-redis.yaml
apiVersion: v1
kind: Pod
metadata:
name: redis
namespace: yuchao
labels:
app: redis
spec:
hostNetwork: true
containers:
- name: redis
image: redis:3.2
ports:
- containerPort: 6379
创建pod，使用宿主机网络
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
eladmin-api 1/1 Running 0 22h 10.244.2.7 k8s-slave1 <none> <none>
redis 1/1 Running 0 6s 10.0.0.82 k8s-slave2 <none> <none>
[root@k8s-master ~/k8s-all]#curl 10.0.0.82:6379
-ERR wrong number of arguments for 'get' command
-ERR unknown command 'User-Agent:'
[root@k8s-master ~/k8s-all]#
目前只是部署到了slave2机器，如果调度到slave1机器呢？所以也不行。
也有特殊的方案，固定部署在某个机器节点，但是不符合k8s的灵活调度理念，不推荐使用。
引入service资源
service-redis.yaml
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: yuchao
spec:
ports:
- port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis
type: ClusterIP
创建svc，service通过label找到对应的redis-pod，以及生成了集群内固定的一个svc-ip地址
[root@k8s-master ~/k8s-all]#
[root@k8s-master ~/k8s-all]#kubectl create -f svc-redis.yaml
service/redis created
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis ClusterIP 10.111.94.30 <none> 6379/TCP 4s
访问svc-ip，就找到后端的pod。
[root@k8s-master ~/k8s-all]#
[root@k8s-master ~/k8s-all]#curl 10.111.94.30:6379
-ERR wrong number of arguments for 'get' command
-ERR unknown command 'User-Agent:'
查看service的Endpoints，负载均衡的后端节点
[root@k8s-master ~/k8s-all]#kubectl -n yuchao describe svc redis
Name: redis
Namespace: yuchao
Labels: <none>
Annotations: <none>
Selector: app=redis
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.111.94.30
IPs: 10.111.94.30
Port: <unset> 6379/TCP
TargetPort: 6379/TCP
Endpoints: 10.244.1.6:6379
Session Affinity: None
Events: <none>
[root@k8s-master ~/k8s-all]#
图解svc与pod
service的ip地址是可以固定在yaml中写死的，用于服务发现。
controller-manager
↓
endpoint-controller-manager
↓
Service
↓
read-server（pod）
svc网段设置
k8s初始化时，配置的网段
# 方法1
[root@k8s-master ~/k8s-all]#kubectl cluster-info dump | grep service-cluster-ip-range
"--service-cluster-ip-range=10.96.0.0/12",
"--service-cluster-ip-range=10.96.0.0/12",
# 方法2
[root@k8s-master ~/k8s-all]#cat /etc/kubernetes/manifests/kube-controller-manager.yaml |grep service
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/12
- --use-service-account-credentials=true
service重要性
在Kubernetes中，Service（服务）是一种抽象的概念，它为一组 Pod 提供了一个统一的入口，使得它们可以被其他应用程序访问和使用。Service是Kubernetes中的核心资源之一，它扮演了多种重要角色，包括：
- 定义稳定的网络地址：通过Service，应用程序可以在不关心后端Pod的具体位置的情况下，使用一个稳定的网络地址来访问后端应用程序，从而提高应用程序的可靠性和可维护性。
- 负载均衡：当有多个后端Pod时，Service可以通过在Pod之间分配负载来实现负载均衡，从而提高应用程序的可伸缩性和性能。
- 服务发现：通过Service，其他应用程序可以发现并连接到后端Pod，从而实现微服务之间的相互通信。
- 外部暴露：Service还可以将后端Pod暴露到集群外部，从而使得外部应用程序可以访问后端应用程序。
综上所述，Service在Kubernetes中具有非常重要的作用，它为Kubernetes中的应用程序提供了稳定的网络地址、负载均衡、服务发现和外部暴露等功能，是Kubernetes中不可或缺的核心资源之一。
mysql-pod改造
思考点
docker启动mysql，你注意哪些问题
docker run -d -p 3306:3306 --name mysql -v /opt/mysql:/var/lib/mysql -e MYSQL_DATABASE=eladmin -e MYSQL_ROOT_PASSWORD=www.yuchaoit.cn mysql:5.7 --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
一样的，k8s中运行mysql，也需要注意上述的参数，密码，镜像，字符集，容器数据持久化存储。
- mysql数据存储，在哪个节点？
- 如何传入环境变量、参数？
因此诞生了如下的yaml过程，当然有更专业的用法，如提供数据卷的持久化pv/pvc，继续向后学习。
pod-mysql.yaml
apiVersion: v1
kind: Pod
metadata:
name: mysql
namespace: yuchao
labels:
app: mysql
spec:
nodeSelector: # 使用节点选择器将Pod调度到指定label的节点
mysql: "true"
containers:
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_DATABASE # 指定数据库地址
value: "eladmin"
- name: MYSQL_ROOT_PASSWORD
value: "www.yuchaoit.cn"
ports:
- containerPort: 3306
args:
- --character-set-server=utf8mb4
- --collation-server=utf8mb4_unicode_ci
volumeMounts:
- name: mysql-data
mountPath: /var/lib/mysql
volumes:
- name: mysql-data
hostPath:
path: /opt/mysql/
改进yaml
这里需要考虑，mysql的数据，存储到哪个k8s节点上，那么得固定mysql的pod运行节点，使用nodeSelector
apiVersion: v1
kind: Pod
metadata:
name: mysql
namespace: yuchao
labels:
app: mysql
spec:
nodeSelector: # 使用节点选择器将Pod调度到指定label的节点
mysql: "true"
containers:
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_DATABASE # 指定数据库地址
value: "eladmin"
- name: MYSQL_ROOT_PASSWORD
value: "www.yuchaoit.cn"
ports:
- containerPort: 3306
args:
- --character-set-server=utf8mb4
- --collation-server=utf8mb4_unicode_ci
volumeMounts:
- name: mysql-data
mountPath: /var/lib/mysql
volumes:
- name: mysql-data
hostPath:
path: /opt/mysql/
给节点打标签
# 给节点打标签
kubectl label node k8s-master mysql=true
查看标签
[root@k8s-master ~/k8s-all]#kubectl get no --show-labels
NAME STATUS ROLES AGE VERSION LABELS
k8s-master Ready control-plane 3d2h v1.24.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master,kubernetes.io/os=linux,mysql=true,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
k8s-slave1 Ready <none> 3d2h v1.24.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-slave1,kubernetes.io/os=linux
k8s-slave2 Ready <none> 3d2h v1.24.4 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-slave2,kubernetes.io/os=linux
[root@k8s-master ~/k8s-all]#
创建pod-mysql
[root@k8s-master ~/k8s-all]#kubectl create -f pod-mysql.yml
pod/mysql created
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
eladmin-api 1/1 Running 0 24h 10.244.2.7 k8s-slave1 <none> <none>
mysql 1/1 Running 0 53s 10.244.0.4 k8s-master <none> <none>
redis 1/1 Running 0 43m 10.244.1.6 k8s-slave2 <none> <none>
[root@k8s-master ~/k8s-all]#
查看下载的mysql镜像，以及运行的具体container
[root@k8s-master ~/k8s-all]#nerdctl -n k8s.io ps |grep mysql
60cec1b2979a docker.io/library/mysql:5.7 "docker-entrypoint.s…" 55 seconds ago Up k8s://yuchao/mysql/mysql
d188590580f8 registry.aliyuncs.com/google_containers/pause:3.6 "/pause" About a minute ago Up k8s://yuchao/mysql
[root@k8s-master ~/k8s-all]#
[root@k8s-master ~/k8s-all]#nerdctl -n k8s.io images|grep mysql
mysql 5.7 f2ad209efe9c About a minute ago linux/amd64 440.3 MiB 147.7 MiB
[root@k8s-master ~/k8s-all]#
检查宿主机持久化的数据
[root@k8s-master ~/k8s-all]# ls /opt/mysql/ -l
total 188484
-rw-r----- 1 polkitd input 56 Mar 13 02:02 auto.cnf
-rw------- 1 polkitd input 1680 Mar 13 02:02 ca-key.pem
-rw-r--r-- 1 polkitd input 1112 Mar 13 02:02 ca.pem
-rw-r--r-- 1 polkitd input 1112 Mar 13 02:02 client-cert.pem
-rw------- 1 polkitd input 1676 Mar 13 02:02 client-key.pem
drwxr-x--- 2 polkitd input 20 Mar 13 02:02 eladmin
-rw-r----- 1 polkitd input 1353 Mar 13 02:02 ib_buffer_pool
-rw-r----- 1 polkitd input 79691776 Mar 13 02:02 ibdata1
-rw-r----- 1 polkitd input 50331648 Mar 13 02:02 ib_logfile0
-rw-r----- 1 polkitd input 50331648 Mar 13 02:02 ib_logfile1
-rw-r----- 1 polkitd input 12582912 Mar 13 02:02 ibtmp1
drwxr-x--- 2 polkitd input 4096 Mar 13 02:02 mysql
drwxr-x--- 2 polkitd input 8192 Mar 13 02:02 performance_schema
-rw------- 1 polkitd input 1676 Mar 13 02:02 private_key.pem
-rw-r--r-- 1 polkitd input 452 Mar 13 02:02 public_key.pem
-rw-r--r-- 1 polkitd input 1112 Mar 13 02:02 server-cert.pem
-rw------- 1 polkitd input 1680 Mar 13 02:02 server-key.pem
drwxr-x--- 2 polkitd input 8192 Mar 13 02:02 sys
[root@k8s-master ~/k8s-all]#id polkitd
uid=999(polkitd) gid=998(polkitd) groups=998(polkitd)
[root@k8s-master ~/k8s-all]#ps -ef|grep polkit
polkitd 684 1 0 Mar09 ? 00:00:00 /usr/lib/polkit-1/polkitd --no-debug
polkitd 127970 127713 0 02:02 ? 00:00:00 mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
root 129167 118663 0 02:05 pts/1 00:00:00 grep --color=auto polkit
Polkitd 是 Linux 操作系统中的一个守护进程，全名为 PolicyKit daemon。它为非特权进程提供了以特权身份执行操作的机制。
在 Linux 中，许多任务需要 root 用户权限才能执行，但是将所有进程都赋予 root 权限会增加安全风险。
Polkitd 的出现解决了这个问题，它允许系统管理员为每个进程指定访问特定资源所需的最低权限。
例如，您可以为普通用户授予访问网络设置的权限，而无需将它们添加到 sudoers 列表中。
Polkitd 通过从本地或远程身份验证源（如本地密码文件、LDAP 目录或 Kerberos KDC）获取身份验证信息，然后根据授权策略来确定进程是否允许执行请求的操作。授权策略通常以 XML 格式存储在 /usr/share/polkit-1/actions 目录中，管理员可以通过修改这些策略来更改系统中进程的权限。
Polkitd 还允许管理员创建自定义策略，以便更好地控制系统的安全性。
进入mysql容器
[root@k8s-master ~/k8s-all]#kubectl -n yuchao exec -it mysql -- bash
root@mysql:/# env
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
MYSQL_MAJOR=5.7
REDIS_PORT_6379_TCP=tcp://10.111.94.30:6379
REDIS_SERVICE_HOST=10.111.94.30
HOSTNAME=mysql
PWD=/
MYSQL_ROOT_PASSWORD=www.yuchaoit.cn
HOME=/root
REDIS_SERVICE_PORT=6379
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
MYSQL_VERSION=5.7.36-1debian10
GOSU_VERSION=1.12
REDIS_PORT_6379_TCP_PROTO=tcp
TERM=xterm
SHLVL=1
REDIS_PORT_6379_TCP_PORT=6379
KUBERNETES_PORT_443_TCP_PROTO=tcp
MYSQL_DATABASE=eladmin
REDIS_PORT_6379_TCP_ADDR=10.111.94.30
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
REDIS_PORT=tcp://10.111.94.30:6379
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
root@mysql:/#
svc-mysql创建
同理mysql也需要svc实现负载均衡
apiVersion: v1
kind: Service
metadata:
name: mysql
namespace: yuchao
spec:
ports:
- port: 3306 # svc代理地址
protocol: TCP
targetPort: 3306 # 后端真实端口号，目标端口，找到后端pod-ip:3306
selector:
app: mysql
type: ClusterIP
创建svc
[root@k8s-master ~/k8s-all]#kubectl create -f svc-mysql.yml
service/mysql created
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mysql ClusterIP 10.110.3.172 <none> 3306/TCP 5s
redis ClusterIP 10.111.94.30 <none> 6379/TCP 59m
# 测试svc
[root@k8s-master ~/k8s-all]#curl 10.110.3.172:3306
5.7.36wM/DÿÿA^U(WhJ?M?
mysql_native_password!ÿ#08S01Got packets out of order
查看svc
[root@k8s-master ~/k8s-all]#kubectl -n yuchao describe svc mysql
Name: mysql
Namespace: yuchao
Labels: <none>
Annotations: <none>
Selector: app=mysql
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.110.3.172
IPs: 10.110.3.172
Port: <unset> 3306/TCP
TargetPort: 3306/TCP
Endpoints: 10.244.0.4:3306
Session Affinity: None
Events: <none>
[root@k8s-master ~/k8s-all]#
图解
测试重建mysql数据是否保留
[root@k8s-master ~/k8s-all]#kubectl -n yuchao exec -it mysql bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@mysql:/# mysql -uroot -pwww.yuchaoit.cn
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.7.36 MySQL Community Server (GPL)
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create database yuchaoit;
Query OK, 1 row affected (0.01 sec)
mysql>
# 检查持久化节点
[root@k8s-master ~/k8s-all]#ls /opt/mysql/yuchaoit/
db.opt
# 重建mysql-pod
[root@k8s-master ~/k8s-all]#kubectl delete -f pod-mysql.yml
pod "mysql" deleted
[root@k8s-master ~/k8s-all]#kubectl create -f pod-mysql.yml
pod/mysql created
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
eladmin-api 1/1 Running 0 24h 10.244.2.7 k8s-slave1 <none> <none>
mysql 1/1 Running 0 6s 10.244.0.5 k8s-master <none> <none>
redis 1/1 Running 0 61m 10.244.1.6 k8s-slave2 <none> <none>
[root@k8s-master ~/k8s-all]#
# 发现mysql-pod ip地址从 0.4 > 0.5
# 数据呢？
[root@k8s-master ~/k8s-all]#kubectl -n yuchao exec -it mysql -- bash
root@mysql:/# mysql -uroot -pwww.yuchaoit.cn
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.36 MySQL Community Server (GPL)
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| eladmin |
| mysql |
| performance_schema |
| sys |
| yuchaoit |
+--------------------+
6 rows in set (0.00 sec)
mysql>
# OK没问题，可以重新部署eladmin后端
重建aladmin后端
[root@k8s-master ~/k8s-all]#kubectl -n yuchao delete po eladmin-api
pod "eladmin-api" deleted
[root@k8s-master ~/k8s-all]#
# 创建pod是异步，支持并发处理
# 删除pod是同步操作，保证安全，等待较久
修改新的yaml，引用我们创建的redis、mysql新资源。
apiVersion: v1
kind: Pod
metadata:
name: eladmin-api
namespace: yuchao
labels:
app: eladmin-api
spec:
containers:
- name: eladmin-api
image: 10.0.0.66:5000/eladmin/eladmin-api:v1
env:
- name: DB_HOST # 指定数据库地址，更新为svc
value: "10.110.3.172"
- name: DB_USER # 指定数据库连接使用的用户
value: "root"
- name: DB_PWD
value: "www.yuchaoit.cn"
- name: REDIS_HOST
value: "10.111.94.30"
- name: REDIS_PORT
value: "6379"
ports:
- containerPort: 8000 # 同EXPOSE，声明业务端口号
创建新eladmin后台
kubectl create -f pod-eladmin-api.yml
看pod后台日志
# 看描述
kubectl -n yuchao describe pod eladmin-api
# 看日志
[root@k8s-master ~/k8s-all]#kubectl -n yuchao logs --tail=20 eladmin-api
# 有报错
Caused by: java.sql.SQLSyntaxErrorException: Table 'eladmin.sys_quartz_job' doesn't exist
# 是因为还没导入数据
[root@k8s-master ~/docker-all/eladmin/sql]#kubectl -n yuchao cp eladmin.sql mysql:/tmp/
[root@k8s-master ~/docker-all/eladmin/sql]#kubectl -n yuchao exec -it mysql -- bash
root@mysql:/# mysql -uroot -pwww.yuchaoit.cn eladmin < /tmp/eladmin.sql
mysql: [Warning] Using a password on the command line interface can be insecure.
root@mysql:/#
再次启动后端
# pod有自动重启的策略，重启时，正确连接mysql的数据库，也能确保正确启动。
# 手工重建eladmin-pod
[root@k8s-master ~/k8s-all]#kubectl create -f pod-eladmin-api.yml
pod/eladmin-api created
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
eladmin-api 1/1 Running 0 8s 10.244.2.9 k8s-slave1 <none> <none>
mysql 1/1 Running 0 48m 10.244.0.5 k8s-master <none> <none>
redis 1/1 Running 0 110m 10.244.1.6 k8s-slave2 <none> <none>
[root@k8s-master ~/k8s-all]#
访问测试后端接口，能看到验证码就是正确的了。
[root@k8s-master ~/k8s-all]#curl -s 10.244.2.9:8000/auth/code|jq
{
"uuid": "code-key-0db18cd151684f569610f42d7fc9f81b",
"img": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAG8AAAAkCAIAAAAIOPOYAAALaklEQVR42u2aCVAUZxbHTdxds+vW1la2tnaT2juVytbWprJZxCMBI5pgLGOpMZWAx6rlhRJUoiTiDdHghWhQThGUS8J9ewDKIocgIAMzHMMwwDiMHCPnMAzMzNv38TXdPQfDUJI1W8tXr6juj29mun/zvvf+7/XMgOkxdWPGNIJpmtM0p2lOj2maUz30emhvB5EIqquhuRlGRp6Jpk6lGpRI+h4+7C0sVIlE+uHhyV5PzxDo9M96U4Mjg+0D7WKl+JHiUWFrYX5zPv4VdYp0et33xXFgAFJTwdMTtm/n7PPP4fp16O6eHM3+R49kvr6i9evLbW0f2tiwVmFn1+zlpcGvy4qR0QgfxMKfAmB/nqVlCXWw6xa09kKfpg8ZxVbHXii+cCj30M6MnZ/Ff+YY6Tg/dL5NsI1Zc0l3UY+opx5lTQ14eBhw5NvevVBbaxXNnoICoZMTn6CpPVq8uL+y0sLFqIZhUwbM8OEsQmBmmVINqxKZBa/4w/Lo1eNRMzX7q/YLwxfigcdtjylGifvaxYVQ27ULYmJAKISqKhAICME7d8DdnXFSmWwCmq3nzrHI6rZtexId3VdWNtzRoRsa0mk0uOXlgYEVCxYQoA4OGrnc7MU098BbYQyjxTHwqj85+OlZEHQYLHuogD8GGBC3CZ6LdNYmrj2Ye9C3yDesIixeGH9TfLOgpaBCUVHfVS/vk/cO9Y7omOB1seQiJYuTU4ZSpSKuh7z27SMeGh0Nrq6cV166BHV1zILjx0GrHZdmZ1IS2cv29rLz59USCRuGVXV1feXlWvwYGk8Egop33sGVTYcPm14MIvvNtwyd7dkwrINsCXP6t1BQjwXxa9Xw0lmO+J475OCtIAdEUywrtvLGi2RFlCaCNv6fpheUVSC/Cy2Z0JwG0hSQJptYCnSWG7/w9m1CascOKC42DprUTpyAnBzmODd3XJoNrq7EK21tMdvg6bBSKQ8KqvrwQ2Z3v/8+MqUr20JCCPcFC3Rq45g1J5xwedkP7vP2gXMqA27f6Kd73ef8cX0aIY5paukN+HvgSkTjV5LPhYJBpaBdkNmQiW64PX379arr/M/CvERplshKDC7i7kYIeRGCZ1hj+We+RK8IDYWyMtBhSjt1imDy9yduyBL09gZ0r64u5hR9lh7w/MmYZqWDAwUncnaW+flRB+RblaOjbnAQV6pbWujMAOoGw/Gzc4QR+p34KTf5ZAB+6UfmZ56CHdkcSs973JoSOfw14F+I5r2IbZhbPo77+N2wd03DJWYqZs+Afnf2bjr54PEDg4sImWklSmIhM5tF8vx8OHmSwBlxH00+eXkkMlJkuNOfjt4MSxMjKcYBetzRYZ5m4xdfWE4+aDT56DUaeoqyyehNXhgj9fZVlDXcfHClQYj80Wkyw4+hGB9e8/ewnHk2p2xmXxJQFkAnMRehfjK4iLDZBFOVL4hjoD4CJHHQlEh3t74pOeVCcn54vLYhHsJ/wQBtzWa+0RLQ7HAjjBITuXCJfktHZCQzg1jZIDC6j83QHG5vx+1smeYAfi2j+omeDorF9LX3pPeO5B1ZE7dm1pmnLDKnFJ4KBrCLZObRT281cf+62QQ/9yXzs89Wvx1kZ4FmnpSRWrlNuXOC59DJkPIQ49gX+Sph9MATQn9ilXu2ZHDS0Pu8caAMGX1/Nlb6+RG9yf4Xk9J4OV0tlbL7vWb1akVExFBrK06yEYDUBgDSI0fwVLBsmV6nwzy7IWmDc4JzSm2KtFu6Lk3F98Gj/+YCa52SxNN/XIV6JfeJUTXw49Pc+tlnBf8McWTxLQpfhOGSHqP21JMvBSRPJSiP2EmNVmN8G/FvTmKno8l5erihgaQgPk30zbg45hg3OGrt777j/jsmvc3rTcw8LT4+bMLRa7WSAweobh+sJ6lzSCYrnzsXZxRXr7b2ti6LWhZXE8e+XNrDhE5qL/joMsT9vJIGtLy6KLCCiwysrUlsj6qKQjeU9coQX6IokYJDfySfPjK0KnYVnbELs2vqbjJzDyVfTo5mt4EUVybe1bjsZjI7Rk9MQRTc0aOAorCtjehQOuPhMYnKEmWm2N2dpu/eoiI6KfH0xJnqFSuGB1Wfxn+KqtDoVdFCAzq/8msz++bolaYof3cJZH284KMbXh69HMGh+1PHDC0PZT03rT7N/HVrh+DG69aiRAEABpUvppyvvuKdp6UxKLGqbmqCQ4c4x4yKspampq1NtHYt0UZLlmCgZDRpaipVUX2lpen16ZuSN2l1WtPXut5ifRPeDL7yUP7QdM0r/sYo/xJooARI0SlMoOAqFSRndam62D2Odee4l46ArHdMdGSTqhKFOTdKSxk3xOKHBgFM/7RSevLEKppY/wg++ojEynXrNAoFnexISCifNw8n265cwVO8nyxxlulri+WMgMf6J1YEMdUxX+d/bboM/8VHuSwOugwzM+uY7jfd6Yx3vjdFaRti26hsHPfqJ6mQQGVQ1KWnk84GN1hhhJKT9UpEWVxsVddDp1IJnZ2RWoObG9XnWFZKjx2juejx5ct0GbqJ6S2liZki58+BUDH6zeEaDAhmP+hEIYPyWAGYtploxMTcLVYS5YAFD0KkNH0KfCxtq/EV0oO45PtRySCJJ2aikEhO7yfbnK0EmXHkCCGIpbqXF0MT5b01XQ/EV79jB1IT79lD+28jSmXthg0UJTLtvnuX2uaDNoOYAfUch9tSJkGvTIBuNVexYKZiJfqQYWDYm0PWb8wwvgwsxlfErEBwB+4coDMo6dl+B9ZIlmg+g0IKCoKwMJM3pEkcA2hGBkMTt7xhX84MTf3ISP3OncQrd+1Cic60VNavt6BAaz75ZLizk678/WWC5vUgZMG9JxYqq244RdYwrZA3gqFcYaBDN452m7zvG1xJal0q3dGoh/A0vzmfTT6meW9KFBKWlZhUMCQODZkGr2JGaWLlw+onjKeWaSrCwymjHl6R07hvn2mViVY2Z0zSV1XRlfx+Bx3CTlgaW/vS2UF+iJx1BoIq+W5IinTMV5ljYUOn162+sZp1TEx0WGhSlEsjl07c0JykQlIrarEKwqQdEGAOJemttRJ8bm6QmQnHjjE0U1ImoEkyz2h7uAN9e6Jx8+RuXCkck/Q4PogFfiFkH2Wctfm2JQs0Y7u+sZvMzL/GnOZIcmjEpHE5tjqWdcx4YfzEzSVUSElzLBPUh85SB7389NIfir7Z6O6uR46YyseXijrYv9+4RkpImIAm+mDlokXIqOX06Qmvuek4yUtF186zM74PLOEztXnXQN4PhY9JyqJyqnM0rWM9juC2pm7F496hXocIB4pyZexKtrk50aOPJ5D2HiTPhRp/eJwDfVLQPluLHitImouoYXxta5uAZt2WLUwRuW4dm216CgpUIpHOZA/Ubd2KK/d4LcLky/aJ2a6lWXvxlPEM1pozeZMFMsDyhvXEQ7mHUF2xp9ni7Of5xI2vkNzdJxE3TQ2rSUTcevp0Z1ISoZyXR1skgswYTNluWW5Y+WGhGSEwz/HXF+F4AbT1wzdFlnBnSaBUXmq25eEU76QH/fOkSVufrLm6Ej1lgSbW4BM25YwM3Vaj1aBE35K6ZcGVBXjbr126MMNHzwLCRH+xjDwpYge/xcm3334LT9VEV5qlmdmQ+ZyfBtOOZ24u89QIjfeszbzerHdxsR5l5cKFZp8Jx9eSwhGzyvVq0lo3LrR04BBjpkKv6WSUpmeOpxHKJdeWmOkV/ZfH4cNcWUlNKp2ApkoorLC3twZlhZ1dp6FKsH4o1eQxEYvS8Qa0qwwWFLYWOic4szQvl15+/r9UyMoy2OmenqOPPn4Yv/VQDJCcjlI/XGA+HGKUvNV4C4Wna6Zrv6b/B/G7j7w80qBD4YkoGxv/93458z3+mmNKx/TvkKZpTtP8fxj/AW6gUX6XqRbaAAAAAElFTkSuQmCC"
}
再聊k8s拉取私有镜像
k8s拉取私有镜像，且携带认证的话，需要额外配置
# 0.查看secret类型，选择是docker镜像类型
[root@k8s-master ~/k8s-all]#kubectl -n yuchao create secret -h
Create a secret using specified subcommand.
Available Commands:
docker-registry Create a secret for use with a Docker registry
generic Create a secret from a local file, directory, or literal value
tls Create a TLS secret
Usage:
kubectl create secret [flags] [options]
Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).
[root@k8s-master ~/k8s-all]#
# 选择docker-registry Create a secret for use with a Docker registry
# 1.创建一个秘钥资源，用于让k8s通过该资源，提交账号密码，下载私有镜像
# 并且只限于yuchao这个名称空间
kubectl -n yuchao create secret docker-registry registry-10.0.0.66 --docker-username=admin --docker-password=admin --docker-email=yc_uuu@163.com --docker-server=10.0.0.66:5000
# 查看secret详细
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get secret registry-10.0.0.66 -o yaml
apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyIxMC4wLjAuNjY6NTAwMCI6eyJ1c2VybmFtZSI6ImFkbWluIiwicGFzc3dvcmQiOiJhZG1pbiIsImVtYWlsIjoieWNfdXV1QDE2My5jb20iLCJhdXRoIjoiWVdSdGFXNDZZV1J0YVc0PSJ9fX0=
kind: Secret
metadata:
creationTimestamp: "2023-03-11T16:19:02Z"
name: registry-10.0.0.66
namespace: yuchao
resourceVersion: "245193"
uid: 82890c61-6ed3-4b70-987b-7949064a0d3e
type: kubernetes.io/dockerconfigjson
# 2.pod资源载入该secret，携带认证去下载资源
## 给pod配置上述密钥
apiVersion: v1
kind: Pod
metadata:
name: eladmin-api
namespace: yuchao
labels:
app: eladmin-api
spec:
imagePullSecrets: # 全局认证，后续容器都使用这个secret
- name: registry-10.0.0.66
containers:
- name: eladmin-api
...省略
pod运行多容器(前后端)
这里先不用做，需要学完svc才能实现业务发布。
涉及复杂的前后端，域名，端口问题。
前置环境
镜像从哪里拉取，前端镜像，以及前后端的通信地址。
这里先不用去操作，测试有点问题。
# 1.这里只是做测试，前端用后端的svc
[root@docker01 ~/eladmin-web]#cat .env.production
ENV = 'production'
# 如果使用 Nginx 代理后端接口，那么此处需要改为 '/'，文件查看 Docker 部署篇，Nginx 配置
# 接口地址，注意协议，如果你没有配置 ssl，需要将 https 改为 http
VUE_APP_BASE_API = 'http://10.0.0.80:31343'
# 如果接口是 http 形式， wss 需要改为 ws
VUE_APP_WS_API = 'ws://10.0.0.80:31343'
# 2.镜像
docker build . -t last-eladmin-web:v2 -f Dockerfile
[root@docker01 ~/eladmin-web]#docker tag last-eladmin-web:v2 10.0.0.66:5000/last-eladmin-web:v2
[root@docker01 ~/eladmin-web]#docker push 10.0.0.66:5000/last-eladmin-web:v2
[root@docker01 ~/eladmin-web]# curl -u admin:admin -X GET http://10.0.0.66:5000/v2/_catalog
# 整个过程可能需要删除registry的镜像
curl -u admin:admin -H "Accept: application/vnd.docker.distribution.manifest.v2+json" 10.0.0.66:5000/v2/eladmin-web/manifests/v1
# 拿到digest
sha256:892b16bd68ad1b02a1968d2514767669f1c3f2a436442899cf35a3db287cc794
sha256:892b16bd68ad1b02a1968d2514767669f1c3f2a436442899cf35a3db287cc794
# 删除镜像
curl -X DELETE http://<registry-hostname>/v2/<image-name>/manifests/<digest>
curl -u admin:admin -X DELETE http://10.0.0.66:5000/v2/eladmin-web/manifests/sha256:892b16bd68ad1b02a1968d2514767669f1c3f2a436442899cf35a3db287cc794
#但是默认不支持删除动作，得修改registry配置
[root@docker01 ~/eladmin-web]#docker exec -it registry sh
/ # cd /etc/docker/registry/
/etc/docker/registry # vi config.yml
/etc/docker/registry # cat config.yml
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
delete:
enabled: true
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
/etc/docker/registry #
# 重启registry，再次删除
curl -u admin:admin -X GET http://10.0.0.66:5000/v2/eladmin-web/tags/list
前/后端yaml
apiVersion: v1
kind: Pod
metadata:
name: eladmin-all
namespace: yuchao
labels:
app: eladmin-all
spec:
imagePullSecrets:
- name: registry-10.0.0.66
containers:
- name: eladmin-api
image: 10.0.0.66:5000/eladmin/eladmin-api:v1
env:
- name: DB_HOST # 指定数据库地址，更新为svc
value: "10.110.3.172"
- name: DB_USER # 指定数据库连接使用的用户
value: "root"
- name: DB_PWD
value: "www.yuchaoit.cn"
- name: REDIS_HOST
value: "10.111.94.30"
- name: REDIS_PORT
value: "6379"
ports:
- containerPort: 8000 # 同EXPOSE，声明业务端口号
- name: eladmin-web
image: 10.0.0.66:5000/last-eladmin-web:v2
ports:
- containerPort: 80
创建前后端pod
[root@k8s-master ~/k8s-all]#kubectl create -f all-eladmin.yml
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
eladmin-all 2/2 Running 0 44s 10.244.2.11 k8s-slave1 <none> <none>
eladmin-api 1/1 Running 0 118m 10.244.2.9 k8s-slave1 <none> <none>
mysql 1/1 Running 0 166m 10.244.0.5 k8s-master <none> <none>
redis 1/1 Running 0 3h48m 10.244.1.6 k8s-slave2 <none> <none>
[root@k8s-master ~/k8s-all]#
前端svc
这一步就等于k8s业务上线，提供访问了
apiVersion: v1
kind: Service
metadata:
name: eladmin-all
namespace: yuchao
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: eladmin-all
type: NodePort
创建
[root@k8s-master ~/k8s-all]#kubectl create -f svc-all-eladmin.yml
service/eladmin-all created
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get svc -owide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
eladmin-all NodePort 10.107.66.168 <none> 80:31428/TCP 6s app=eladmin-all
mysql ClusterIP 10.110.3.172 <none> 3306/TCP 173m app=mysql
redis ClusterIP 10.111.94.30 <none> 6379/TCP 3h52m app=redis
[root@k8s-master ~/k8s-all]#
#访问集群任意一个31428端口即可
[root@k8s-master ~/k8s-all]#curl 10.0.0.80:31428 -I
HTTP/1.1 200 OK
Server: nginx/1.21.5
Date: Sun, 12 Mar 2023 21:23:09 GMT
Content-Type: text/html
Content-Length: 4445
Last-Modified: Sun, 12 Mar 2023 12:58:51 GMT
Connection: keep-alive
ETag: "640dcc8b-115d"
Accept-Ranges: bytes
后端svc
apiVersion: v1
kind: Service
metadata:
name: eladmin-api
namespace: yuchao
spec:
ports:
- port: 8000
protocol: TCP
targetPort: 8000
selector:
app: eladmin-all
type: NodePort
svc检查
[root@k8s-master ~/k8s-all]#kubectl -n yuchao get svc -owide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
eladmin-api NodePort 10.104.12.218 <none> 8000:31012/TCP 5s app=eladmin-api
mysql ClusterIP 10.110.3.172 <none> 3306/TCP 3h56m app=mysql
redis ClusterIP 10.111.94.30 <none> 6379/TCP 4h55m app=redis
[root@k8s-master ~/k8s-all]#
[root@k8s-master ~/k8s-all]#curl 10.0.0.80:31012
"Backend service started successfully;"
最终结果
测试部署ok
小白目前不需要操作这部分的练习。