kind+docker+k8s+nginx+s3fs

韦志新

2023-12-01

使用kind部署实验用k8s集群，部署nginx服务挂载s3目录实例

需求

需要在内网访问S3目录，且S3不能开启公网访问。

解决方案

1.使用VPC Endpoint服务，内网与VPC打通，实现通过VPC Endpoint访问S3。
遇到的问题：可以访问单个文件，但只支持以xml格式列出目录，不支持html格式。
2.部署nginx服务，使用s3fs挂载S3目录，再以nginx_autoindex自动生成挂载目录索引并提供访问。
本案例选择方案2

概要

1.使用kind搭建k8s集群
2.在k8s集群中部署nginx服务，并配置S3挂载和访问
3.其他

使用kind搭建k8s集群

kind为实验用微型k8s集群部署工具，他可以使用docker生成一个容器做为k8s集群的node。

yum安装docker

#卸载旧版本(如果安装过旧版本的话)
yum list installed | grep docker
yum remove docker docker-common docker-selinux docker-engine

#yum-util 提供yum-config-manager功能，另外两个是devicemapper驱动依赖的
yum install -y yum-utils device-mapper-persistent-data lvm2

#设置yum源
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

#安装
yum install -y docker-ce

#启动服务
systemctl start docker

#开机启动
systemctl enable docker

#验证版本
docker version
Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:51:00 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:48:42 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

#在daemon.json中配置docker虚拟网段，防止IP冲突
[userlocal@localhost ~]$ cat /etc/docker/daemon.json
{
 "bip":"192.168.1.1/24"
}

安装kubectl

#下载安装包
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

#安装
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

#验证版本
/usr/local/bin/kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.2", GitCommit:"fc04e732bb3e7198d2fa44efa5457c7c6f8c0f5b", GitTreeState:"clean", BuildDate:"2023-02-22T13:39:03Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-25T19:35:11Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

Kind安装与使用

#下载执行软件
wget -c https://github.com/kubernetes-sigs/kind/releases/download/v0.17.0/kind-linux-amd64

#添加执行权限
chmod +x kind-linux-amd64

#copy至/usr/local/bin目录下
cp kind-linux-amd64 /usr/local/bin/kind

#验证版本
kind version
kind v0.17.0 go1.19.2 linux/amd64

#基本操作
kind -h	#查看命令帮助

创建集群

#查看命令帮助
kind create -h

#简单创建一个名为test的集群
[root@localhost userlocal]# kind delete cluster -n kind
Deleting cluster "kind" ...
[root@localhost userlocal]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[root@localhost userlocal]# kind create cluster -n test
Creating cluster "test" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 
 ✓ Preparing nodes 
 ✓ Writing configuration 
 ✓ Starting control-plane ️
 ✓ Installing CNI 
 ✓ Installing StorageClass 
Set kubectl context to "kind-test"
You can now use your cluster with:

kubectl cluster-info --context kind-test

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 

#kind成功部署集群后，可以看到一个dokcer容器被用作k8s集群的node。
[root@localhost userlocal]# docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS              PORTS                       NAMES
8e88589a1e16   kindest/node:v1.25.3   "/usr/local/bin/entr…"   About a minute ago   Up About a minute   127.0.0.1:34595->6443/tcp   test-control-plane

#使用kubctl来管理集群，到此为止k8s集群已经成功创建
[root@localhost userlocal]# kubectl get nodes
NAME                 STATUS   ROLES           AGE     VERSION
test-control-plane   Ready    control-plane   4m44s   v1.25.3

在k8s集群中部署nginx服务

构建镜像并上传至集群

#查找docker官方镜像
docker search nginx

#下载镜像到本地
docker pull nginx

#查看本地镜像
[root@localhost userlocal]# docker images
REPOSITORY     TAG        IMAGE ID       CREATED        SIZE
nginx          latest     904b8cb13b93   4 weeks ago    142MB
kindest/node   <none>     d8644f660df0   5 months ago   898MB

#编辑docker_file
cat > ./dockerfile <<EOF
FROM nginx
## Some utilities
RUN apt-get update -y && \
    apt-get install -y build-essential libfuse-dev libcurl4-openssl-dev libxml2-dev pkg-config libssl-dev mime-support automake libtool wget tar git unzip
RUN apt-get install lsb-release -y  && apt-get install zip -y && apt-get install vim -y

## Install S3 Fuse
RUN rm -rf /usr/src/s3fs-fuse
RUN git clone https://github.com/s3fs-fuse/s3fs-fuse/ /usr/src/s3fs-fuse
WORKDIR /usr/src/s3fs-fuse 
RUN ./autogen.sh && ./configure && make && make install

## Create folder
WORKDIR /var/www
RUN mkdir s3

## Set Your AWS Access credentials
ENV AWS_ACCESS_KEY=YOURAWSACCESSKEY
ENV AWS_SECRET_ACCESS_KEY=YOURAWSSECRETACCESSKEY
ENV S3_BUCKET_REGION=YOURS3BUCKETREGION

## Set the directory where you want to mount your s3 bucket
ENV S3_MOUNT_DIRECTORY=/var/www/s3

## Replace with your s3 bucket name
ENV S3_BUCKET_NAME=your-s3-bucket-name

## S3fs-fuse credential config
RUN echo $AWS_ACCESS_KEY:$AWS_SECRET_ACCESS_KEY > /root/.passwd-s3fs && \
chmod 600 /root/.passwd-s3fs

## change workdir to /
WORKDIR /

## Entry Point
ADD start-script.sh /start-script.sh
RUN chmod 755 /start-script.sh 
CMD ["/start-script.sh"]
EOF

#编辑启动脚本start-script.sh
cat > ./start-script.sh <<EOF
#!/bin/bash

set -x

## Create folder
rm -rf $S3_MOUNT_DIRECTORY
mkdir $S3_MOUNT_DIRECTORY

## S3fs-fuse credential config
echo $AWS_ACCESS_KEY:$AWS_SECRET_ACCESS_KEY > /root/.passwd-s3fs && \
chmod 600 /root/.passwd-s3fs
s3fs $S3_BUCKET_NAME $S3_MOUNT_DIRECTORY -o passwd_file=/root/.passwd-s3fs -o endpoint=$S3_BUCKET_REGION

## Start nginx
nginx -g "daemon off;"
EOF

#使用dockerfile生成镜像(注意尾部的.指当前命令执行路径，该参数为必要项)
docker build -f dockerfile -t nginx_s3fs:test .

#查看build的镜像
[root@localhost userlocal]# docker images
REPOSITORY     TAG        IMAGE ID       CREATED        SIZE
nginx-s3fs     test       fe056762cfbe   2 weeks ago    722MB
nginx          latest     904b8cb13b93   4 weeks ago    142MB
kindest/node   <none>     d8644f660df0   5 months ago   898MB

#进入到kind节点（容器）查看当前集群镜像
[root@localhost userlocal]# docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                       NAMES
8e88589a1e16   kindest/node:v1.25.3   "/usr/local/bin/entr…"   27 minutes ago   Up 26 minutes   127.0.0.1:34595->6443/tcp   test-control-plane
[root@localhost userlocal]# docker exec -it 8e88589a1e16 /bin/bash
root@test-control-plane:/# ctr -n k8s.io i ls -q|grep nginx
root@test-control-plane:/# exit
exit

#导入镜像到集群test,node test-control-plane.
[root@docker ~]# kind load docker-image --nodes test-control-plane nginx-s3fs:test -n test
[root@localhost userlocal]# docker exec -it 8e88589a1e16 /bin/bash
root@test-control-plane:/# ctr -n k8s.io i ls -q|grep nginx
docker.io/library/nginx-s3fs:test

#验证
[root@localhost userlocal]# docker exec -it 8e88589a1e16 /bin/bash
root@test-control-plane:/# ctr -n k8s.io i ls -q|grep nginx
docker.io/library/nginx-s3fs:test

部署服务

#编写deployment.yml
cat > ./nginxs3fs_deploy.yml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  name: nginx-s3fs
  labels:
	run: nginx-s3fs
spec:
  replicas: 1
  selector:
	matchLabels:
	  app: nginx-s3fs
  template:
	metadata:
	  labels:
		app: nginx-s3fs
	spec:
	  containers:
	  - image: docker.io/library/nginx-s3fs:test
		imagePullPolicy: IfNotPresent
		name: nginx
		resources: {}
		securityContext:
		  privileged: true
		ports:
		- name: http
		  containerPort: 80
		  protocol: TCP
		env:
		- name: AWS_ACCESS_KEY
		  value: "YourAWSAccessKey"
		- name: AWS_SECRET_ACCESS_KEY
		  value: "YourAWSSecretAccessKey"
		- name: S3_MOUNT_DIRECTORY
		  value: "/var/s3"
		- name: S3_BUCKET_NAME
		  value: "YourBucketName"
		- name: S3_BUCKET_REGION
		  value: "ap-northeast-1"
	  dnsPolicy: ClusterFirst
	  restartPolicy: Always
EOF			

#编写service.yml
cat > ./nginxs3fs_service.yml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: nginx-s3fs
  labels:
	app: nginx-s3fs
spec:
  type: NodePort
  ports:
	- port: 80 
	  targetPort: 80
	  nodePort: 30080
  selector:
	app: nginx-s3fs
EOF

#部署应用和服务
kubectl apply -f nginxs3fs_deployment.yml
kubectl apply -f nginxs3fs_service.yml

#验证服务
[root@localhost ~]# kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
nginx-s3fs-8c8c9bd6b-drpsk   1/1     Running   0          27m

#POD端口转发至本地虚拟机端口（因集群node为本地dokcer容器，无法直接访问，需做端口转发将service的80端口转发至本地local的30080上来验证服务，正常生产只需访问service即可）
kubectl port-forward --address 172.24.4.219 service/nginx-s3fs-***** 30080:80

nginx配置文件

cat > ./nginx.conf <<EOF
nginx.conf
user  root;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
	worker_connections  1024;
}


http {
	include       /etc/nginx/mime.types;
	default_type  application/octet-stream;

	log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
					  '$status $body_bytes_sent "$http_referer" '
					  '"$http_user_agent" "$http_x_forwarded_for"';

	access_log  /var/log/nginx/access.log  main;

	sendfile        on;
	#tcp_nopush     on;

	keepalive_timeout  65;

	#gzip  on;

	#include /etc/nginx/conf.d/*.conf;
	server {
		listen       80;
		server_name  localhost;
		charset utf-8,gbk;

		#access_log  /var/log/nginx/host.access.log  main;

		location / {
				#root   /usr/share/nginx/html;
				#index  index.html index.htm;
				root /var/s3;
				autoindex on;
				autoindex_exact_size off;
				autoindex_localtime on;
		}
		error_page   500 502 503 504  /50x.html;
		#location = /50x.html {
		#        root   /usr/share/nginx/html;
		#}
	}
}
EOF

其他

在部署过程中可能会出现一些不可预测的问题，下面简要举例一些碰到的问题：
1.docker容器无法启动，可能与系统防护墙有关。
2.s3fs无法挂载，需要授予容器特权，即securityContext: privileged: true。
3.服务端口不通，可能因node为本地docker容器所致，需要做端口转发。

参考链接

s3fs挂载：
https://aws.amazon.com/cn/blogs/china/s3fs-amazon-ec2-linux/
kind部署k8s集群：
https://blog.csdn.net/weixin_39246554/article/details/128201011
k8s镜像构建：
https://blog.csdn.net/zuozewei/article/details/118313537?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522167894579116782425149123%2522%252C%2522scm%2522%253A%252220140713.130102334…%2522%257D&request_id=167894579116782425149123&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2_allsobaiduend~default-1-118313537-null-null.142^v73insert_down4,201^v4add_ask,239^v2insert_chatgpt&utm_term=%E6%9E%84%E5%BB%BA%E9%95%9C%E5%83%8F&spm=1018.2226.3001.4187
k8s部署服务：
https://blog.csdn.net/m0_73257876/article/details/126901630?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522167903513916800215015314%2522%252C%2522scm%2522%253A%252220140713.130102334…%2522%257D&request_id=167903513916800215015314&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2_allsobaiduend~default-1-126901630-null-null.142^v74insert_down3,201^v4add_ask,239^v2insert_chatgpt&utm_term=k8s%E5%88%9B%E5%BB%BApod&spm=1018.2226.3001.4187

kind+docker+k8s+nginx+s3fs

需求

解决方案

概要

使用kind搭建k8s集群

在k8s集群中部署nginx服务

其他

参考链接

相关阅读

相关文章

相关问答

相关文档