docker k8s

高可用k3s 安装

Posted by shenfh on November 14, 2020

准备一台外部mysql 数据库

准备机器

  • 每台机器的机器名称必须不一样
192.168.3.225  k3s-master-0
192.168.3.229  k3s-master-0
192.168.3.232  k3s-worker-0
  • ubuntu18 修改机器名称

vim /etc/cloud/cloud.cfg


preserve_hostname: true # 修改为true
  • 添加本地dns解析
sudo vim /etc/systemd/resolved.conf

DNS=192.168.3.20

## 重启服务
systemctl restart systemd-resolved.service

配置负载均衡器

负载均衡器的地址为k8s.dafen.com.负载均衡器需要负载到2台master 节点上。在这里我通过内网的dns 解析服务器接到2台master 节点上

192.168.3.225    k8s.dafen.com
192.168.3.229    k8s.dafen.com

安装k3s master 节点

安装master点解

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - server \
  --datastore-endpoint="mysql://root:root@tcp(192.168.3.20:3306)/k3s"

安装worker节点

  • 查看token cat /var/lib/rancher/k3s/server/node-token
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_URL=https://k8s.dafen.com:6443 K3S_TOKEN=K10303ccb557a9659fc6a6099d4c34c161a5ee204200ba8e12ca71f094045990eee::server:0e42f2e3e1cc69631203df2bf116cce1 sh -
  • 添加worker角色
kubectl label node ${node} node-role.kubernetes.io/worker=worker
  • 查看worker 的server节点配置
cat /var/lib/rancher/k3s/agent/etc/k3s-agent-load-balancer.json
{
  "ServerURL": "https://k3s-t2.niusmallnan.club:6443",
  "ServerAddresses": [
    "172.31.4.119:6443",
    "172.31.7.200:6443"
  ]
}

常用的命令

k3s重启

sudo systemctl restart k3s

sudo systemctl restart k3s-agent

查看所有的节点

kubectl get nodes

查看所有的pod

kubectl get pods --all-namespaces

删除节点

kubectl delete pod,service test-admin-ui

查看节点信息

kubectl describe pod test-admin-ui

删除部署

kubectl create deployment test-admin-ui --image=register.vafen.cn:3880/release/dafen-admin-ui:1.3.0.23


kubectl get deployment
kubectl delete deployment test-admin-ui

列出集群中的镜像

k3s crictl images

列出当前集群的容器

k3s crictl ps -a
k3s crictl ps

查看pod占用的cpu和资源

k3s crictl stats

k3s 部署支持私有tls Harbor镜像

生成ssl证书

#!/bin/bash -e


# * 为必改项
# * 服务器FQDN或颁发者名(更换为你自己的域名)，没有就写localhost
CN='harbor.xxxx.com'


# 扩展信任IP或域名


## 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，用逗号隔开。
SSL_IP=''
SSL_DNS='*.xxxx.com'


# 国家名(2个字母的代号)
C=CN


# 证书加密位数
SSL_SIZE=2048


# 证书有效期
DATE=${DATE:-3650}


# 配置文件
SSL_CONFIG='openssl.cnf'


if [[ -z $SILENT ]]; then
echo "----------------------------"
echo "| SSL Cert Generator |"
echo "----------------------------"
echo
fi


export CA_KEY=${CA_KEY-"cakey.pem"}
export CA_CERT=${CA_CERT-"cacerts.pem"}
export CA_SUBJECT=ca-$CN
export CA_EXPIRE=${DATE}


export SSL_CONFIG=${SSL_CONFIG}
export SSL_KEY=$CN.key
export SSL_CSR=$CN.csr
export SSL_CERT=$CN.crt
export SSL_EXPIRE=${DATE}


export SSL_SUBJECT=${CN}
export SSL_DNS=${SSL_DNS}
export SSL_IP=${SSL_IP}


export K8S_SECRET_COMBINE_CA=${K8S_SECRET_COMBINE_CA:-'true'}


[[ -z $SILENT ]] && echo "--> Certificate Authority"


if [[ -e ./${CA_KEY} ]]; then
    [[ -z $SILENT ]] && echo "====> Using existing CA Key ${CA_KEY}"
else
    [[ -z $SILENT ]] && echo "====> Generating new CA key ${CA_KEY}"
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE} > /dev/null
fi


if [[ -e ./${CA_CERT} ]]; then
    [[ -z $SILENT ]] && echo "====> Using existing CA Certificate ${CA_CERT}"
else
    [[ -z $SILENT ]] && echo "====> Generating new CA Certificate ${CA_CERT}"
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_EXPIRE} -out ${CA_CERT} -subj "/CN=${CA_SUBJECT}" > /dev/null || exit 1
fi


echo "====> Generating new config file ${SSL_CONFIG}"
cat > ${SSL_CONFIG} <<EOM
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOM


if [[ -n ${SSL_DNS} || -n ${SSL_IP} ]]; then
    cat >> ${SSL_CONFIG} <<EOM
subjectAltName = @alt_names
[alt_names]
EOM
    IFS=","
    dns=(${SSL_DNS})
    dns+=(${SSL_SUBJECT})
    for i in "${!dns[@]}"; do
      echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}
    done


    if [[ -n ${SSL_IP} ]]; then
        ip=(${SSL_IP})
        for i in "${!ip[@]}"; do
          echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
        done
    fi
fi


[[ -z $SILENT ]] && echo "====> Generating new SSL KEY ${SSL_KEY}"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE} > /dev/null || exit 1


[[ -z $SILENT ]] && echo "====> Generating new SSL CSR ${SSL_CSR}"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/CN=${SSL_SUBJECT}" -config ${SSL_CONFIG} > /dev/null || exit 1


[[ -z $SILENT ]] && echo "====> Generating new SSL CERT ${SSL_CERT}"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
    -days ${SSL_EXPIRE} -extensions v3_req -extfile ${SSL_CONFIG} > /dev/null || exit 1


if [[ -z $SILENT ]]; then
echo "====> Complete"
echo "keys can be found in volume mapped to $(pwd)"
echo
echo "====> Output results as YAML"
echo "---"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/  /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/  /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/  /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/  /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo
fi


if [[ -n $K8S_SECRET_NAME ]]; then


  if [[ -n $K8S_SECRET_COMBINE_CA ]]; then
    [[ -z $SILENT ]] && echo "====> Adding CA to Cert file"
    cat ${CA_CERT} >> ${SSL_CERT}
  fi


  [[ -z $SILENT ]] && echo "====> Creating Kubernetes secret: $K8S_SECRET_NAME"
  kubectl delete secret $K8S_SECRET_NAME --ignore-not-found


  if [[ -n $K8S_SECRET_SEPARATE_CA ]]; then
    kubectl create secret generic \
    $K8S_SECRET_NAME \
    --from-file="tls.crt=${SSL_CERT}" \
    --from-file="tls.key=${SSL_KEY}" \
    --from-file="ca.crt=${CA_CERT}"
  else
    kubectl create secret tls \
    $K8S_SECRET_NAME \
    --cert=${SSL_CERT} \
    --key=${SSL_KEY}
  fi


  if [[ -n $K8S_SECRET_LABELS ]]; then
    [[ -z $SILENT ]] && echo "====> Labeling Kubernetes secret"
    IFS=$' \n\t' # We have to reset IFS or label secret will misbehave on some systems
    kubectl label secret \
      $K8S_SECRET_NAME \
      $K8S_SECRET_LABELS
  fi
fi


echo "4. 重命名服务证书"
mv ${CN}.key tls.key
mv ${CN}.crt tls.crt

把最后生成的tls.key tls.crt文件配置到Harbor作为https证书

注意

一定要配置支持SAN 的证书。就是配置脚本里面的下面内容不能缺少

SSL_DNS='*.xxxx.com'

k3s配置访问私有镜像

编辑文件/etc/rancher/k3s/registries.yaml 配置内容如下

mirrors:
  docker.io:
    endpoint:
      - "https://aaaaa.mirror.aliyuncs.com"
  aly-s:
    endpoint:
      - "https://registry.cn-shenzhen.aliyuncs.com"
  xxx-harbor:
    endpoint:
      - "https://harbor.xxx.com"
configs:
  "harbor.xxx.com":
    auth:
      username: admin
      password: Harbor12345
    tls:
      cert_file: /etc/rancher/k3s/ssl/tls.crt
      key_file: /etc/rancher/k3s/ssl/tls.key
      ca_file: /etc/rancher/k3s/ssl/cacerts.pem

  "registry.cn-shenzhen.aliyuncs.com":
    auth:
      username: xxx
      password: xxxx

重启k3s

## 重启k3s
systemctl restart k3s

## 查看配置结果
cd /var/lib/rancher/k3s/agent/etc/containerd

测试镜像是否可以正常拉取

crictl pull  harbor.xxx.com/test/xxxx:1.0

docker配置访问私有镜像

cat /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://xxx.mirror.aliyuncs.com"],
  "insecure-registries" : [
    "https://harbor.xxx.com" ## 配置私有镜像地址
  ]
}

## 登陆镜像地址
sudo docker login --username=admin https://harbor.xxx.com

### 拉取镜像
docker pull harbor.xxx.com/test/xxxx:1.0

nsf 部署

  • docker-compose 部署脚本
version: '2'
networks:
  default:
    external:
      name: little-bees-soft
services:
  dafen-nfs:
    environment:
      SHARED_DIRECTORY: /nfsshare
    container_name: dafen-nfs
    mem_limit: 500m
    privileged: true
    restart: always
    image: itsthenetwork/nfs-server-alpine:12
    ports:
      - 2049:2049
    volumes:
      - ./data/nfs:/nfsshare
  • 主机安装nfs工具和挂载. k3s环境下如果要挂载nfs数据卷，每个主机是都需要安装nfs-common
apt-get install nfs-common

sudo mount -v -t nfs -o vers=4,port=2049 192.168.3.185:/ /data/test

参考文档

目录
    分类
    Other linux iOS 回顾 Git sql docker k8s