XTY Blog | Linux Ops Docs | SRE

kubernetes 1.2.1 kubernetes集群安装(centos7裸机)

27 Jul 2017

0. 背景介绍

1) 参照文档

教程参照文档-[Creating a Custom Cluster from Scratch]
网络参考-理解Kubernetes网络之Flannel网络

不参照[CentOS]和[Using kubeadm to Create a Cluster]的原因是，前者已经废弃，后者在beta阶段。另外此文档只是一个大纲，这样能够更深入的了解kubernetes的组件和原理。

文档中有很多细节，实际操作之外的步骤大部分忽略掉了，推荐详读一遍文档。

2) 软件版本

items	version	comment
OS	centos7
kubernetes	1.9.1	最新稳定版本
docker	17.09.0-ce
etcd	3.0.7
flannel		使用flannel做overlay网络，支持不同主机间pods间网络互通

docker（或者rkt）是必备的，因为kubernetes本身就是一个容器的编排工具

etcd给kubernetes和flannel提供数据存储支持，可部署在kubernetes master节点上，也可以单独启用一个集群

flannel给kubernetes提供了overlay网络支持（可选，也有其他选择，详细见文章开头的文档链接中的描述），实现了不同主机pods之间的直接互通

kubernetes包含以下组件

在master节点上运行的kube-apiserver,kube-controller-manager,kube-scheduler

在node节点上运行的kubelet,kube-proxy

3) 节点规划

hostname	ip address	service	comment
master	172.16.1.100	etcd,kube-apiserver,kube-controller-manager,kube-scheduler,docker	主节点
node01	172.16.1.101	flannel,docker,kubelet,kube-proxy	node 1
node02	172.16.1.102	flannel,docker,kubelet,kube-proxy	node 2
node03	172.16.1.103	flannel,docker,kubelet,kube-proxy	node 3

1. 主机环境

为了将系统环境和软件环境对安装的影响度降低，需要确保以下几项需求满足

安装必要的工具包
yum install -y wget vim iptables iptables-services
关闭selinux
```
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
setenforce 0
```
- 关闭iptables-services和firewalld
  systemctl stop firewalld;systemctl stop iptables
  > 防火墙后期需要开启，并开放api服务的端口
- 设定hostname到hosts文件中 bash echo "172.16.1.100 master 172.16.1.101 node01 172.16.1.102 node02 172.16.1.103 node03" >> /etc/hosts
设定sysctl中的net.ipv4.ip_forward = 1
```
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p
```
net.ipv4.ip_forward = 1的配置确保了可以通过映射docker容器端口到外网，否则我们无法通过外网ip访问容器
- 关闭系统swap
  bash swapoff -a

注释swap的开机挂载项，修改/etc/fstab

#/dev/mapper/VolGroup00-LogVol01 swap                    swap    defaults        0 0

关闭系统swap，是为了严格的按照cpu和内存的限制，这样scheduler在规划pod的时候就不会把pod放进swap中了，这是为了性能考虑。

2. kubernetes master节点

1) 配置kubernetes环境变量（master节点）

echo 'export MASTER_IP=172.16.1.100
export SERVICE_CLUSTER_IP_RANGE=10.254.0.0/16
export CLUSTER_NAME=KubeTest
export PATH=$PATH:/usr/local/kubernetes/bin' > /etc/profile.d/kubernetes.sh
source /etc/profile.d/kubernetes.sh

规划集群中需要重复使用的内容为变量

MASTER_IP - master的静态ip

SERVICE_CLUSTER_IP_RANGE - service对象使用的ip范围

CLUSTER_NAME - kubernetes集群的名称

2) 获取kubernetes（master节点）

kubernetes的二进制包里面包含了kubernetes的二进制文件和支持的etcd版本

# 下载kubernetes
wget https://dl.k8s.io/v1.9.1/kubernetes-server-linux-amd64.tar.gz
tar zxvf kubernetes-server-linux-amd64.tar.gz

# 拷贝二进制文件到server端
mkdir -p /usr/local/kubernetes/{bin,security,conf}
cp kubernetes/server/bin/{kube-apiserver,kube-scheduler,kube-controller-manager,kubectl} /usr/local/kubernetes/bin/
chmod 750 /usr/local/kubernetes/bin/*
# 如果使用docker启动kube-apiserver,kube-scheduler,kube-controller-manager这三个服务的话，不需要拷贝它们的二进制文件，只需要拷贝kubectl即可

# 拷贝二进制文件到node端(提前做好ssh信任)
scp kubernetes/server/bin/{kubelet,kube-proxy} root@node01:/usr/local/bin
scp kubernetes/server/bin/{kubelet,kube-proxy} root@node02:/usr/local/bin
scp kubernetes/server/bin/{kubelet,kube-proxy} root@node03:/usr/local/bin

因为kubernetes这个项目是使用go语言编写，而go语言程序的部署方式很简单，就是拷贝二进制文件就可以，所以在这里，我们通过简单的复制各服务的二进制文件，就可以通过启动它们来启动相应的服务。

本文开头的参照文档中说:
node需要运行的kubelet,kube-proxy,docker，推荐直接在系统层面上启动服务;
而对于etcd, kube-apiserver, kube-controller-manager 和 kube-scheduler，推荐我们使用容器来运行它们，文档中给出了几种镜像的获取方式，当然，我们下载的二进制文件中也有这样的镜像文件（bin目录中tar结尾的文件）可以本地加载（使用docker load命令）镜像到本机的docker中。

3). 配置和安装kubernetes master服务

1) 部署etcd

在kubernetes/cluster/images/etcd/Makefile中查找到对应的etcd版本

etcd 单点的安装可以参照etcd install single node with systemd

使用etcd储存flannel的网络配置

etcdctl --endpoints http://$MASTER_IP:2379 set /kube-centos/network/config '{ "Network": "10.5.0.0/16", "Backend": {"Type": "vxlan"}}'

为了测试，在主节点上只启动一个节点的etcd，etcd集群参照etcd 集群文档

2) 启动kubernets Apiserver, Controller Manager, 和 Scheduler服务

准备配置文件：

config, 通用配置
apiserver, kube-apiserver配置
controller-manager, kube-controller-manager配置
scheduler, kube-scheduler配置

cat > /usr/local/kubernetes/conf/config << EOF
###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"


# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://127.0.0.1:8080"
EOF

cat > /usr/local/kubernetes/conf/apiserver << EOF
###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

# The port on the local server to listen on.
KUBE_API_PORT="--insecure-port=8080"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379,http://127.0.0.1:4001"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=$SERVICE_CLUSTER_IP_RANGE"

# default admission control policies
# KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"
KUBE_ADMISSION_CONTROL=""

# Add your own!
KUBE_API_ARGS="--service-node-port-range=1-65535"
EOF

cat > /usr/local/kubernetes/conf/controller-manager << EOF
###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS=""
EOF

cat > /usr/local/kubernetes/conf/scheduler << EOF
###
# kubernetes scheduler config

# default config should be adequate

# Add your own!
KUBE_SCHEDULER_ARGS=""
EOF

错误: No API token found for service account “default”, retry after the token，解决办法是配置KUBE_ADMISSION_CONTROL=""禁用KUBE_ADMISSION_CONTROL

准备systemd unit文件:

kube-apiserver.service
kube-controller-manager.service
kube-scheduler.service

echo '[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/usr/local/kubernetes/conf/config
EnvironmentFile=-/usr/local/kubernetes/conf/apiserver
User=kube
ExecStart=/usr/local/kubernetes/bin/kube-apiserver \
	    $KUBE_LOGTOSTDERR \
	    $KUBE_LOG_LEVEL \
	    $KUBE_ETCD_SERVERS \
	    $KUBE_API_ADDRESS \
	    $KUBE_API_PORT \
	    $KUBELET_PORT \
	    $KUBE_ALLOW_PRIV \
	    $KUBE_SERVICE_ADDRESSES \
	    $KUBE_ADMISSION_CONTROL \
	    $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/kube-apiserver.service

mkdir /usr/lib/systemd/system/kube-apiserver.service.d
echo '[Service]
PermissionsStartOnly=yes
ExecStartPre=/usr/bin/mkdir -p /var/run/kubernetes
ExecStartPre=/usr/bin/chown kube.kube /var/run/kubernetes' > /usr/lib/systemd/system/kube-apiserver.service.d/pre-start.conf

echo '[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/usr/local/kubernetes/conf/config
EnvironmentFile=-/usr/local/kubernetes/conf/controller-manager
User=kube
ExecStart=/usr/local/kubernetes/bin/kube-controller-manager \
	    $KUBE_LOGTOSTDERR \
	    $KUBE_LOG_LEVEL \
	    $KUBE_MASTER \
	    $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/kube-controller-manager.service


echo '[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/usr/local/kubernetes/conf/config
EnvironmentFile=-/usr/local/kubernetes/conf/scheduler
User=kube
ExecStart=/usr/local/kubernetes/bin/kube-scheduler \
	    $KUBE_LOGTOSTDERR \
	    $KUBE_LOG_LEVEL \
	    $KUBE_MASTER \
	    $KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/kube-scheduler.service

依次启动kube-apiserver.service, kube-controller-manager.service, kube-scheduler.service

# 重载systemd unit文件
systemctl daemon-reload

# 创建spawn服务的用户kube（在配置文件中配置）
useradd -r -s /sbin/nologin kube
chown :kube /usr/local/kubernetes/bin/*

systemctl enable kube-apiserver.service
systemctl enable kube-controller-manager.service
systemctl enable kube-scheduler.service
systemctl start kube-apiserver.service
systemctl start kube-controller-manager.service
systemctl start kube-scheduler.service

3. node节点配置和安装基本软件

1) 部署flannel(node节点)

# 下载flannel
FLANNEL_VER=v0.9.1
wget https://github.com/coreos/flannel/releases/download/v0.9.1/flannel-${FLANNEL_VER}-linux-amd64.tar.gz
mkdir flannel
tar zxvf flannel-${FLANNEL_VER}-linux-amd64.tar.gz -C flannel
cp flannel/flanneld /usr/local/bin
mkdir -p /usr/libexec/flannel
cp flannel/mk-docker-opts.sh /usr/libexec/flannel/

# 准备flannel配置文件
## !!重点!! ##
# -iface，根据实际情况设定
# FLANNELD_PUBLIC_IP，每个节点不同
#############
cat > /etc/sysconfig/flanneld << EOF
FLANNELD_PUBLIC_IP="172.16.1.101"
FLANNELD_ETCD_ENDPOINTS="http://172.16.1.100:2379"
FLANNELD_ETCD_PREFIX="/kube-centos/network"
# Any additional options that you want to pass
FLANNELD_OPTIONS="-iface=eth1"
EOF

# 准备flannel systemd unit文件
echo '[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
ExecStart=/usr/local/bin/flanneld $FLANNELD_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -c
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service' > /usr/lib/systemd/system/flannel.service

systemctl daemon-reload
systemctl enable flannel
systemctl start flannel

每个节点的flannel需要根据自己情况来填写配置文件

flannel启动后生成了以下文件：

/var/run/flannel/subnet.env, 从etcd中获取信息然后生成的flanneld配置文件

/run/docker_opts.env, flannel service文件中指定的/usr/libexec/flannel/mk-docker-opts.sh生成的docker环境变量文件

2) 安装docker(node节点)

# 安装docker底包
yum install -y git libcgroup libcgroup-tools
systemctl enable cgconfig
systemctl start cgconfig

# 下载安装docker
DOCKER_VER=17.09.0
wget https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_VER}-ce.tgz
tar zxvf docker-${DOCKER_VER}-ce.tgz
cp docker/* /usr/local/bin/
wget https://github.com/docker/compose/releases/download/1.17.1/docker-compose-Linux-x86_64
cp docker-compose-Linux-x86_64 /usr/local/bin/docker-compose
chmod 755 /usr/local/bin/*

# 准备systemd unit文件
echo '[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket flannel.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/docker_opts.env
ExecStart=/usr/local/bin/dockerd -H fd:// $DOCKER_OPTS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/docker.service


echo '[Unit]
Description=Docker Socket for the API
PartOf=docker.service

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target' > /usr/lib/systemd/system/docker.socket

groupadd docker

systemctl daemon-reload
systemctl enable docker
systemctl start docker

docker systemd

3) 安装kubelet(node节点)