Kubespray 离线安装配置

Kubespray Offline

Posted by BlueFat on Tuesday, May 11, 2021

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/offline-environment.md

背景

在国内使用 kubespray 安装 Kubernetes 集群,下载依赖的文件和镜像时,往往会遇到下载失败,这时我们可以利用 kubespray 离线安装配置的能力来部署集群。

准备工作

要想离线安装,首先做下以下准备:

一台不受GFW限制的机器或者代理或者下载国内镜像,用于下载安装 Kubernetes 所依赖的海外文件和镜像。 一个用于离线安装的静态服务器,存储安装集群所需的二进制静态文件。通常使用 nginx 搭建静态服务器即可。 一个用于离线安装的镜像仓库,存储安装集群所需的依赖镜像。比如自己搭建的 Harbor,只要网络可以通,能够正常拉取到镜像即可。

生成依赖文件和镜像的列表

cd contrib/offline
bash generate_list.sh
tree temp/

temp/
├── files.list
├── files.list.template
├── images.list
└── images.list.template
  • flies.list 是依赖文件的列表。
  • images.list 是依赖镜像的列表。

搬运文件

执行以下命令将依赖的静态文件全部下载到 temp/files 目录下:

# 这里使用代理 
export https_proxy="192.168.10.250:7890"
wget -x -P temp/files -i temp/files.list

将静态文件通过静态服务器暴露出来,比如使用 nginx,根据情况修改 nginx 配置,比如:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    location /k8s/ {
        alias /root/kubespray/contrib/offline/temp/files/;
    }
}

搬运镜像

Harbor Proxy Cache

这里为快速使用skopeo 将依赖的镜像同步到我们自己的镜像仓库,安装方法参考 官方安装文档.

yum -y install skopeo

安装好后,登录下自己的镜像仓库:

docker login registry.sundayhk.com Username: admin Password: Login Succeeded! 然后将所有依赖镜像同步到我们自己的镜像仓库:

# 这里使用代理 
export https_proxy="192.168.10.250:7890"

for image in $(cat temp/images.list); do skopeo copy docker://${image} docker://registry.sundayhk.com/k8s/${image#*/}; done

google_containers 注意事项:

替换成自己的仓库地址。 提前创建好仓库,比如用 harbor,提前创建好名为 “k8s” 的项目,以便将所有镜像都同步到 “k8s” 这个项目路径下。 如果直接二进制安装 skopeo,需提前创建好配置文件 /etc/containers/policy.json,内容可以用默认的,参考 default-policy.json

修改 offline.yml

搬运好了文件和镜像,我们来修改下 kubespray 的地址,让依赖的文件和镜像下载地址使用我们自己的地址,修改 /root/kubespray/inventory/mycluster/group_vars/all/offline.yml:

  • registry_host: registry.sundayhk.com/k8s
  • files_repo: http://192.168.10.220/k8s
# 备份
cp inventory/mycluster/group_vars/all/offline.yml{,.bak}
# registry_host: "myprivateregisry.com"
sed -i 's@^# registry_host: "myprivateregisry.com"@registry_host: "registry.sundayhk.com/k8s"@g' inventory/mycluster/group_vars/all/offline.yml
# files_repo: "http://myprivatehttpd" 
sed -i 's@^# files_repo: "http://myprivatehttpd"@files_repo: "http://192.168.10.220/k8s"@g' inventory/mycluster/group_vars/all/offline.yml
# 取消注释 files_repo和registry_host
sed -i -E '/# .*\{\{ files_repo/s/^# //g' inventory/mycluster/group_vars/all/offline.yml
sed -i -E '/# .*\{\{ registry_host/s/^# //g' inventory/mycluster/group_vars/all/offline.yml
cat inventory/mycluster/group_vars/all/offline.yml

# 替换后的镜像地址
registry_host: "registry.sundayhk.com/k8s"
# 替换后的静态文件地址
files_repo: "http://192.168.10.220/k8s"

## Container Registry overrides
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
github_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"

## Kubernetes components
kubeadm_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"
kubectl_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl"
kubelet_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet"

## CNI Plugins
cni_download_url: "{{ files_repo }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"

## cri-tools
crictl_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

## [Optional] etcd: only if you **DON'T** use etcd_deployment=host
etcd_download_url: "{{ files_repo }}/github.com/etcd-io/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"

# [Optional] Calico: If using Calico network plugin
calicoctl_download_url: "{{ files_repo }}/github.com/projectcalico/calico/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
calicoctl_alternate_download_url: "{{ files_repo }}/github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
# [Optional] Calico with kdd: If using Calico network plugin with kdd datastore
calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz"

# [Optional] Cilium: If using Cilium network plugin
ciliumcli_download_url: "{{ files_repo }}/github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_version }}/cilium-linux-{{ image_arch }}.tar.gz"

# [Optional] Flannel: If using Falnnel network plugin
flannel_cni_download_url: "{{ files_repo }}/kubernetes/flannel/{{ flannel_cni_version }}/flannel-{{ image_arch }}"

# [Optional] helm: only if you set helm_enabled: true
helm_download_url: "{{ files_repo }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz"

# [Optional] crun: only if you set crun_enabled: true
crun_download_url: "{{ files_repo }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}"

# [Optional] kata: only if you set kata_containers_enabled: true
kata_containers_download_url: "{{ files_repo }}/github.com/kata-containers/kata-containers/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz"

# [Optional] cri-dockerd: only if you set container_manager: docker
cri_dockerd_download_url: "{{ files_repo }}/github.com/Mirantis/cri-dockerd/releases/download/v{{ cri_dockerd_version }}/cri-dockerd-{{ cri_dockerd_version }}.{{ image_arch }}.tgz"

# [Optional] cri-o: only if you set container_manager: crio
# crio_download_base: "download.opensuse.org/repositories/devel:kubic:libcontainers:stable"
# crio_download_crio: "http://{{ crio_download_base }}:/cri-o:/"

# [Optional] runc,containerd: only if you set container_runtime: containerd
runc_download_url: "{{ files_repo }}/github.com/opencontainers/runc/releases/download/{{ runc_version }}/runc.{{ image_arch }}"
containerd_download_url: "{{ files_repo }}/github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
nerdctl_download_url: "{{ files_repo }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"