⇧ Oh, my gosh...
VagrantでKubernetsの環境をオンプレミス環境に構築したいのだが...
相も変わらず、オンプレミス環境でKubernetesを構築する情報が少ないのですが、
⇧ 公式のドキュメントによると、「kind」は紹介されておらず、
- kubeadm
- Kubernetes Cluster API
- kOps
- kubespray
の4つの導入方法があるらしいですと。
「kubeadm」による導入方法は、
⇧ 上記のドキュメントがあるのだけど、Kubernetes環境構築に必要な情報が分り辛い...
「Vagrant」と「Ansible」を利用する場合、
⇧ ユーザーの問題があると。
「Ansible」は、
⇧ ファイルの権限の問題もありますと。
VagrantとVirtualBoxが安定しなさ過ぎるんだが...
とりあえず、
のどちらについても、不具合が多過ぎて環境構築が全くスムーズにいかない...
環境に依存しないのが目的のはずなんだが、正常に動作させるための条件がブラックボックス過ぎて、実用に耐えない感じになっているのが誠に遺憾ではある...
wingetでインストールしたのだけど、
No | ソフトウェア | バージョン |
---|---|---|
1 | Vagrant | 2.4.3 |
2 | VirtualBox | 7.1.4 |
上記のようなバージョンを利用する形になったのだが、対応している「Linux ディストリビューション」のバージョンが分り辛い...
というか、エラーになって利用できないものが多過ぎて、最早、メンテナンスがされていないのではないかという気がしてならない。
そもそも、「Vagrant」で利用可能な「BOXファイル」をコマンドで確認できないのが残念過ぎる...
そして、Windowsをホストとした場合のVagrantfileのサンプルが公式のドキュメントに無いのが辛い...
⇧ Windows環境については、全く考慮してくれていない...
雰囲気で設定するしかないってカオス過ぎるんだが...
VagrantとVirtualBoxでAnsibleをゲストOSにインストールしKubernetes環境を構築したかったが...
ちょっと情報が古いのだけど、
⇧ 公式のブログでAnsibleとVagrantで構築する情報があるのだけど、「ホスト側」と「ゲスト側」のどちらに何が必要なのかが分り辛い...
何故、要件を整理しないのかサッパリ理解できない...
ホスト側のWindows 10 Homeに以下のファイルを用意しておく。
vagrant/ ├── Vagrantfile # Vagrantの設定ファイル ├── ansible/ │ ├── ansible.cfg # Ansible の設定ファイル │ ├── inventory.ini # 対象マシンのIPアドレスなど │ └── k8s-setup.yml # Ansibleプレイブック(Kubernetesセットアップ用) ├── k8s-manifests/ # Kubernetesマニフェストファイル │ ├── awx-deployment.yml # AWXのKubernetesデプロイメント │ └── squid-deployment.yml # SquidのKubernetesデプロイメント └── docker-compose.yml # ワーカーノード2用のdocker-compose設定
で、諸々のバグのせいで、ファイルの内容の設定が遅々として進まないのだが...
情報が錯綜し過ぎていて、正しい情報が不明ということもあって、最終的に1週間ぐらいかかったのだが、以下のような内容になりました。
■D:\work-soft\vagrant\ansible\ansible.cfg
[defaults] host_key_checking = False
■D:\work-soft\vagrant\ansible\inventory.ini
[master] 192.168.50.10 ansible_user=vagrant ansible_ssh_private_key_file=/home/vagrant/.ssh/id_rsa [worker1] 192.168.50.11 ansible_user=vagrant ansible_ssh_private_key_file=/home/vagrant/.ssh/id_rsa [worker2] 192.168.50.12 ansible_user=vagrant ansible_ssh_private_key_file=/home/vagrant/.ssh/id_rsa [workers:children] worker1 worker2 [all:children] master workers
■D:\work-soft\vagrant\ansible\k8s-setup.yml
--- - hosts: all become: yes tasks: - name: Install required packages apt: name: - apt-transport-https - ca-certificates - curl - gpg state: present update_cache: yes - name: Download Google Cloud public signing key shell: curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | gpg --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg - name: Add Kubernetes apt repository shell: echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list - name: Update apt package index apt: update_cache: yes - name: Install Docker apt: name: docker.io state: present update_cache: yes - name: Install Kubernetes packages apt: name: - kubeadm - kubelet - kubectl state: present update_cache: yes - name: Hold Kubernetes packages command: apt-mark hold kubeadm kubelet kubectl - name: Start Docker service service: name: docker state: started enabled: true - name: Start kubelet service systemd: name: kubelet enabled: yes state: started - name: Disable swap shell: | swapoff -a sed -i '/swap/d' /etc/fstab - hosts: master become: yes tasks: - name: Initialize Kubernetes master node shell: kubeadm init --pod-network-cidr=10.244.0.0/16 register: kubeadm_output ignore_errors: yes - name: Print kubeadm init logs for debugging debug: var: kubeadm_output.stderr_lines - name: Save the kubeadm join command for workers set_fact: join_command: "{{ kubeadm_output.stdout | regex_search('kubeadm join .*') }}" - name: Set up kubeconfig for kubectl shell: | mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config when: kubeadm_output.rc == 0 - name: Wait for Kubernetes API server to be ready shell: | until kubectl get nodes; do sleep 10; done when: kubeadm_output.rc == 0 - name: Apply Flannel CNI plugin shell: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml when: kubeadm_output.rc == 0 - hosts: workers become: yes tasks: - name: Join Kubernetes cluster shell: | export node_ip={{ ansible_host }} && {{ hostvars['master'].join_command }} when: hostvars['master'].join_command is defined - name: Generate AWX and Squid Pods on worker1 shell: | kubectl apply -f /home/vagrant/k8s-manifests/awx-deployment.yml kubectl apply -f /home/vagrant/k8s-manifests/squid-deployment.yml when: inventory_hostname == 'worker1' - name: Set up Docker Compose on worker2 shell: | cd /home/vagrant/docker-compose docker-compose up -d when: inventory_hostname == 'worker2'
■D:\work-soft\vagrant\k8s-manifests\awx-deployment.yml
apiVersion: apps/v1 kind: Deployment metadata: name: awx spec: replicas: 1 selector: matchLabels: app: awx template: metadata: labels: app: awx spec: containers: - name: awx image: ansible/awx:latest ports: - containerPort: 80
■D:\work-soft\vagrant\k8s-manifests\squid-deployment.yml
apiVersion: apps/v1 kind: Deployment metadata: name: squid spec: replicas: 1 selector: matchLabels: app: squid template: metadata: labels: app: squid spec: containers: - name: squid image: sameersbn/squid:latest ports: - containerPort: 3128
■D:\work-soft\vagrant\docker-compose.yml
version: "3.7" services: kubectl: image: bitnami/kubectl:latest container_name: kubectl volumes: - /vagrant/k8s-manifests:/workspace entrypoint: "/bin/bash" tty: true k3s: image: rancher/k3s:v1.25.6-k3s1 container_name: k3s command: server --no-deploy=traefik volumes: - /var/lib/rancher/k3s:/var/lib/rancher/k3s ports: - "6443:6443" environment: - K3S_KUBEVIRT_VERSION=v0.37.1 restart: always
■D:\work-soft\vagrant\Vagrantfile
Vagrant.configure("2") do |config| # 仮想マシンのOS config.vm.box = "bento/ubuntu-24.04" config.vm.box_version = "202404.26.0" # タイムアウトの増加 config.vm.boot_timeout = 900 # common ssh-private-key config.ssh.insert_key = false config.ssh.private_key_path = "C:/Users/toshinobu/.vagrant.d/insecure_private_key" # copy to private-key config.vm.provision "file", source: "C:/Users/toshinobu/.vagrant.d/insecure_private_key", destination: "/home/vagrant/.ssh/id_rsa" config.vm.provision "shell", privileged: false, inline: <<-SHELL chmod 600 /home/vagrant/.ssh/id_rsa SHELL # ワーカーノード1の設定 config.vm.define "worker1" do |worker1| # 仮想マシンのプロバイダ設定 worker1.vm.provider "virtualbox" do |vb| vb.memory = "1024" vb.cpus = 1 end worker1.vm.network "private_network", type: "static", ip: "192.168.50.11" worker1.vm.hostname = "worker1" # フォルダの同期(パーミッション設定を含む) worker1.vm.synced_folder "./ansible", "/home/vagrant/ansible", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] worker1.vm.synced_folder "./k8s-manifests", "/home/vagrant/k8s-manifests", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] # # SSHキーの受け取り # worker1.vm.provision "shell", privileged: true, inline: <<-SHELL # mkdir -p /home/vagrant/.ssh # chmod 700 /home/vagrant/.ssh # [ ! -f /home/vagrant/.ssh/authorized_keys ] && touch /home/vagrant/.ssh/authorized_keys # chmod 600 /home/vagrant/.ssh/authorized_keys # SHELL end # ワーカーノード2の設定 config.vm.define "worker2" do |worker2| # 仮想マシンのプロバイダ設定 worker2.vm.provider "virtualbox" do |vb| vb.memory = "1024" vb.cpus = 1 end worker2.vm.network "private_network", type: "static", ip: "192.168.50.12" worker2.vm.hostname = "worker2" # フォルダの同期(パーミッション設定を含む) worker2.vm.synced_folder "./ansible", "/home/vagrant/ansible", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] worker2.vm.synced_folder "./k8s-manifests", "/home/vagrant/k8s-manifests", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] worker2.vm.provision "file", source: "./docker-compose.yml", destination: "/home/vagrant/docker-compose/docker-compose.yml" # # SSHキーの受け取り # worker2.vm.provision "shell", privileged: true, inline: <<-SHELL # mkdir -p /home/vagrant/.ssh # chmod 700 /home/vagrant/.ssh # [ ! -f /home/vagrant/.ssh/authorized_keys ] && touch /home/vagrant/.ssh/authorized_keys # chmod 600 /home/vagrant/.ssh/authorized_keys # SHELL end # マスターノードの設定 config.vm.define "master" do |master| # 仮想マシンのプロバイダ設定 master.vm.provider "virtualbox" do |vb| vb.memory = "2048" vb.cpus = 2 end master.vm.network "private_network", type: "static", ip: "192.168.50.10" master.vm.hostname = "master" # フォルダの同期(パーミッション設定を含む) master.vm.synced_folder "./ansible", "/home/vagrant/ansible", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] master.vm.synced_folder "./k8s-manifests", "/home/vagrant/k8s-manifests", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] master.vm.provision "shell", privileged: true, inline: <<-SHELL # # Ansibleの対象マシンに接続するためのSSHキー生成 # ssh-keygen -t rsa -b 2048 -f /home/vagrant/.ssh/id_rsa -q -N "" # Ansible設定(ホスト鍵検証無効化) echo "[defaults]" > /home/vagrant/ansible/ansible.cfg echo "host_key_checking = False" >> /home/vagrant/ansible/ansible.cfg sudo chmod -R 755 /home/vagrant/ansible sudo apt-get install -y sshpass ansible bash-completion # # Ansibleの対象マシンに公開鍵を登録する # for ip in 192.168.50.11 192.168.50.12; do # sshpass -p "vagrant" ssh-copy-id -i /home/vagrant/.ssh/id_rsa.pub -o StrictHostKeyChecking=no vagrant@$ip # done cd /home/vagrant/ansible # Ansibleのplaybookを実行する ansible-playbook -i inventory.ini k8s-setup.yml -u vagrant SHELL end end
で、一応、仮想マシン自体は動作してる。
ただ、処理がスキップされてたりするから、Ansibleのplaybookの設定が間違っているのってことなんですかね...
エラーになってましたわ...
休日が潰れてただけという不毛な時間を浪費してしまったという...
何も得るものがなく、徒労感しかないんだが...
2025年1月4日(土)追記:↓ ここから
とりあえず、
⇧ ログを出力することにしました。
■D:\work-soft\vagrant\Vagrantfile
Vagrant.configure("2") do |config| # 仮想マシンのOS config.vm.box = "bento/ubuntu-24.04" config.vm.box_version = "202404.26.0" # タイムアウトの増加 config.vm.boot_timeout = 900 # common ssh-private-key config.ssh.insert_key = false config.ssh.private_key_path = "C:/Users/toshinobu/.vagrant.d/insecure_private_key" # copy to private-key config.vm.provision "file", source: "C:/Users/toshinobu/.vagrant.d/insecure_private_key", destination: "/home/vagrant/.ssh/id_rsa" config.vm.provision "shell", privileged: false, inline: <<-SHELL chmod 600 /home/vagrant/.ssh/id_rsa SHELL # ワーカーノード1の設定 config.vm.define "worker1" do |worker1| # 仮想マシンのプロバイダ設定 worker1.vm.provider "virtualbox" do |vb| vb.memory = "1024" vb.cpus = 1 end worker1.vm.network "private_network", type: "static", ip: "192.168.50.11" worker1.vm.hostname = "worker1" # フォルダの同期(パーミッション設定を含む) worker1.vm.synced_folder "./ansible", "/home/vagrant/ansible", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] worker1.vm.synced_folder "./k8s-manifests", "/home/vagrant/k8s-manifests", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] # # SSHキーの受け取り # worker1.vm.provision "shell", privileged: true, inline: <<-SHELL # mkdir -p /home/vagrant/.ssh # chmod 700 /home/vagrant/.ssh # [ ! -f /home/vagrant/.ssh/authorized_keys ] && touch /home/vagrant/.ssh/authorized_keys # chmod 600 /home/vagrant/.ssh/authorized_keys # SHELL end # ワーカーノード2の設定 config.vm.define "worker2" do |worker2| # 仮想マシンのプロバイダ設定 worker2.vm.provider "virtualbox" do |vb| vb.memory = "1024" vb.cpus = 1 end worker2.vm.network "private_network", type: "static", ip: "192.168.50.12" worker2.vm.hostname = "worker2" # フォルダの同期(パーミッション設定を含む) worker2.vm.synced_folder "./ansible", "/home/vagrant/ansible", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] worker2.vm.synced_folder "./k8s-manifests", "/home/vagrant/k8s-manifests", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] worker2.vm.provision "file", source: "./docker-compose.yml", destination: "/home/vagrant/docker-compose/docker-compose.yml" # # SSHキーの受け取り # worker2.vm.provision "shell", privileged: true, inline: <<-SHELL # mkdir -p /home/vagrant/.ssh # chmod 700 /home/vagrant/.ssh # [ ! -f /home/vagrant/.ssh/authorized_keys ] && touch /home/vagrant/.ssh/authorized_keys # chmod 600 /home/vagrant/.ssh/authorized_keys # SHELL end # マスターノードの設定 config.vm.define "master" do |master| # 仮想マシンのプロバイダ設定 master.vm.provider "virtualbox" do |vb| vb.memory = "2048" vb.cpus = 2 end master.vm.network "private_network", type: "static", ip: "192.168.50.10" master.vm.hostname = "master" # フォルダの同期(パーミッション設定を含む) master.vm.synced_folder "./ansible", "/home/vagrant/ansible", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] master.vm.synced_folder "./k8s-manifests", "/home/vagrant/k8s-manifests", type: "virtualbox", create: true, mount_options: ["dmode=775", "fmode=664"] master.vm.provision "shell", privileged: true, inline: <<-SHELL # # Ansibleの対象マシンに接続するためのSSHキー生成 # ssh-keygen -t rsa -b 2048 -f /home/vagrant/.ssh/id_rsa -q -N "" # Ansible設定(ホスト鍵検証無効化) echo "[defaults]" > /home/vagrant/ansible/ansible.cfg echo "host_key_checking = False" >> /home/vagrant/ansible/ansible.cfg sudo chmod -R 755 /home/vagrant/ansible sudo apt-get install -y sshpass ansible bash-completion # # Ansibleの対象マシンに公開鍵を登録する # for ip in 192.168.50.11 192.168.50.12; do # sshpass -p "vagrant" ssh-copy-id -i /home/vagrant/.ssh/id_rsa.pub -o StrictHostKeyChecking=no vagrant@$ip # done cd /home/vagrant/ansible # Ansibleのplaybookを実行する ansible-playbook -i inventory.ini k8s-setup.yml -u vagrant -e 'ansible_python_interpreter=/usr/bin/python3' -vvv | tee /home/vagrant/ansible/ansible-playbook.log SHELL end end
AnsibleのPlaybookのkubeadmの方も、
⇧ 上記の情報を元に追記してみたが、
■D:\work-soft\vagrant\ansible\k8s-setup.yml
--- - hosts: all become: yes tasks: - name: Install required packages apt: name: - apt-transport-https - ca-certificates - curl - gpg state: present update_cache: yes - name: Download Google Cloud public signing key shell: curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | gpg --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg - name: Add Kubernetes apt repository shell: echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list - name: Update apt package index apt: update_cache: yes - name: Install Docker apt: name: docker.io state: present update_cache: yes - name: Install Kubernetes packages apt: name: - kubeadm - kubelet - kubectl state: present update_cache: yes - name: Hold Kubernetes packages command: apt-mark hold kubeadm kubelet kubectl - name: Start Docker service service: name: docker state: started enabled: true - name: Start kubelet service systemd: name: kubelet enabled: yes state: started - name: Disable swap shell: | swapoff -a sed -i '/swap/d' /etc/fstab - hosts: master become: yes tasks: - name: Pull required images for Kubernetes shell: kubeadm config images pull - name: Initialize Kubernetes master node shell: > kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.50.10 --image-repository=registry.k8s.io --kubernetes-version=stable-1 --cri-socket=unix:///var/run/containerd/containerd.sock register: kubeadm_output ignore_errors: yes - name: Print kubeadm init logs for debugging debug: var: kubeadm_output.stderr_lines - name: Save the kubeadm join command for workers set_fact: join_command: "{{ kubeadm_output.stdout | regex_search('kubeadm join .*') }}" - name: Set up kubeconfig for kubectl shell: | mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config when: kubeadm_output.rc == 0 - name: Check if admin.conf exists stat: path: /etc/kubernetes/admin.conf register: admin_conf - name: Fail if admin.conf does not exist fail: msg: "admin.conf does not exist" when: not admin_conf.stat.exists - name: Configure kubectl to access the cluster shell: | export KUBECONFIG=/etc/kubernetes/admin.conf when: kubeadm_output.rc == 0 - name: Wait for Kubernetes API server to be ready shell: | until kubectl get nodes; do sleep 10; done when: kubeadm_output.rc == 0 - name: Apply Flannel CNI plugin shell: | kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml when: kubeadm_output.rc == 0 - hosts: workers become: yes tasks: - name: Test SSH connection to master node shell: | ssh -o StrictHostKeyChecking=no -i /home/vagrant/.ssh/id_rsa root@192.168.50.10 echo "SSH connection established" - name: Copy admin.conf from master node shell: | scp -o StrictHostKeyChecking=no -o ConnectTimeout=30 -i /home/vagrant/.ssh/id_rsa root@192.168.50.10:/etc/kubernetes/admin.conf /home/vagrant/admin.conf - name: Proxy API Server to localhost shell: | kubectl --kubeconfig /home/vagrant/admin.conf proxy - name: Join Kubernetes cluster shell: | export node_ip={{ ansible_host }} && {{ hostvars['master'].join_command }} when: hostvars['master'].join_command is defined - name: Generate AWX and Squid Pods on worker1 shell: | kubectl apply -f /home/vagrant/k8s-manifests/awx-deployment.yml kubectl apply -f /home/vagrant/k8s-manifests/squid-deployment.yml when: inventory_hostname == 'worker1' - name: Install Docker Compose on worker2 shell: | curl -L "https://github.com/docker/compose/releases/download/$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep tag_name | cut -d '\"' -f 4)/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose when: inventory_hostname == 'worker2' - name: Set up Docker Compose on worker2 shell: | cd /home/vagrant/docker-compose docker-compose up -d when: inventory_hostname == 'worker2'
⇧ エラーが解消されていない状態...
ログを確認しつつ、対応していく感じになりますか...
Kubernetes導入の公式のドキュメントが分り辛過ぎるんよね...
2025年1月4日(土)追記:↑ ここまで
毎度モヤモヤ感が半端ない…
今回はこのへんで。