dohoon-kim.kr
Login
Infra
수정·

K8S 클러스터 장애

by Dohoon Kim · 26년 04월 27일 00:43:38

인증서 갱신 후 클러스터 내부 네트워크 장애

증상

  • 파드에서 다른 파드로 통신 불가
  • 파드에서 외부와 통신 불가
  • CoreDNS를 통한 DNS 조회 불가(외부/내부 모두)

확인 사항

(base) ➜ study-crawler git:(backend) ✗ kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide 
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 
coredns-55cb58b774-cmpwv 1/1 Running 1 (4d2h ago) 433d 10.244.0.58 respect-me-test <none> <none> 
coredns-55cb58b774-n82bg 1/1 Running 1 (4d2h ago) 433d 10.244.0.59 respect-me-test <none> <none> 

(base) ➜ study-crawler git:(backend) ✗ kubectl get svc -n kube-system kube-dns -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 433d k8s-app=kube-dns

(base) ➜ study-crawler git:(backend) ✗ kubectl run dns-test \ --image=busybox \ -it --rm --restart=Never -- sh 
  If you don't see a command prompt, try pressing enter. / 
  # nc -zv 10.96.0.10 53 [반응 없음] 
(base) ➜ study-crawler git:(backend) ✗ kubectl run dns-test \ --image=busybox \ -it --rm --restart=Never -- sh 
  If you don't see a command prompt, try pressing enter. /
  # telnet 10.96.0.10 53 [반응없음]

CoreDNS는 살아있지만, 파드 → CoreDNS(Service IP 10.96.0.10) 통신이 안 됨

조치

CoreDNS 재시작 및 Kubelet 재시작

효과 없음

노드 포워딩 상태 확인

(base) ➜  / sudo iptables -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-FORWARD  all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             192.168.43.0/24      ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.43.0/24      anywhere
ACCEPT     all  --  192.168.43.0/24      anywhere
KUBE-PROXY-FIREWALL  all  --  anywhere             anywhere             ctstate NEW /* kubernetes load balancer firewall */
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
DROP       all  --  anywhere             anywhere
FLANNEL-FWD  all  --  anywhere             anywhere             /* flanneld forward */

DROP이 FLANNEL-FWD보다 먼저 존재하여 CNI 패킷이 모두 드랍됨

sudo iptables -I FORWARD 1 -j FLANNEL-FWD
sudo netfilter-persistent save

ping 8.8.8.8 은 성공, dns 조회는 여전히 실패

(base) ➜  / kubectl run dns-test \
  --image=busybox \
  -it --rm --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # nslookup google.com
;; connection timed out; no servers could be reached

/ # ping 8.8.4.4
PING 8.8.4.4 (8.8.4.4): 56 data bytes
64 bytes from 8.8.4.4: seq=0 ttl=120 time=0.863 ms
64 bytes from 8.8.4.4: seq=1 ttl=120 time=1.397 ms
64 bytes from 8.8.4.4: seq=2 ttl=120 time=0.902 ms
^C
--- 8.8.4.4 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.863/1.054/1.397 ms
/ # ping google.com

CoreDNS 재시작

(base) ➜  / kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
NAME                       READY   STATUS    RESTARTS      AGE    IP             NODE              NOMINATED NODE   READINESS GATES
coredns-55cb58b774-cmpwv   0/1     Running   7 (57s ago)   433d   10.244.0.120   respect-me-test   <none>           <none>
coredns-55cb58b774-n82bg   0/1     Running   7 (63s ago)   433d   10.244.0.138   respect-me-test   <none>           <none>

(base) ➜  / kubectl get svc -n kube-system kube-dns -o yaml | grep selector -A 3
  selector:
    k8s-app: kube-dns
  sessionAffinity: None

(base) ➜  / kubectl get pods -n kube-system --show-labels | grep coredns
coredns-55cb58b774-cmpwv                  0/1     Running   7 (73s ago)   433d   k8s-app=kube-dns,pod-template-hash=55cb58b774
coredns-55cb58b774-n82bg                  0/1     Running   7 (79s ago)   433d   k8s-app=kube-dns,pod-template-hash=55cb58b774  type: ClusterIP

CoreDNS가 제대로 실행되어있지 않음

kubectl delete pod -n kube-system -l k8s-app=kube-dns

네임스페이스 별 전체 파드 재시작

kubectl delete pod -A -n <namespace>