kube-proxy and weave-net keep hanging in ContainerCreating state on master node












1















We are running a small kubernetes cluster with 4 nodes. After the master-node hang, we had to restart it.



Now kube-proxy and weave-net pods on the master node keep hanging in state ContainerCreating, while running fine on all other nodes:



kubectl get pod -o wide  -n kube-system 
NAME READY STATUS RESTARTS AGE IP NODE
coredns-78fcdf6894-67wkh 1/1 Running 0 12h 10.42.0.40 i101v182
coredns-78fcdf6894-8xwkq 1/1 Running 3 122d 10.44.0.65 i101v181
etcd-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
heapster-74db55987-sthnp 1/1 Running 1 40d 10.44.0.60 i101v181
kube-apiserver-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
kube-controller-manager-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
kube-proxy-6dqzf 0/1 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
kube-proxy-nvghr 1/1 Running 9 187d 10.1.101.182 i101v182
kube-proxy-spchz 1/1 Running 8 187d 10.1.101.181 i101v181
kube-proxy-xhg77 1/1 Running 7 187d 10.1.101.183 i101v183
kube-scheduler-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
kubernetes-dashboard-767dc7d4d-ws79h 1/1 Running 0 12h 10.42.0.41 i101v182
metrics-server-696868464d-b8njh 1/1 Running 0 12h 10.42.0.42 i101v182
monitoring-influxdb-848b9b66f6-gwbf5 1/1 Running 1 46d 10.44.0.61 i101v181
weave-net-465np 2/2 Running 3 64d 10.1.101.182 i101v182
weave-net-5mgdp 0/2 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
weave-net-chsqv 2/2 Running 2 40d 10.1.101.183 i101v183
weave-net-mcn77 2/2 Running 18 187d 10.1.101.181 i101v181


There are no errors in the log, nor any events that would give a reason for this:



kubectl describe pod kube-proxy-6dqzf  -n kube-system 
Name: kube-proxy-6dqzf
Namespace: kube-system
Node: i101v180.intra.graz.at/10.1.101.180
Start Time: Wed, 16 Jan 2019 14:32:51 +0100
Labels: controller-revision-hash=1151982146
k8s-app=kube-proxy
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 10.1.101.180
Controlled By: DaemonSet/kube-proxy
Containers:
kube-proxy:
Container ID:
Image: k8s.gcr.io/kube-proxy-amd64:v1.11.0
Image ID:
Port: <none>
Host Port: <none>
Command:
/usr/local/bin/kube-proxy
--config=/var/lib/kube-proxy/config.conf
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/var/lib/kube-proxy from kube-proxy (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-9t8cp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-proxy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-proxy
Optional: false
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
kube-proxy-token-9t8cp:
Type: Secret (a volume populated by a Secret)
SecretName: kube-proxy-token-9t8cp
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/arch=amd64
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events: <none>


kubectl describe pod weave-net-5mgdp  -n kube-system 
Name: weave-net-5mgdp
Namespace: kube-system
Node: i101v180.intra.graz.at/10.1.101.180
Start Time: Wed, 16 Jan 2019 14:21:57 +0100
Labels: controller-revision-hash=2963469815
name=weave-net
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 10.1.101.180
Controlled By: DaemonSet/weave-net
Containers:
weave:
Container ID:
Image: weaveworks/weave-kube:2.3.0
Image ID:
Port: <none>
Host Port: <none>
Command:
/home/weave/launch.sh
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 10m
Liveness: http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s #success=1 #failure=3
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/host/etc from cni-conf (rw)
/host/home from cni-bin2 (rw)
/host/opt from cni-bin (rw)
/host/var/lib/dbus from dbus (rw)
/lib/modules from lib-modules (rw)
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
/weavedb from weavedb (rw)
weave-npc:
Container ID:
Image: weaveworks/weave-npc:2.3.0
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 10m
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
weavedb:
Type: HostPath (bare host directory volume)
Path: /var/lib/weave
HostPathType:
cni-bin:
Type: HostPath (bare host directory volume)
Path: /opt
HostPathType:
cni-bin2:
Type: HostPath (bare host directory volume)
Path: /home
HostPathType:
cni-conf:
Type: HostPath (bare host directory volume)
Path: /etc
HostPathType:
dbus:
Type: HostPath (bare host directory volume)
Path: /var/lib/dbus
HostPathType:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType:
weave-net-token-dcjzm:
Type: Secret (a volume populated by a Secret)
SecretName: weave-net-token-dcjzm
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events: <none>


Any ideas what might be the cause of this?










share|improve this question



























    1















    We are running a small kubernetes cluster with 4 nodes. After the master-node hang, we had to restart it.



    Now kube-proxy and weave-net pods on the master node keep hanging in state ContainerCreating, while running fine on all other nodes:



    kubectl get pod -o wide  -n kube-system 
    NAME READY STATUS RESTARTS AGE IP NODE
    coredns-78fcdf6894-67wkh 1/1 Running 0 12h 10.42.0.40 i101v182
    coredns-78fcdf6894-8xwkq 1/1 Running 3 122d 10.44.0.65 i101v181
    etcd-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
    heapster-74db55987-sthnp 1/1 Running 1 40d 10.44.0.60 i101v181
    kube-apiserver-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
    kube-controller-manager-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
    kube-proxy-6dqzf 0/1 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
    kube-proxy-nvghr 1/1 Running 9 187d 10.1.101.182 i101v182
    kube-proxy-spchz 1/1 Running 8 187d 10.1.101.181 i101v181
    kube-proxy-xhg77 1/1 Running 7 187d 10.1.101.183 i101v183
    kube-scheduler-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
    kubernetes-dashboard-767dc7d4d-ws79h 1/1 Running 0 12h 10.42.0.41 i101v182
    metrics-server-696868464d-b8njh 1/1 Running 0 12h 10.42.0.42 i101v182
    monitoring-influxdb-848b9b66f6-gwbf5 1/1 Running 1 46d 10.44.0.61 i101v181
    weave-net-465np 2/2 Running 3 64d 10.1.101.182 i101v182
    weave-net-5mgdp 0/2 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
    weave-net-chsqv 2/2 Running 2 40d 10.1.101.183 i101v183
    weave-net-mcn77 2/2 Running 18 187d 10.1.101.181 i101v181


    There are no errors in the log, nor any events that would give a reason for this:



    kubectl describe pod kube-proxy-6dqzf  -n kube-system 
    Name: kube-proxy-6dqzf
    Namespace: kube-system
    Node: i101v180.intra.graz.at/10.1.101.180
    Start Time: Wed, 16 Jan 2019 14:32:51 +0100
    Labels: controller-revision-hash=1151982146
    k8s-app=kube-proxy
    pod-template-generation=1
    Annotations: <none>
    Status: Pending
    IP: 10.1.101.180
    Controlled By: DaemonSet/kube-proxy
    Containers:
    kube-proxy:
    Container ID:
    Image: k8s.gcr.io/kube-proxy-amd64:v1.11.0
    Image ID:
    Port: <none>
    Host Port: <none>
    Command:
    /usr/local/bin/kube-proxy
    --config=/var/lib/kube-proxy/config.conf
    State: Waiting
    Reason: ContainerCreating
    Ready: False
    Restart Count: 0
    Environment: <none>
    Mounts:
    /lib/modules from lib-modules (ro)
    /run/xtables.lock from xtables-lock (rw)
    /var/lib/kube-proxy from kube-proxy (rw)
    /var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-9t8cp (ro)
    Conditions:
    Type Status
    Initialized True
    Ready False
    ContainersReady False
    PodScheduled True
    Volumes:
    kube-proxy:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: kube-proxy
    Optional: false
    xtables-lock:
    Type: HostPath (bare host directory volume)
    Path: /run/xtables.lock
    HostPathType: FileOrCreate
    lib-modules:
    Type: HostPath (bare host directory volume)
    Path: /lib/modules
    HostPathType:
    kube-proxy-token-9t8cp:
    Type: Secret (a volume populated by a Secret)
    SecretName: kube-proxy-token-9t8cp
    Optional: false
    QoS Class: BestEffort
    Node-Selectors: beta.kubernetes.io/arch=amd64
    Tolerations: CriticalAddonsOnly
    node-role.kubernetes.io/master:NoSchedule
    node.kubernetes.io/disk-pressure:NoSchedule
    node.kubernetes.io/memory-pressure:NoSchedule
    node.kubernetes.io/not-ready:NoExecute
    node.kubernetes.io/unreachable:NoExecute
    Events: <none>


    kubectl describe pod weave-net-5mgdp  -n kube-system 
    Name: weave-net-5mgdp
    Namespace: kube-system
    Node: i101v180.intra.graz.at/10.1.101.180
    Start Time: Wed, 16 Jan 2019 14:21:57 +0100
    Labels: controller-revision-hash=2963469815
    name=weave-net
    pod-template-generation=1
    Annotations: <none>
    Status: Pending
    IP: 10.1.101.180
    Controlled By: DaemonSet/weave-net
    Containers:
    weave:
    Container ID:
    Image: weaveworks/weave-kube:2.3.0
    Image ID:
    Port: <none>
    Host Port: <none>
    Command:
    /home/weave/launch.sh
    State: Waiting
    Reason: ContainerCreating
    Ready: False
    Restart Count: 0
    Requests:
    cpu: 10m
    Liveness: http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s #success=1 #failure=3
    Environment:
    HOSTNAME: (v1:spec.nodeName)
    Mounts:
    /host/etc from cni-conf (rw)
    /host/home from cni-bin2 (rw)
    /host/opt from cni-bin (rw)
    /host/var/lib/dbus from dbus (rw)
    /lib/modules from lib-modules (rw)
    /run/xtables.lock from xtables-lock (rw)
    /var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
    /weavedb from weavedb (rw)
    weave-npc:
    Container ID:
    Image: weaveworks/weave-npc:2.3.0
    Image ID:
    Port: <none>
    Host Port: <none>
    State: Waiting
    Reason: ContainerCreating
    Ready: False
    Restart Count: 0
    Requests:
    cpu: 10m
    Environment:
    HOSTNAME: (v1:spec.nodeName)
    Mounts:
    /run/xtables.lock from xtables-lock (rw)
    /var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
    Conditions:
    Type Status
    Initialized True
    Ready False
    ContainersReady False
    PodScheduled True
    Volumes:
    weavedb:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/weave
    HostPathType:
    cni-bin:
    Type: HostPath (bare host directory volume)
    Path: /opt
    HostPathType:
    cni-bin2:
    Type: HostPath (bare host directory volume)
    Path: /home
    HostPathType:
    cni-conf:
    Type: HostPath (bare host directory volume)
    Path: /etc
    HostPathType:
    dbus:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/dbus
    HostPathType:
    lib-modules:
    Type: HostPath (bare host directory volume)
    Path: /lib/modules
    HostPathType:
    xtables-lock:
    Type: HostPath (bare host directory volume)
    Path: /run/xtables.lock
    HostPathType:
    weave-net-token-dcjzm:
    Type: Secret (a volume populated by a Secret)
    SecretName: weave-net-token-dcjzm
    Optional: false
    QoS Class: Burstable
    Node-Selectors: <none>
    Tolerations: :NoSchedule
    node.kubernetes.io/disk-pressure:NoSchedule
    node.kubernetes.io/memory-pressure:NoSchedule
    node.kubernetes.io/not-ready:NoExecute
    node.kubernetes.io/unreachable:NoExecute
    Events: <none>


    Any ideas what might be the cause of this?










    share|improve this question

























      1












      1








      1








      We are running a small kubernetes cluster with 4 nodes. After the master-node hang, we had to restart it.



      Now kube-proxy and weave-net pods on the master node keep hanging in state ContainerCreating, while running fine on all other nodes:



      kubectl get pod -o wide  -n kube-system 
      NAME READY STATUS RESTARTS AGE IP NODE
      coredns-78fcdf6894-67wkh 1/1 Running 0 12h 10.42.0.40 i101v182
      coredns-78fcdf6894-8xwkq 1/1 Running 3 122d 10.44.0.65 i101v181
      etcd-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
      heapster-74db55987-sthnp 1/1 Running 1 40d 10.44.0.60 i101v181
      kube-apiserver-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
      kube-controller-manager-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
      kube-proxy-6dqzf 0/1 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
      kube-proxy-nvghr 1/1 Running 9 187d 10.1.101.182 i101v182
      kube-proxy-spchz 1/1 Running 8 187d 10.1.101.181 i101v181
      kube-proxy-xhg77 1/1 Running 7 187d 10.1.101.183 i101v183
      kube-scheduler-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
      kubernetes-dashboard-767dc7d4d-ws79h 1/1 Running 0 12h 10.42.0.41 i101v182
      metrics-server-696868464d-b8njh 1/1 Running 0 12h 10.42.0.42 i101v182
      monitoring-influxdb-848b9b66f6-gwbf5 1/1 Running 1 46d 10.44.0.61 i101v181
      weave-net-465np 2/2 Running 3 64d 10.1.101.182 i101v182
      weave-net-5mgdp 0/2 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
      weave-net-chsqv 2/2 Running 2 40d 10.1.101.183 i101v183
      weave-net-mcn77 2/2 Running 18 187d 10.1.101.181 i101v181


      There are no errors in the log, nor any events that would give a reason for this:



      kubectl describe pod kube-proxy-6dqzf  -n kube-system 
      Name: kube-proxy-6dqzf
      Namespace: kube-system
      Node: i101v180.intra.graz.at/10.1.101.180
      Start Time: Wed, 16 Jan 2019 14:32:51 +0100
      Labels: controller-revision-hash=1151982146
      k8s-app=kube-proxy
      pod-template-generation=1
      Annotations: <none>
      Status: Pending
      IP: 10.1.101.180
      Controlled By: DaemonSet/kube-proxy
      Containers:
      kube-proxy:
      Container ID:
      Image: k8s.gcr.io/kube-proxy-amd64:v1.11.0
      Image ID:
      Port: <none>
      Host Port: <none>
      Command:
      /usr/local/bin/kube-proxy
      --config=/var/lib/kube-proxy/config.conf
      State: Waiting
      Reason: ContainerCreating
      Ready: False
      Restart Count: 0
      Environment: <none>
      Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/kube-proxy from kube-proxy (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-9t8cp (ro)
      Conditions:
      Type Status
      Initialized True
      Ready False
      ContainersReady False
      PodScheduled True
      Volumes:
      kube-proxy:
      Type: ConfigMap (a volume populated by a ConfigMap)
      Name: kube-proxy
      Optional: false
      xtables-lock:
      Type: HostPath (bare host directory volume)
      Path: /run/xtables.lock
      HostPathType: FileOrCreate
      lib-modules:
      Type: HostPath (bare host directory volume)
      Path: /lib/modules
      HostPathType:
      kube-proxy-token-9t8cp:
      Type: Secret (a volume populated by a Secret)
      SecretName: kube-proxy-token-9t8cp
      Optional: false
      QoS Class: BestEffort
      Node-Selectors: beta.kubernetes.io/arch=amd64
      Tolerations: CriticalAddonsOnly
      node-role.kubernetes.io/master:NoSchedule
      node.kubernetes.io/disk-pressure:NoSchedule
      node.kubernetes.io/memory-pressure:NoSchedule
      node.kubernetes.io/not-ready:NoExecute
      node.kubernetes.io/unreachable:NoExecute
      Events: <none>


      kubectl describe pod weave-net-5mgdp  -n kube-system 
      Name: weave-net-5mgdp
      Namespace: kube-system
      Node: i101v180.intra.graz.at/10.1.101.180
      Start Time: Wed, 16 Jan 2019 14:21:57 +0100
      Labels: controller-revision-hash=2963469815
      name=weave-net
      pod-template-generation=1
      Annotations: <none>
      Status: Pending
      IP: 10.1.101.180
      Controlled By: DaemonSet/weave-net
      Containers:
      weave:
      Container ID:
      Image: weaveworks/weave-kube:2.3.0
      Image ID:
      Port: <none>
      Host Port: <none>
      Command:
      /home/weave/launch.sh
      State: Waiting
      Reason: ContainerCreating
      Ready: False
      Restart Count: 0
      Requests:
      cpu: 10m
      Liveness: http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s #success=1 #failure=3
      Environment:
      HOSTNAME: (v1:spec.nodeName)
      Mounts:
      /host/etc from cni-conf (rw)
      /host/home from cni-bin2 (rw)
      /host/opt from cni-bin (rw)
      /host/var/lib/dbus from dbus (rw)
      /lib/modules from lib-modules (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
      /weavedb from weavedb (rw)
      weave-npc:
      Container ID:
      Image: weaveworks/weave-npc:2.3.0
      Image ID:
      Port: <none>
      Host Port: <none>
      State: Waiting
      Reason: ContainerCreating
      Ready: False
      Restart Count: 0
      Requests:
      cpu: 10m
      Environment:
      HOSTNAME: (v1:spec.nodeName)
      Mounts:
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
      Conditions:
      Type Status
      Initialized True
      Ready False
      ContainersReady False
      PodScheduled True
      Volumes:
      weavedb:
      Type: HostPath (bare host directory volume)
      Path: /var/lib/weave
      HostPathType:
      cni-bin:
      Type: HostPath (bare host directory volume)
      Path: /opt
      HostPathType:
      cni-bin2:
      Type: HostPath (bare host directory volume)
      Path: /home
      HostPathType:
      cni-conf:
      Type: HostPath (bare host directory volume)
      Path: /etc
      HostPathType:
      dbus:
      Type: HostPath (bare host directory volume)
      Path: /var/lib/dbus
      HostPathType:
      lib-modules:
      Type: HostPath (bare host directory volume)
      Path: /lib/modules
      HostPathType:
      xtables-lock:
      Type: HostPath (bare host directory volume)
      Path: /run/xtables.lock
      HostPathType:
      weave-net-token-dcjzm:
      Type: Secret (a volume populated by a Secret)
      SecretName: weave-net-token-dcjzm
      Optional: false
      QoS Class: Burstable
      Node-Selectors: <none>
      Tolerations: :NoSchedule
      node.kubernetes.io/disk-pressure:NoSchedule
      node.kubernetes.io/memory-pressure:NoSchedule
      node.kubernetes.io/not-ready:NoExecute
      node.kubernetes.io/unreachable:NoExecute
      Events: <none>


      Any ideas what might be the cause of this?










      share|improve this question














      We are running a small kubernetes cluster with 4 nodes. After the master-node hang, we had to restart it.



      Now kube-proxy and weave-net pods on the master node keep hanging in state ContainerCreating, while running fine on all other nodes:



      kubectl get pod -o wide  -n kube-system 
      NAME READY STATUS RESTARTS AGE IP NODE
      coredns-78fcdf6894-67wkh 1/1 Running 0 12h 10.42.0.40 i101v182
      coredns-78fcdf6894-8xwkq 1/1 Running 3 122d 10.44.0.65 i101v181
      etcd-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
      heapster-74db55987-sthnp 1/1 Running 1 40d 10.44.0.60 i101v181
      kube-apiserver-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
      kube-controller-manager-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
      kube-proxy-6dqzf 0/1 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
      kube-proxy-nvghr 1/1 Running 9 187d 10.1.101.182 i101v182
      kube-proxy-spchz 1/1 Running 8 187d 10.1.101.181 i101v181
      kube-proxy-xhg77 1/1 Running 7 187d 10.1.101.183 i101v183
      kube-scheduler-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
      kubernetes-dashboard-767dc7d4d-ws79h 1/1 Running 0 12h 10.42.0.41 i101v182
      metrics-server-696868464d-b8njh 1/1 Running 0 12h 10.42.0.42 i101v182
      monitoring-influxdb-848b9b66f6-gwbf5 1/1 Running 1 46d 10.44.0.61 i101v181
      weave-net-465np 2/2 Running 3 64d 10.1.101.182 i101v182
      weave-net-5mgdp 0/2 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
      weave-net-chsqv 2/2 Running 2 40d 10.1.101.183 i101v183
      weave-net-mcn77 2/2 Running 18 187d 10.1.101.181 i101v181


      There are no errors in the log, nor any events that would give a reason for this:



      kubectl describe pod kube-proxy-6dqzf  -n kube-system 
      Name: kube-proxy-6dqzf
      Namespace: kube-system
      Node: i101v180.intra.graz.at/10.1.101.180
      Start Time: Wed, 16 Jan 2019 14:32:51 +0100
      Labels: controller-revision-hash=1151982146
      k8s-app=kube-proxy
      pod-template-generation=1
      Annotations: <none>
      Status: Pending
      IP: 10.1.101.180
      Controlled By: DaemonSet/kube-proxy
      Containers:
      kube-proxy:
      Container ID:
      Image: k8s.gcr.io/kube-proxy-amd64:v1.11.0
      Image ID:
      Port: <none>
      Host Port: <none>
      Command:
      /usr/local/bin/kube-proxy
      --config=/var/lib/kube-proxy/config.conf
      State: Waiting
      Reason: ContainerCreating
      Ready: False
      Restart Count: 0
      Environment: <none>
      Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/kube-proxy from kube-proxy (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-9t8cp (ro)
      Conditions:
      Type Status
      Initialized True
      Ready False
      ContainersReady False
      PodScheduled True
      Volumes:
      kube-proxy:
      Type: ConfigMap (a volume populated by a ConfigMap)
      Name: kube-proxy
      Optional: false
      xtables-lock:
      Type: HostPath (bare host directory volume)
      Path: /run/xtables.lock
      HostPathType: FileOrCreate
      lib-modules:
      Type: HostPath (bare host directory volume)
      Path: /lib/modules
      HostPathType:
      kube-proxy-token-9t8cp:
      Type: Secret (a volume populated by a Secret)
      SecretName: kube-proxy-token-9t8cp
      Optional: false
      QoS Class: BestEffort
      Node-Selectors: beta.kubernetes.io/arch=amd64
      Tolerations: CriticalAddonsOnly
      node-role.kubernetes.io/master:NoSchedule
      node.kubernetes.io/disk-pressure:NoSchedule
      node.kubernetes.io/memory-pressure:NoSchedule
      node.kubernetes.io/not-ready:NoExecute
      node.kubernetes.io/unreachable:NoExecute
      Events: <none>


      kubectl describe pod weave-net-5mgdp  -n kube-system 
      Name: weave-net-5mgdp
      Namespace: kube-system
      Node: i101v180.intra.graz.at/10.1.101.180
      Start Time: Wed, 16 Jan 2019 14:21:57 +0100
      Labels: controller-revision-hash=2963469815
      name=weave-net
      pod-template-generation=1
      Annotations: <none>
      Status: Pending
      IP: 10.1.101.180
      Controlled By: DaemonSet/weave-net
      Containers:
      weave:
      Container ID:
      Image: weaveworks/weave-kube:2.3.0
      Image ID:
      Port: <none>
      Host Port: <none>
      Command:
      /home/weave/launch.sh
      State: Waiting
      Reason: ContainerCreating
      Ready: False
      Restart Count: 0
      Requests:
      cpu: 10m
      Liveness: http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s #success=1 #failure=3
      Environment:
      HOSTNAME: (v1:spec.nodeName)
      Mounts:
      /host/etc from cni-conf (rw)
      /host/home from cni-bin2 (rw)
      /host/opt from cni-bin (rw)
      /host/var/lib/dbus from dbus (rw)
      /lib/modules from lib-modules (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
      /weavedb from weavedb (rw)
      weave-npc:
      Container ID:
      Image: weaveworks/weave-npc:2.3.0
      Image ID:
      Port: <none>
      Host Port: <none>
      State: Waiting
      Reason: ContainerCreating
      Ready: False
      Restart Count: 0
      Requests:
      cpu: 10m
      Environment:
      HOSTNAME: (v1:spec.nodeName)
      Mounts:
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
      Conditions:
      Type Status
      Initialized True
      Ready False
      ContainersReady False
      PodScheduled True
      Volumes:
      weavedb:
      Type: HostPath (bare host directory volume)
      Path: /var/lib/weave
      HostPathType:
      cni-bin:
      Type: HostPath (bare host directory volume)
      Path: /opt
      HostPathType:
      cni-bin2:
      Type: HostPath (bare host directory volume)
      Path: /home
      HostPathType:
      cni-conf:
      Type: HostPath (bare host directory volume)
      Path: /etc
      HostPathType:
      dbus:
      Type: HostPath (bare host directory volume)
      Path: /var/lib/dbus
      HostPathType:
      lib-modules:
      Type: HostPath (bare host directory volume)
      Path: /lib/modules
      HostPathType:
      xtables-lock:
      Type: HostPath (bare host directory volume)
      Path: /run/xtables.lock
      HostPathType:
      weave-net-token-dcjzm:
      Type: Secret (a volume populated by a Secret)
      SecretName: weave-net-token-dcjzm
      Optional: false
      QoS Class: Burstable
      Node-Selectors: <none>
      Tolerations: :NoSchedule
      node.kubernetes.io/disk-pressure:NoSchedule
      node.kubernetes.io/memory-pressure:NoSchedule
      node.kubernetes.io/not-ready:NoExecute
      node.kubernetes.io/unreachable:NoExecute
      Events: <none>


      Any ideas what might be the cause of this?







      linux virtualization






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Jan 16 at 14:52









      PeterPeter

      61




      61






















          0






          active

          oldest

          votes











          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "3"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1394977%2fkube-proxy-and-weave-net-keep-hanging-in-containercreating-state-on-master-node%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Super User!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1394977%2fkube-proxy-and-weave-net-keep-hanging-in-containercreating-state-on-master-node%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          How to make a Squid Proxy server?

          Is this a new Fibonacci Identity?

          19世紀