k8s

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

        Unfortunately, an error has occurred:
                timed out waiting for the condition

        This error is likely caused by:
                - The kubelet is not running
                - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                - 'systemctl status kubelet'
                - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI.

        Here is one example how you may list all Kubernetes containers running in docker:
                - 'docker ps -a | grep kube | grep -v pause'
                Once you have found the failing container, you can inspect its logs with:
                - 'docker logs CONTAINERID'

[ERROR Port-10259]: Port 10259 is in use
[ERROR Port-10257]: Port 10257 is in use
[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
[ERROR Port-10250]: Port 10250 is in use

[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
error execution phase preflight

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.9.43:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:5660805073db31916952821a8751ca0ee0644ce4205f616805f8a7f175ff8b33

jd@jnode:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                            READY   STATUS    RESTARTS   AGE
kube-system   coredns-74ff55c5b-f76bb         1/1     Running   0          5m52s
kube-system   coredns-74ff55c5b-frgkw         1/1     Running   0          5m52s
kube-system   etcd-jnode                      1/1     Running   0          6m8s
kube-system   kube-apiserver-jnode            1/1     Running   0          6m8s
kube-system   kube-controller-manager-jnode   1/1     Running   0          6m8s
kube-system   kube-flannel-ds-xghcx           1/1     Running   0          4m4s
kube-system   kube-proxy-nnbbx                1/1     Running   0          5m52s
kube-system   kube-scheduler-jnode            1/1     Running   0          6m8s

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.9.43:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:5660805073db31916952821a8751ca0ee0644ce4205f616805f8a7f175ff8b33

# 重新生成新的token
kubeadm token create
kubeadm token list

# 获取ca证书sha256编码hash值，拿到的hash值要记录下来
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

# 节点加入集群，token和hash值使用上面生成的
kubeadm join ip:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx

$ kubectl get nodes
The connection to the server 192.168.9.43:6443 was refused - did you specify the right host or port?

# 在主节点上使用scp将配置文件复制到worker
scp ~/.kube/config xx@xxx.xxx.xxx.xxx:~/.kube/config

kubectl describe node
The connection to the server localhost:8080 was refused - did you specify the right host or port?

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
  - groups:
      - system:bootstrappers:kubeadm:default-node-token
    token: abcdef.0123456789abcdef
    ttl: 24h0m0s
    usages:
      - signing
      - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.9.43 # 本机的ip地址，可通过`ip a`获取
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: jnode
  taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.20.0
networking:
  dnsDomain: cluster.local
  podSubnet: 10.46.128.0/21 # pod subnet值需要与kube-flannel.yml里的网络设置一致
  serviceSubnet: 192.168.1.0/24 # 跟本机ip同一网段
scheduler: {}

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
    - configMap
    - secret
    - emptyDir
    - hostPath
  allowedHostPaths:
    - pathPrefix: "/etc/cni/net.d"
    - pathPrefix: "/etc/kube-flannel"
    - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ["NET_ADMIN", "NET_RAW"]
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
    - min: 0
      max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: "RunAsAny"
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    verbs: ["use"]
    resourceNames: ["psp.flannel.unprivileged"]
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
  - kind: ServiceAccount
    name: flannel
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }    
  net-conf.json: |
    {
      "Network": "10.46.128.0/21", # 与init-default.yaml配置文件里的podSubnet值一致
      "Backend": {
        "Type": "vxlan"
      }
    }    
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
        - operator: Exists
          effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
        - name: install-cni
          image: quay.io/coreos/flannel:v0.13.1-rc2
          command:
            - cp
          args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
          volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
      containers:
        - name: kube-flannel
          image: quay.io/coreos/flannel:v0.13.1-rc2
          command:
            - /opt/bin/flanneld
          args:
            - --ip-masq
            - --kube-subnet-mgr
          resources:
            requests:
              cpu: "100m"
              memory: "50Mi"
            limits:
              cpu: "100m"
              memory: "50Mi"
          securityContext:
            privileged: false
            capabilities:
              add: ["NET_ADMIN", "NET_RAW"]
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

#!/bin/bash

sudo kubeadm join ip:6443 --token wxjhuh.z9ru6bbz990m7v0i --discovery-token-ca-cert-hash sha256:8809f1cc27401d704faa104008a48034b51e8e5ef8f4a8f33f0e267db0124a2f

The connection to the server localhost:8080 was refused - did you specify the right host or port?

# 针对deployment hello-world以NodePort方式、example-service名称、端口是port-value暴露应用到公网
kubectl expose deployment hello-world --type=NodePort --name=example-service -- port=port-value

jd@wnode1:~/Project/jdnote$ kubectl describe svc
Name:                     jdnote-server-service
Namespace:                default
Labels:                   app=jdnote-server
Annotations:              <none>
Selector:                 app=jdnote-server
Type:                     NodePort
IP Families:              <none>
IP:                       192.168.1.198
IPs:                      192.168.1.198
Port:                     <unset>  8890/TCP
TargetPort:               8890/TCP
NodePort:                 <unset>  32256/TCP # 这个值作为端口
Endpoints:                10.46.129.11:8890
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

jd@wnode1:~/Project/jdnote$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:cf:aa:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.9.16/24 brd 192.168.9.255 scope global dynamic enp0s3 # 192.168.9.16作为ip
       valid_lft 9843sec preferred_lft 9843sec
    inet6 fe80::a00:27ff:fecf:aa13/64 scope link
       valid_lft forever preferred_lft forever

apiVersion: networking.k8s.io/v1
kind: Ingress # kind必须指定为ingress
metadata:
  name: minimal-ingress # 名称
  annotations: # 注解
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules: # 规则
    - http:
        paths: # 路径
          - path: /testpath
            pathType: Prefix
            backend: # 重定向到service
              service:
                name: test # 必须存在test service
                port:
                  number: 80 # service的端口

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-wildcard-host
spec:
  rules:
    - host: "foo.bar.com" # 域名，表示该域名的流量将被转到配置的service
      http:
        paths:
          - pathType: Prefix
            path: "/bar"
            backend:
              service:
                name: service1
                port:
                  number: 80
    - host: "*.foo.com"
      http:
        paths:
          - pathType: Prefix
            path: "/foo"
            backend:
              service:
                name: service2
                port:
                  number: 80

$ kubectl exec -it skm-7fb6bd989c-kg9xj -n skm-system -- sh # 使用 kubectl 进入容器的 sh 界面

sh-4.4# dlv exec /root/Project/rancher/bin/rancher # exec 报错：版本太旧
Version of Delve is too old for this version of Go (maximum supported version 1.13, suppress this error with --check-go-version=false)
sh-4.4# dlv attach /root/Project/rancher/bin/rancher # attach 需要指定 pid
Invalid pid: /root/Project/rancher/bin/rancher
sh-4.4# dlv attach 653                              # 通过 ps aux 拿到 pid
Type 'help' for list of commands.
(dlv)