k8s监控篇-02 prometheus处理监控组件自带metrics和非自带metrics监控问题

组件自带metrics接口，且有ServiceMonitor，但没有service和endpoint：

二进制安装的kube-controller-manager和kube-scheduler无法获取metrics（kubeadm方式不会出现此问题）

k8s监控篇-02 prometheus处理监控组件自带metrics和非自带metrics监控问题

查看问题：

寻找kube-controller-manager和kube-scheduler的红色的告警

controller-manager监控

1、因为controller监听了127.0.0.1，所以监听地址需要修改成全网段

2、prometheus自动注册了targets相关监控项，但是没有对应标签的service

kubectl -n monitoring get servicemonitor

kubectl -n monitoring get servicemonitor kube-controller-manager -o yaml | less

可查出，这个监控项在kube-system的ns中找一个标签为k8s-app: kube-controller-manager的service但是kube-system没有这个service

处理问题

所有master修改：

vim /usr/lib/systemd/system/kube-controller-manager.service

创建endpoint和svc，名称保持一致（kube-controller-manage-monitor）

vim endpoint-svc.yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      k8s-app: kube-controller-manager # svc添加label，需要和servicemonitor寻找的一致
    name: kube-controller-manage-monitor
    namespace: kube-system # 需要和servicemonitor寻找的名称空间一致
  spec:
    ports:
    - name: http-metrics # 端口名称需要匹配servicemonitor寻找的名称
      port: 10252 # 监听kube-controller-manager暴露的10252端口
      protocol: TCP
      targetPort: 10252
    type: ClusterIP
- apiVersion: v1
  kind: Endpoints
  metadata:
    labels:
      k8s-app: kube-controller-manager # endpoint添加label
    name: kube-controller-manage-monitor #名称与service一致
    namespace: kube-system
  subsets:
  - addresses:
    - ip: 192.168.1.10 # 监听节点
    - ip: 192.168.1.11
    - ip: 192.168.1.12
    ports:
    - name: http-metrics  # 端口名称需要匹配servicemonitor寻找的名称
      port: 10252 # 端口一致
      protocol: TCP
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

kubectl create -f endpoint-svc.yaml

成功获取metrics

kube-scheduler监控

同理，所有master修改：

vim /usr/lib/systemd/system/kube-scheduler.service

systemctl daemon-reload
systemctl restart kube-scheduler.service

vim endpoint-svc-scheduler.yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      k8s-app: kube-scheduler # svc添加label
    name: kube-scheduler-monitor
    namespace: kube-system
  spec:
    ports:
    - name: http-metrics # 端口名称需要匹配
      port: 10251 # 监听kube-scheduler暴露的10252端口
      protocol: TCP
      targetPort: 10251
    type: ClusterIP
- apiVersion: v1
  kind: Endpoints
  metadata:
    labels:
      k8s-app: kube-scheduler # endpoint添加label
    name: kube-scheduler-monitor #名称与service一致
    namespace: kube-system
  subsets:
  - addresses:
    - ip: 192.168.1.10 # 监听节点
    - ip: 192.168.1.11
    - ip: 192.168.1.12
    ports:
    - name: http-metrics  # 端口名称需要匹配
      port: 10251 # 端口一致
      protocol: TCP
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

kubectl create -f endpoint-svc-scheduler.yaml

组件自带metrics接口，但没有ServiceMonitor

etcd监控

获取etcd的metrics：

curl https://192.168.1.10:2379/metrics --cert /etc/etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem -k

尝试创建endpoint+service连接到这个端口，然后使用service获取到metrics

vim etcd-svc-edpoint.yml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      k8s-app: etcd
    name: etcd-monitor
    namespace: kube-system
  spec:
    ports:
    - name: etcd
      port: 2379
      protocol: TCP
      targetPort: 2379
    type: ClusterIP

- apiVersion: v1
  kind: Endpoints
  metadata:
    labels:
      k8s-app: etcd
    name: etcd-monitor
    namespace: kube-system
  subsets:
  - addresses:
    - ip: 192.168.1.10
    - ip: 192.168.1.11
    - ip: 192.168.1.12
    ports:
    - name: etcd
      port: 2379
      protocol: TCP
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

kubectl create -f etcd-svc-edpoint.yml

使用创建的svc地址测试下metrics

curl https://10.102.133.91:2379/metrics --cert /etc//etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem -k

生成secret形式证书：

kubectl -n monitoring create secret generic etcd-certs --from-file=/etc/etcd/ssl/etcd.pem --from-file=/etc/etcd/ssl/etcd-key.pem --from-file=/etc/etcd/ssl/etcd-ca.pem

创建etcd的ServiceMonitor

vim etcd-serviceMonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: etcd
  name: etcd
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: etcd # 名字对应
    scheme: https # 需要配置成https证书形式
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-certs/etcd-ca.pem
      certFile: /etc/prometheus/secrets/etcd-certs/etcd.pem
      keyFile: /etc/prometheus/secrets/etcd-certs/etcd-key.pem
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - kube-system # 名称空间对应
  selector:
    matchLabels:
      k8s-app: etcd # 与etcd service相匹配

修改kube-prometheus/manifests下的prometheus-prometheus.yaml

vim prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.15.2
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 1
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.15.2
  secrets: # 添加证书配置
  - etcd-certs

kubectl replace -f prometheus-prometheus.yaml

成功获取

可以添加一个dashboard https://grafana.com/grafana/dashboards/3070-etcd/

导入json

组件没有metrics接口

如果是非云原生应用组件，开发时可能没有metrics接口，只能使用第三方exporter（或exporter插件）定期去采集相关数据

可以通过github直接寻找中间件的exporter，下面以这个kafka为例

https://github.com/search?q=exporter&type=repositories

先部署一套kafka集群，依据https://www.ljh.cool/37543.html文档方式部署

kubectl get po -n public-service

直接将镜像部署到k8s中

vim kafka-exporter.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: kafka-exporter
  name: kafka-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: kafka-exporter
  template:
    metadata:
      labels:
        app: kafka-exporter
    spec:
      containers:
      - args:
        - --kafka.server=kafka-0.kafka-headless.public-service:9092
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        image: danielqsj/kafka-exporter:latest
        imagePullPolicy: IfNotPresent
        name: kafka-exporter
        ports:
        - containerPort: 9308
          name: web
          protocol: TCP
        volumeMounts:
        - mountPath: /usr/share/zoneinfo/Asia/Shanghai
          name: tz-config
        - mountPath: /etc/localtime
          name: tz-config
        - mountPath: /etc/timezone
          name: timezone
      dnsPolicy: ClusterFirst
      volumes:
      - hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
          type: ""
        name: tz-config
      - hostPath:
          path: /etc/timezone
          type: ""
        name: timezone

kubectl create -f kafka-exporter.yaml

kubectl get deployments.apps -n monitoring

查看pod日志

kubectl -n monitoring logs -f kafka-exporter-74cc5d69c6-vmwvc

vim kafka-exporter-svc.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-exporter
  name: kafka-exporter-monitor
  namespace: monitoring
spec:
  ports:
  - name: container-port
    port: 9308
    protocol: TCP
    targetPort: 9308
  selector:
    app: kafka-exporter
  type: ClusterIP

kubectl create -f kafka-exporter-svc.yaml

kubectl get svc -n monitoring

curl 10.109.248.95:9308/metrics

serviceMonitor注册

vim kafka-serviceMonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kafka-exporter
  name: kafka-exporter
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: container-port
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      app: kafka-exporter