k8s监控篇-03黑盒监控

白盒监控：监控一些内部的数据，topic的监控数据，Redis key的大小。内部暴露的指标被称为白盒监控。比较关注的是原因。

黑盒监控：站在用户的角度看到的东西。网站不能打开，网站打开的比较慢。比较关注现象，表示正在发生的问题，正在发生的告警。

注意：新版Prometheus已经默认安装了黑盒监控，如果blackbox的deployment已经存在，就不用在安装了，可以直接使用。不过需要将Blackbox的Servicemonitor改成http

github文档：

https://github.com/prometheus/blackbox_exporter

创建deployment configMap service暴露服务

vim blackbox_exporter.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: blackbox-exporter
  name: blackbox-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: blackbox-exporter
  template:
    metadata:
      labels:
        app: blackbox-exporter
    spec:
      containers:
      - args:
        - --config.file=/mnt/blackbox.yml
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        image: prom/blackbox-exporter:master
        imagePullPolicy: IfNotPresent
        name: blackbox-exporter
        ports:
        - containerPort: 9115
          name: web
          protocol: TCP
        volumeMounts:
        - mountPath: /usr/share/zoneinfo/Asia/Shanghai
          name: tz-config
        - mountPath: /etc/localtime
          name: tz-config
        - mountPath: /etc/timezone
          name: timezone
        - mountPath: /mnt
          name: config
      volumes:
      - hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
          type: ""
        name: tz-config
      - hostPath:
          path: /etc/timezone
          type: ""
        name: timezone
      - configMap:
          defaultMode: 420
          name: blackbox-conf
        name: config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-conf
  namespace: monitoring
data:
  blackbox.yml: |-
    modules:
      http_2xx:
        prober: http
      #  http: # 国内建议强调使用ipv4，可以解开注释
      #    preferred_ip_protocol: "ip4"
      http_post_2xx:
        prober: http
        http:
          method: POST
      tcp_connect:
        prober: tcp
      pop3s_banner:
        prober: tcp
        tcp:
          query_response:
          - expect: "^+OK"
          tls: true
          tls_config:
            insecure_skip_verify: false
      ssh_banner:
        prober: tcp
        tcp:
          query_response:
          - expect: "^SSH-2.0-"
      irc_banner:
        prober: tcp
        tcp:
          query_response:
          - send: "NICK prober"
          - send: "USER prober prober prober :prober"
          - expect: "PING :([^ ]+)"
            send: "PONG ${1}"
          - expect: "^:[^ ]+ 001"
      icmp:
        prober: icmp
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: blackbox-exporter
  name: blackbox-exporter
  namespace: monitoring
spec:
  ports:
  - name: container-port
    port: 9115
    protocol: TCP
    targetPort: 9115
  selector:
    app: blackbox-exporter

部署后测试网站：

例如百度：

curl "http://10.107.135.39:9115/probe?target=baidu.com&module=http_2xx" | less

黄色高亮分别代表service服务IP，要检测的网址，使用的检测模块类型

获取到dns解析时间，网络延迟等信息

如果在prometheus中习惯使用传统配置方式监控，可以将传统文件集成到当中

ExtraScrapeConfigs 允许指定包含附加 Prometheus 抓取配置的 Secret 密钥。指定的抓取配置将附加到 Prometheus Operator 生成的配置中。

旧版参考文档：

https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md

vim prometheus-additional.yaml

- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]  # Look for a HTTP 200 response.
  static_configs:
    - targets:
      - https://www.baidu.com    # Target to probe with https.
      # - http://example1.com    # Target to probe with http.
      # - http://example2.com:8080     # Target to probe with http on port 8080.
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115  # The blackbox exporter's real hostname:port.

然后需要对此配置secret加密

kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml

接下来，应用生成的 kubernetes 清单

kubectl apply -f additional-scrape-configs.yaml -n monitoring

最后，在 CRD 中引用此附加配置prometheus.yaml

vim prometheus-prometheus.yaml

添加

  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml

kubectl replace -f prometheus-prometheus.yaml

kubectl -n monitoring delete pod prometheus-k8s-0 prometheus-k8s-1

查看配置，已集成到prometheus

DNS解析时间

添加dashboard：https://grafana.com/grafana/dashboards/19342-blackbox-exporter/

温馨提示：

kubectl -n monitoring logs -f prometheus-k8s-0 prometheus

当日志为level=info ts=2023-09-03T07:16:25.123Z caller=main.go:762 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml表示成功热更新，否则会报错，根据报错内容进行查错