k8s监控篇-05 Prometheus自动发现及监控java项目

Prometheus自动发现监控域名

经常修改服务端配置文件会比较麻烦，使用服务机制解决这个问题

参考03黑盒监控prometheus-additional.yaml配置文件继续添加：

- job_name: 'auto_discovery'
  metrics_path: /probe
  params:
    module: [http_2xx]  
  kubernetes_sd_configs:
  - role: ingress
  relabel_configs:
  - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_http_probe]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
    regex: (.+);(.+);(.+)
    replacement: ${1}://${2}${3}
    target_label: __param_target
  - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
    regex: (.+);(.+);(.+)
    replacement: ${1}://${2}${3}
    target_label: target
  - target_label: __address__
    replacement: blackbox-exporter:9115
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_ingress_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_ingress_name]
    target_label: kubernetes_name

kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml

kubectl apply -f additional-scrape-configs.yaml -n monitoring

查看日志：serviceaccount无权访问ingress资源权限

kubectl -n monitoring logs -f prometheus-k8s-0 prometheus

此时需要添加一个clusterrolebinding

vim autodiscover-clusterrolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-discovery 
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ratel-resource-readonly
subjects:
- namespace: monitoring 
  kind: ServiceAccount
  name: prometheus-k8s

这里需要角色ratel-resource-readonly，根据文章https://www.ljh.cool/wp-admin/post.php?post=37543&action=edit中ratel-resource-readonly角色创建

没有继续报错后job产生

添加一个域名：

创建nginx应用，添加annotations进行测试

kubectl create deployment web --image=nginx
kubectl expose deployment web --port=80 --target-port=80

vim ingress-network.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx" #声名需要使用nginx方式解析ingress
    prometheus.io/http_probe: "true" #添加域名自动发现
spec:
  rules: # 一个Ingress可以配置多个rules
  - host: example.ingredemo.com # 域名配置，可以不写（不建议），支持匹配，一般写法：*.bar.com
    http:
      paths: # 相当于nginx的location配合，同一个host可以配置多个path 例如“/”或“/abc”
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: web
            port:
              number: 80

kubectl create -f ingress-network.yaml

域名添加到本地的hosts中进行访问

查看监控

Prometheus监控Java JVM

JVM (Micrometer)插件方式进行监控（SpringBoot）

Demo项目：https://github.com/gongchangwangpi/spring-cloud-demo2

git clone https://github.com/gongchangwangpi/spring-cloud-demo2.git
cd spring-cloud-demo2

跑一个docker环境

docker pull maven:3.5.3
docker run -it --rm -v /opt/m2:/root/.m2 -v `pwd`:/opt/ -p 18761:8761 maven:3.5.3 bash
root@4ef5b21ec133:/# cd /opt/
root@4ef5b21ec133:/opt# ls

root@4ef5b21ec133:/opt# vi spring-cloud-eureka/pom.xml

添加内容

<!-- Micrometer Prometheus registry  -->
        <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
          </dependency>
         <dependency>
                <groupId>io.micrometer</groupId>
                 <artifactId>micrometer-core</artifactId>
         </dependency>
         <dependency>
                <groupId>io.micrometer</groupId>
                <artifactId>micrometer-registry-prometheus</artifactId>
         </dependency>
        <!-- finished -->

当前目录下src/main/resources/application.yml配置暴露接口

参考 https://grafana.com/grafana/dashboards/4701-jvm-micrometer/修改文件

cat <<'EOF'>> src/main/resources/application.yml
management:
  endpoints:
    web:
      exposure:
        include: '*'
    shutdown:
      enable: false
  metrics:
    tags:
      application: "${spring.application.name}" 
EOF

maven编译命令：mvn clean package -DskipTests

启动服务

java -jar target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar

成功启动

访问页面：http://192.168.1.10:18761/

查看插件提供的metrics数据接口 http://192.168.1.10:18761/actuator/prometheus

添加到prometheus监控

prometheus-additional.yaml添加：

- job_name: 'jvm-prometheus'
  scheme: http
  metrics_path: '/actuator/prometheus'
  static_configs:
  - targets: ['192.168.1.10:18761'] 

kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml
kubectl apply -f additional-scrape-configs.yaml -n monitoring

观察prometheus targets

添加dashboard

https://grafana.com/grafana/dashboards/4701-jvm-micrometer/

对于开发来讲，展示相关垃圾数据和黑盒数据，可以省去开发登陆服务器进行jmap jstat去查看，还可以去做一些预测性的告警

自动监控JAVA SpringCloud项目

刚才使用单个SpringBoot，需要一个一个项目添加到target中，但是一般使用场景都是SpringCloud全家桶，下面是用eureka注册功能自动发现，监控SpringCloud项目

复制创建一个新的eureka项目，在此eureka中配置consul插件

vim pom.xml

        <dependency>
            <groupId>at.twinformatics</groupId>
            <artifactId>eureka-consul-adapter</artifactId>
            <version>1.1.0</version>
        </dependency>

同理：修改项目下的 src/main/resources/application.yml

server:
  port: 8080

eureka:
  client:
    serviceUrl:
      defaultZone: http://192.168.1.10:18761/eureka/ #填写第一个项目的注册地址

spring:
  application:
    name: cloud-user # 新项目名称

management:
  endpoints:
    web:
      exposure:
        include: '*'
    shutdown:
      enable: false
  metrics:
    tags:
      application: "${spring.application.name}"

重新启动新容器，开启spring-cloud-user项目

docker run -it --rm -v /opt/m2:/root/.m2 -v `pwd`:/opt/ --network=host maven:3.5.3 bash

编译：mvn clean package -DskipTests

开启服务

java -jar target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar --server.port=8080

检测8080端口页面以及18761端口注册情况

在prometheus中配置插件

添加到prometheus监控

prometheus-additional.yaml修改：

### other configurations
- job_name: 'jvm-discovery-prometheus'
  scheme: http
  metrics_path: '/actuator/prometheus'
  consul_sd_configs:
    - server: '192.168.1.10:18761' #eureka的地址
      scheme: http
      services: []

kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml
kubectl apply -f additional-scrape-configs.yaml -n monitoring

检测target，成功实现自动注册