Prometheus自动发现监控域名
经常修改服务端配置文件会比较麻烦，使用服务机制解决这个问题
参考03黑盒监控prometheus-additional.yaml配置文件继续添加：
- job_name: 'auto_discovery'
metrics_path: /probe
params:
module: [http_2xx]
kubernetes_sd_configs:
- role: ingress
relabel_configs:
- source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_http_probe]
action: keep
regex: true
- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
regex: (.+);(.+);(.+)
replacement: ${1}://${2}${3}
target_label: __param_target
- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
regex: (.+);(.+);(.+)
replacement: ${1}://${2}${3}
target_label: target
- target_label: __address__
replacement: blackbox-exporter:9115
- source_labels: [__param_target]
target_label: instance
- action: labelmap
regex: __meta_kubernetes_ingress_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_ingress_name]
target_label: kubernetes_namekubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yamlkubectl apply -f additional-scrape-configs.yaml -n monitoring查看日志：serviceaccount无权访问ingress资源权限
kubectl -n monitoring logs -f prometheus-k8s-0 prometheus
此时需要添加一个clusterrolebinding
vim autodiscover-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus-discovery
namespace: monitoring
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ratel-resource-readonly
subjects:
- namespace: monitoring
kind: ServiceAccount
name: prometheus-k8s这里需要角色ratel-resource-readonly，根据文章https://www.ljh.cool/wp-admin/post.php?post=37543&action=edit中ratel-resource-readonly角色创建
没有继续报错后job产生
添加一个域名：
创建nginx应用，添加annotations进行测试
kubectl create deployment web --image=nginxkubectl expose deployment web --port=80 --target-port=80
vim ingress-network.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
kubernetes.io/ingress.class: "nginx" #声名需要使用nginx方式解析ingress
prometheus.io/http_probe: "true" #添加域名自动发现
spec:
rules: # 一个Ingress可以配置多个rules
- host: example.ingredemo.com # 域名配置，可以不写（不建议），支持匹配，一般写法：*.bar.com
http:
paths: # 相当于nginx的location配合，同一个host可以配置多个path 例如“/”或“/abc”
- path: "/"
pathType: Prefix
backend:
service:
name: web
port:
number: 80kubectl create -f ingress-network.yaml
域名添加到本地的hosts中进行访问
查看监控
Prometheus监控Java JVM
JVM (Micrometer)插件方式进行监控（SpringBoot）
Demo项目：https://github.com/gongchangwangpi/spring-cloud-demo2
git clone https://github.com/gongchangwangpi/spring-cloud-demo2.git
cd spring-cloud-demo2
跑一个docker环境
docker pull maven:3.5.3
docker run -it --rm -v /opt/m2:/root/.m2 -v `pwd`:/opt/ -p 18761:8761 maven:3.5.3 bash
root@4ef5b21ec133:/# cd /opt/
root@4ef5b21ec133:/opt# ls
root@4ef5b21ec133:/opt# vi spring-cloud-eureka/pom.xml
添加内容
<!-- Micrometer Prometheus registry -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<!-- finished -->
当前目录下src/main/resources/application.yml配置暴露接口
参考 https://grafana.com/grafana/dashboards/4701-jvm-micrometer/修改文件
cat <<'EOF'>> src/main/resources/application.yml
management:
endpoints:
web:
exposure:
include: '*'
shutdown:
enable: false
metrics:
tags:
application: "${spring.application.name}"
EOFmaven编译命令：mvn clean package -DskipTests
启动服务
java -jar target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar
成功启动
访问页面：http://192.168.1.10:18761/
查看插件提供的metrics数据接口 http://192.168.1.10:18761/actuator/prometheus
添加到prometheus监控
prometheus-additional.yaml添加：
- job_name: 'jvm-prometheus'
scheme: http
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['192.168.1.10:18761'] kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml
kubectl apply -f additional-scrape-configs.yaml -n monitoring观察prometheus targets
添加dashboard
https://grafana.com/grafana/dashboards/4701-jvm-micrometer/
对于开发来讲，展示相关垃圾数据和黑盒数据，可以省去开发登陆服务器进行jmap jstat去查看，还可以去做一些预测性的告警
自动监控JAVA SpringCloud项目
刚才使用单个SpringBoot，需要一个一个项目添加到target中，但是一般使用场景都是SpringCloud全家桶，下面是用eureka注册功能自动发现，监控SpringCloud项目
复制创建一个新的eureka项目，在此eureka中配置consul插件
vim pom.xml
<dependency>
<groupId>at.twinformatics</groupId>
<artifactId>eureka-consul-adapter</artifactId>
<version>1.1.0</version>
</dependency>
同理：修改项目下的 src/main/resources/application.yml
server:
port: 8080
eureka:
client:
serviceUrl:
defaultZone: http://192.168.1.10:18761/eureka/ #填写第一个项目的注册地址
spring:
application:
name: cloud-user # 新项目名称
management:
endpoints:
web:
exposure:
include: '*'
shutdown:
enable: false
metrics:
tags:
application: "${spring.application.name}"重新启动新容器，开启spring-cloud-user项目
docker run -it --rm -v /opt/m2:/root/.m2 -v `pwd`:/opt/ --network=host maven:3.5.3 bash编译：mvn clean package -DskipTests
开启服务
java -jar target/spring-cloud-eureka-0.0.1-SNAPSHOT.jar --server.port=8080
检测8080端口页面以及18761端口注册情况
在prometheus中配置插件
添加到prometheus监控
prometheus-additional.yaml修改：
### other configurations
- job_name: 'jvm-discovery-prometheus'
scheme: http
metrics_path: '/actuator/prometheus'
consul_sd_configs:
- server: '192.168.1.10:18761' #eureka的地址
scheme: http
services: []
kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml
kubectl apply -f additional-scrape-configs.yaml -n monitoring检测target，成功实现自动注册
发布者：LJH，转发请注明出处：https://www.ljh.cool/38066.html
