Prometheus Consul Blackbox | Export 监控实现

前言：

blackbox_exporter

是Prometheus 官方提供的 exporter 之一,主要提供http、dns、tcp、icmp 的监控数据采集。

Consul

主要提供，服务发现，健康检查，等功能，本次集成主要使用到服务发现功能。

本文主要实现，基于consul_sd_config & consul 的 prometheus 服务发现，实现网路设备ping监控，站点可用行监控，以及证书相关信息监控。

安装环境：

k8s
consul
Prometheus
blackbox_exporter

1: Consul 安装

1.1：使用helm 安装 consul

Bash
# 添加 consul helm 源
helm repo add hashicorp https://helm.releases.hashicorp.com
# 安装consul 
helm -n consul  install  \
--set storageClass=alicloud-disk-efficiency  \
consul hashicorp/consul \
--version=0.32.1

1.2：查看服务安装状态

Bash
[root@xxxxxxxx consul_install]# kubectl -n consul get pods
NAME                     READY   STATUS    RESTARTS   AGE
consul-consul-9lxfc      1/1     Running   06d1h
consul-consul-ntqcf      1/1     Running   06d1h
consul-consul-q7c6f      1/1     Running   06d1h
consul-consul-server-01/1     Running   06d1h
consul-consul-server-11/1     Running   06d1h
consul-consul-server-21/1     Running   06d1h

1.3：nginx-ingress consul

consul_ingress.yml

Bash
# consul.xxxxxx.cn----->  替换为正确域名

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: consul-ingress
  namespace: consul
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target:/
spec:
  rules:
- host: consul.xxxxxx.cn
    http:
      paths:
- path:/
        pathType: Prefix
        backend:
          service:
            name: consul-consul-ui
            port:
              number:80

执行部署

Bash
kubectl apply -f consul_ingress.yml

1.4：访问测试

2: Blackbox_export

2.1：blackbox 安装

blackbox-exporter-config.yaml

Bash
apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-exporter
  labels:
    app: blackbox-exporter
data:
  blackbox.yml:|-
    modules:
      ## ----------- DNS 检测配置 -----------
      dns_tcp:
        prober: dns
        dns:
          transport_protocol:"tcp"
          preferred_ip_protocol:"ip4"
          query_name:"kubernetes.default.svc.cluster.local" # 用于检测域名可用的网址
          query_type:"A"
      ## ----------- TCP 检测模块配置 -----------
      tcp_connect:
        prober: tcp
        timeout:5s
      ## ----------- ICMP 检测配置 -----------
      ping:
        prober: icmp
        timeout:5s
        icmp:
          preferred_ip_protocol:"ip4"
      ## ----------- HTTP GET 2xx 检测模块配置 -----------
      http_get_2xx:
        prober: http
        timeout:10s
        http:
          method: GET
          preferred_ip_protocol:"ip4"
          valid_http_versions:["HTTP/1.1","HTTP/2"]
          valid_status_codes:[200]           # 验证的HTTP状态码,默认为2xx
          no_follow_redirects:false          # 是否不跟随重定向
      ## ----------- HTTP GET 3xx 检测模块配置 -----------
      http_get_3xx:
        prober: http
        timeout:10s
        http:
          method: GET
          preferred_ip_protocol:"ip4"
          valid_http_versions:["HTTP/1.1","HTTP/2"]
          valid_status_codes:[301,302,304,305,306,307]  # 验证的HTTP状态码,默认为2xx
          no_follow_redirects:false                     # 是否不跟随重定向
      ## ----------- HTTP POST 监测模块 -----------
      http_post_2xx:
        prober: http
        timeout:10s
        http:
          method: POST
          preferred_ip_protocol:"ip4"
          valid_http_versions:["HTTP/1.1","HTTP/2"]
          #headers:                             # HTTP头设置
          #  Content-Type: application/json
          #body:'{}'                           # 请求体设置

blackbox-exporter-deploy.yaml

Bash
apiVersion: v1
kind: Service
metadata:
  name: blackbox-exporter
  labels:
    k8s-app: blackbox-exporter
spec:
  type: ClusterIP
  ports:
- name: http
    port:9115
    targetPort:9115
  selector:
    k8s-app: blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox-exporter
  labels:
    k8s-app: blackbox-exporter
spec:
  replicas:1
  selector:
    matchLabels:
      k8s-app: blackbox-exporter
  template:
    metadata:
      labels:
        k8s-app: blackbox-exporter
    spec:
      containers:
- name: blackbox-exporter
        image: prom/blackbox-exporter:v0.19.0
        args:
---config.file=/etc/blackbox_exporter/blackbox.yml
---web.listen-address=:9115
---log.level=info
        ports:
- name: http
          containerPort:9115
        resources:
          limits:
            cpu:3
            memory:6000Mi
          requests:
            cpu:100m
            memory:50Mi
        livenessProbe:
          tcpSocket:
            port:9115
          initialDelaySeconds:5
          timeoutSeconds:5
          periodSeconds:10
          successThreshold:1
          failureThreshold:3
        readinessProbe:
          tcpSocket:
            port:9115
          initialDelaySeconds:5
          timeoutSeconds:5
          periodSeconds:10
          successThreshold:1
          failureThreshold:3
        volumeMounts:
- name: config
          mountPath:/etc/blackbox_exporter
      volumes:
- name: config
        configMap:
          name: blackbox-exporter
          defaultMode:420

执行安装

Bash
kubectl apply -f blackbox-exporter-deploy.yaml
kubectl apply -f blackbox-exporter-config.yaml

2.2：nginx ingress blackbox-exporter • blackbox_ingress.yml

Bash
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blackbox-ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target:/
spec:
  rules:
- host: blackbox-devops.lululemon.cn
    http:
      paths:
- path:/
        pathType: Prefix
        backend:
          service:
            name: blackbox-exporter
            port:
              number:9115

执行安装

Bash
kubectl apply -f blackbox_ingress.yml

3: rometheus 添加服务动态发现

Bash
#####  http_get_2xx 数据获取
- job_name: http_get_2xx
      params:
        module:
- http_get_2xx
      scrape_interval:2s
      scrape_timeout:2s
      metrics_path:/probe
      consul_sd_configs:
      # consul 服务地址
- server: consul-consul-server.consul.svc.cluster.local:8500
        tag_separator:','
        services:
- http_get_2xx
      relabel_configs:
- source_labels:['__meta_consul_service_address']
          target_label: __param_target
- source_labels:['__meta_consul_service_address']
          target_label: instance
- target_label: __address__
          ## blackbox-export  地址
          replacement: blackbox-exporter.monitoring.svc.cluster.local:9115
 ####### icmp 配置 
- job_name: blackbox_icmp
      params:
        module:
- ping
      scrape_interval:2s
      scrape_timeout:2s
      metrics_path:/probe
      consul_sd_configs:
      # consul 服务地址
- server: consul-consul-server.consul.svc.cluster.local:8500
        tag_separator:','
        services:
- ping
      relabel_configs:
- source_labels:['__meta_consul_service_address']
          target_label: __param_target
- source_labels:['__meta_consul_service_address']
          target_label: instance
- target_label: __address__
          ## blackbox-export  地址
          replacement: blackbox-exporter.monitoring.svc.cluster.local:9115

4：添加 icmp 监控

4.1：添加监控地址到consul

icmp_list

Bash
192.168.1.1
192.168.1.2

add_consul_service_icmp.sh

Bash
#!/usr/bin/env bash

ip_addr=$1


if test "$ip_addr";then
        curl -X PUT -d '{
            "id": "icmp_'${ip_addr}'",
            "name": "ping",
            "address": "'${ip_addr}'",
            "port": 443,
            "Meta": {
              "env": "prod",
              "team": "network",
              "project": "network",
              "owner": "Mike"
            },
            "tags": ["node"],
            "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
            http://consul-consul-server:8500/v1/agent/service/register
else
        echo "请输入参数"
fi

添加service ping

Bash
for i in `cat icmp_list`;do bash add_consul_service_icmp.sh  $i;done

4.2：查看consul 服务

4.3：删除ping 监控地址脚本

Bash
#!/usr/bin/env bash
ip_addr=$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ip_addr}

5: 添加http_get_2xx

5.1：添加监控域名

domain_name_list

Bash
wwww.baidu.com
wwww.1111.com
wwww.2222.com

add_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash

service_name=$1


if test "$service_name";then
        curl -X PUT -d '{
            "id": "http_get_2xx_'${service_name}'",
            "name": "http_get_2xx",
            "address": "https://'${service_name}'",
            "port": 443,
            "Meta": {
              "env": "prod",
              "team": "web",
              "project": "web",
              "owner": "Devops"
            },
            "tags": ["node"],
            "checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \
            http://consul-consul-server:8500/v1/agent/service/register
else
        echo "请输入参数"
fi

添加 service http_get_2xx

Bash
for i  in `cat domain_name_list`;do bash  add_consul_service_http_get_2xx.sh  $i;done

5.2：查看consul 服务

5.3：删除域名监控脚本

del_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash
ip_addr=$1

curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ip_addr}

6：查看prometheus 监控

总结：

使用上述方案，黑盒监控与自建cmdb 平台很容易进行集成，使其监控自动化，不需要过多的人工干预，可以省去大量的人工成本，grafana 的配置这里就不进行过多介绍，自行通过谷歌完成。

文章来源网络，作者：运维，如若转载，请注明出处：https://shuyeidc.com/wp/143738.html<