首页 / K8S / k8s 集群部署prometheus + alertmanager + grafana

k8s 集群部署prometheus + alertmanager + grafana

内容导读

互联网集市收集整理的这篇技术教程文章主要介绍了k8s 集群部署prometheus + alertmanager + grafana，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含11008字，纯文字阅读大概需要16分钟。

内容图文

k8s 集群部署prometheus + alertmanager + grafana

准备k8s 集群

前言
准备好k8s 集群，通过部署prometheus 达到获取k8s 容器资源，根据收集指标制定报警策略，从而提高监控响应能力。

$ kubectl get  node
NAME       STATUS     ROLES    AGE   VERSION
master01   Ready      master   13d   v1.16.0
master02   Ready      master   13d   v1.16.0
master03   Ready      master   13d   v1.16.0
node01     Ready      <none>   13d   v1.16.0
node02     Ready      <none>   13d   v1.16.0
node03     Ready      <none>   13d   v1.16.0

部署prometheus

prometheus 配置文件主要监控项目为 kubernetes-nodes （集群各个节点）
prometheus 监控prometheus 服务
kubernetes-services 通过blackbox-exporter 监控 service
kubernetes-nodes-cadvisor 通过 cadvisor 收集各个node 节点Pod 信息
kubernetes-ingresses 通过blackbox-exporter 监控 ingresses
kubernetes-kubelet 监控各个节点 kubelet
traefik 监控traefik 存活
kubernetes-apiservers 监控 apiservers 存活
关键性服务监控监控各个关键性服务状态
blackbox_http_pod_probe 通过blackbox-exporter监控各个pod

├── prometheus-conf.yaml
├── prometheus-deployment.yaml
├── prometheus-ingress.yaml
├── prometheus-pv-pvc.yaml
├── prometheus-rules.yaml
└── prometheus-svc.yaml

一、准备 pv、pvc 配置文件
# cat prometheus-pv-pvc.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-server
  namespace: prometheus
  labels:
    name: prometheus-server
spec:
  nfs:
    path: /export/nfs_share/volume-prometheus/prometheus-server
    server: 10.65.0.94
  accessModes: ["ReadWriteMany","ReadOnlyMany"]
  capacity:
    storage: 50Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-server
  namespace: prometheus
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 50Gi

二、准备prometheus-conf 文件

# cat prometheus-conf.yaml  
apiVersion: v1
data:
  prometheus.yml: |-
    # my global config
    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).

    # Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
            - 'alertmanager-service.prometheus:9093'
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      - "/etc/prometheus/rules/nodedown.rule.yml"
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
          - targets: ['localhost:9090']


      - job_name: '关键性服务监控'
        metrics_path: /probe
        params:
          module: [http_2xx]
        static_configs:
          - targets:
            - https://news-gg-xy.com/healthz
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: blackbox-exporter.prometheus.svc.cluster.local:9115

      - job_name: blackbox_http_pod_probe
        honor_timestamps: true
        params:
          module:
          - http_2xx
        scrape_interval: 15s
        scrape_timeout: 10s
        metrics_path: /probe
        scheme: http
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_blackbox_scheme]
          separator: ;
          regex: http
          replacement: $1
          action: keep
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_blackbox_port, __meta_kubernetes_pod_annotation_blackbox_path]
          separator: ;
          regex: ([^:]+)(?::\d+)?;(\d+);(.+)
          target_label: __param_target
          replacement: $1:$2$3
          action: replace
        - separator: ;
          regex: (.*)
          target_label: __address__
          replacement: blackbox-exporter.prometheus.svc.cluster.local:9115
          action: replace
        - source_labels: [__param_target]
          separator: ;
          regex: (.*)
          target_label: instance
          replacement: $1
          action: replace
        - separator: ;
          regex: __meta_kubernetes_pod_label_(.+)
          replacement: $1
          action: labelmap
        - source_labels: [__meta_kubernetes_namespace]
          separator: ;
          regex: (.*)
          target_label: kubernetes_namespace
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_pod_name]
          separator: ;
          regex: (.*)
          target_label: kubernetes_pod_name
          replacement: $1
          action: replace
        kubernetes_sd_configs:
        - role: pod


      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        job_name: kubernetes-nodes
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - replacement: kubernetes.default.svc:443
          target_label: __address__
        - regex: (.+)
          replacement: /api/v1/nodes/${1}/proxy/metrics
          source_labels:
          - __meta_kubernetes_node_name
          target_label: __metrics_path__
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true

      - job_name: 'kubernetes-services'
        metrics_path: /probe
        params:
          module: [http_2xx]
        kubernetes_sd_configs:
        - role: service
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
          action: keep
          regex: true
        - source_labels: [__address__]
          target_label: __param_target
        - target_label: __address__
          replacement: blackbox-exporter.prometheus.svc.cluster.local:9115
        - source_labels: [__param_target]
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          target_label: kubernetes_name

      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - replacement: kubernetes.default.svc:443
          target_label: __address__
        - regex: (.+)
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
          source_labels:
          - __meta_kubernetes_node_name
          target_label: __metrics_path__
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true

      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        job_name: 'kubernetes-ingresses'
        metrics_path: /probe
        params:
          module: [http_2xx]
        kubernetes_sd_configs:
        - role: ingress
        relabel_configs:
        - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
          regex: (.+);(.+);(.+)
          replacement: ${1}://${2}${3}
          target_label: __param_target
        - target_label: __address__
          replacement: blackbox-exporter.prometheus.svc.cluster.local:9115
        - source_labels: [__param_target]
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_ingress_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_ingress_name]
          target_label: kubernetes_name

      - job_name: 'kubernetes-kubelet'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics

      - job_name: "traefik"
        static_configs:
        - targets: ['traefik-ingress-service.kube-system.svc.cluster.local:8080']

      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        job_name: kubernetes-apiservers
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: keep
          regex: default;kubernetes;https
          source_labels:
          - __meta_kubernetes_namespace
          - __meta_kubernetes_service_name
          - __meta_kubernetes_endpoint_port_name
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
kind: ConfigMap
metadata:
  labels:
    app: prometheus
  name: prometheus-conf
  namespace: prometheus

三、准备prometheus-rules 配置文件
# cat prometheus-rules.yaml 
apiVersion: v1
data:
  nodedown.rule.yml: |
    groups:
    - name: YingPuDev-Alerting
      rules:
      - alert: 实例崩溃
        expr: up {instance !~""} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "实例{{ $labels.instance }}崩溃"
          description: "{{ $labels.instance }}的{{ $labels.job }}实例已经崩溃超过了1分钟。"

      - alert: HTTP探测发现不健康服务端点
        expr: probe_http_status_code >= 400 or probe_http_status_code {instance !~ "videoai-php-dev.videoai.svc:22"} == 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "API {{ $labels.kubernetes_name }} 服务不可用"
          description: "{{ $labels.kubernetes_name }}的{{ $labels.job }}服务已经超过1分钟不可用了。当前状态码为：{{ $value }}"
kind: ConfigMap
metadata:
  labels:
    app: prometheus
  name: prometheus-rules
  namespace: prometheus

四、准备prometheus-deployment 配置文件
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus
  name: prometheus
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: prometheus
    spec:
      containers:
      - args:
        - --config.file=/etc/prometheus/prometheus.yml
        - --storage.tsdb.path=/prometheus
        - --storage.tsdb.retention=30d
        env:
        - name: STAKATER_PROMETHEUS_CONF_CONFIGMAP
          value: e4dd2838dd54e8392b62d85898083cc3d20210cc
        - name: STAKATER_PROMETHEUS_RULES_CONFIGMAP
          value: ca65a78fcb15d2c767166e468e8e734c6d4e267f
        image: prom/prometheus:latest
        imagePullPolicy: Always
        name: prometheus
        ports:
        - containerPort: 9090
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /prometheus
          name: prometheus-data-volume
        - mountPath: /etc/prometheus/prometheus.yml
          name: prometheus-conf-volume
          subPath: prometheus.yml
        - mountPath: /etc/prometheus/rules
          name: prometheus-rules-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsUser: 0
      serviceAccount: prometheus
      serviceAccountName: prometheus
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - name: prometheus-data-volume
        persistentVolumeClaim:
          claimName: prometheus-server
      - configMap:
          defaultMode: 420
          name: prometheus-conf
        name: prometheus-conf-volume
      - configMap:
          defaultMode: 420
          name: prometheus-rules
        name: prometheus-rules-volume

五、准备 prometheus-svc 配置文件
# cat prometheus-svc.yaml  
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus
  name: prometheus-service
  namespace: prometheus
spec:
  ports:
  - port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: prometheus
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

六、准备 prometheus-ingress 配置文件
# cat prometheus-ingress.yaml  
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus-ingress
  namespace: prometheus
spec:
  rules:
  - host: prometheus.movie.cn
    http:
      paths:
      - backend:
          serviceName: prometheus-service
          servicePort: 9090
status:
  loadBalancer: {}


依次生成配置文件
kubectl apply  -f prometheus-conf.yaml
kubectl apply  -f prometheus-pv-pvc.yaml
kubectl apply  -f prometheus-rules.yaml
kubectl apply  -f prometheus-deployment.yaml
kubectl apply  -f prometheus-svc.yaml
kubectl apply  -f prometheus-ingress.yaml

内容总结

以上是互联网集市为您收集整理的k8s 集群部署prometheus + alertmanager + grafana全部内容，希望文章能够帮你解决k8s 集群部署prometheus + alertmanager + grafana所遇到的程序开发问题。如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。

内容备注

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容，请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。

内容手机端

扫描二维码推送至手机访问。

本文链接：https://qyyshop.com/info/1022007.html

来源：【匿名】

【上一篇】1.20版本k8s简单部署教程【下一篇】关于k8s部署php镜像实例

更多 ►

【k8s 集群部署prometheus + alertmanager + grafana】教程文章相关的互联网学习教程文章

k8s创建完集群之后coredns一直处于pending状态【代码】【图】

按照官网教程https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/，master节点kubectl init,每个从节点kubectl join之后，在master节点执行kubectl get pods -n kube-system，发现coredns一直处于pending状态网上找的资料都是安装一个网络插件，怕他们乱写不敢这样做，最后在官网上找到了答案，地址是：https://kubernetes.io/zh/docs/setup/production-environment/tools/kubea...

K8S(十)——验证kubernetes集群【代码】

一、在一个节点配置资源清单[root@sx7-22 ~]# vim nginx-ds.yamlapiVersion: extensions/v1beta1 kind: DaemonSet metadata:name: nginx-ds spec:template:metadata:labels:app: nginx-dsspec:containers:- name: my-nginximage: harbor.od.com/public/nginx:v1.7.9ports:- containerPort: 80二、配置和检查[root@sx7-22 ~]# kubectl create -f nginx-ds.yaml daemonset.extensions/nginx-ds created [root@sx7-22 ~]# kubectl get...

K8S集群Ingress https实践【代码】【图】

前文介绍使用ingress结合traefik实现了入口的动静分离，本文将在前文基础上实现ingress的https配置。为了简单且高效，建议应用容器化部署之后，https卸载在ingress这一级实现。通俗一点来说就是用户到ingress的连接走https协议，ingress到后端服务的连接走https协议。我们对https的配置要求也比较简单，主要如下：1、http自动重定向到https2、https支持虚拟主机（TLS SNI）一、初始环境准备1、这里为了方便测试，把前文配置的网站动...

使用kubeadm部署k8s集群v1.21【其他版本通用】【代码】【图】

1.服务器要求：建议最小硬件配置：2核/2G/20G 网络需求：最好能访问外网，需要拉取一些必要的镜像；若没有网络条件，需要提前导入必要的镜像2.kubeadm常用工具功能：?kubeadm init：初始化一个Master节点 ?kubeadm join：将工作节点加入集群 ?kubeadm upgrade：升级K8s版本 ?kubeadm token：管理 kubeadm join 使用的令牌 ?kubeadm reset：清空 kubeadm init 或者 kubeadm join 对主机所做的任何更改 ?kubeadm version：打印 kube...

K8s集群内热改代码

1.登录到k8s master服务器$ ssh developer@192.168.3.2482.查看服务容器所在的节点(以wx-server为例)$ kubectl get pods -nwx -o wide3.远程连接所在节点$ ssh <节点>4.筛选出正在运行的容器(以wx-server为例)$ docker ps | grep wx5.进入docker(以wx-server为例)$ docker exec -it <docker container> sh6.修改相关代码vi <文件名>7.重启docker$ docker restart <docker container>注：此处不需要删除原有的k8s pod，直接重启相关...

K8S多集群切换【代码】

Kubeconfig 用途kubectl 命令行工具通过 kubeconfig 文件的配置来选择集群以及集群API Server通信的所有信息。kubeconfig 文件用来保存关于集群用户、命名空间和身份验证机制的信息。默认情况下 kubectl 读取 $HOME/.kube/config 文件，也可以通过设置环境变量 KUBECONFIG 或者 --kubeconfig 指定其他的配置文件。Kubeconfig 文件结构kubeconfig 文件主要由下面几部分构成：集群参数用户参数上下文参数当前上下文apiVersion: v1 ki...

使用kubeadm部署k8s集群00-缓存rpm包【代码】

使用kubeadm部署k8s集群00-缓存rpm包 2018/4/12鉴于部分童鞋不知道如何缓存 rpm 包到本地，本文作为简单指导下述操作在国外节点上操作使用官方 yum 源# mkdir k8s_rpms && cd k8s_rpms # cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/...

kubernetes之3---kubeadm部署k8s集群【代码】

kubeadm部署k8s集群目录kubeadm部署k8s集群环境准备安装docker-ce安装kubeadm使用kubeadm初始化k8s集群master节点配置flannel网络metric-server弹性伸缩StatefulSet 资源Job资源CronJob资源官方文档环境准备主机IP配置软件k8s-adm-master10.0.0.152核2Gdocker-ce，kubelet，kubeadm，kubectlk8s-adm-node110.0.0.162核2Gdocker-ce，kubelet，kubeadm，kubectl关闭：selinux，firewalld和NetworkManager，postfix（非必须）修改IP地...

Centos7搭建k8s集群【代码】【图】

一、部署环境操作系统：CentOS Linux release 7.6.1810 (Core)安装软件：docker：18.06.3-cekubernetes：v1.15.4二、部署架构：部署开始前，先把主机名改成如上图所示。同时修改/etc/hosts，如下：三、部署过程1、安装docker（所有节点）1.1、卸载可能存在的依赖包sudoyum remove docker docker-common container-selinux docker-selinux docker-engine1.2、安装 yum-utils，它提供了 yum-config-manager，可用来管理yum源sudoy...

7，k8s 的etcd集群的搭建【代码】

Etcd 是一个分布式键值存储系统，Kubernetes使用Etcd进行数据存储，所以先准备一个Etcd数据库，为解决Etcd单点故障，应采用集群方式部署，这里使用3台组建集群，可容忍1台机器故障，当然，你也可以使用5台组建集群，可容忍2台机器故障。1 准备cfssl证书生成工具 cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl更方便使用。找任意一台服务器操作，这里用Master节点。 wget https://pkg.cfssl.org/R1.2/cfssl_li...

生产环境二进制k8s集群扩容node节点的实践

K8s二进制生产环境扩容node节点由于项目微服务也是部署在k8s集群中去维护的，所以扩容node节点也是必要的联系，扩容node节点一定要保证你整个集群的容器环境的网络都是互通的，这也是很重要的一步，这里我根据自己的经验去扩容，仅供参考首先我这里是安装的二进制方式去部署的k8s集群，进行扩容node的时候，也是非常方便的扩容node节点分为两步，第一步先将我们旧的node节点上的配置先去拷贝到我们新的节点上，第二点就是将我们的容...

十五，K8S集群调度原理及调度策略【代码】【图】

目录k8s调度器SchedulerScheduler工作原理请求及Scheduler调度步骤：k8s的调用工作方式常用预选策略常用优先函数节点亲和性调度节点硬亲和性节点软亲和性Pod资源亲和调度Pod硬亲和调度Pod软亲和调度Pod反亲和调度污点和容忍度定义污点和容忍度effect定义排斥等级：在Pod上定义容忍度时：管理节点的污点k8s调度器SchedulerScheduler工作原理请求及Scheduler调度步骤：节点预选(Predicate)：排除完全不满足条件的节点，如内存大小，端...

lvs+keepalived部署k8s v1.16.4高可用集群【代码】【图】

一、部署环境1.1 主机列表主机名Centos版本ipdocker versionflannel versionKeepalived version主机配置备注lvs-keepalived017.6.1810172.27.34.28//v1.3.54C4Glvs-keepalivedlvs-keepalived017.6.1810172.27.34.29//v1.3.54C4Glvs-keepalivedmaster017.6.1810172.27.34.3518.09.9v0.11.0/4C4Gcontrol planemaster027.6.1810172.27.34.3618.09.9v0.11.0/4C4Gcontrol planemaster037.6.1810172.27.34.3718.09.9v0.11.0/4C4Gcontrol ...

K8S实践Ⅷ（HA集群部署）【代码】

一、环境准备 1.集群规划主机名IP角色VIP20.0.20.200master-VIPk8s-master0120.0.20.201masterk8s-master0220.0.20.202masterk8s-master0320.0.20.203masterk8s-node0120.0.20.204nodek8s-node0220.0.20.205nodek8s-node0320.0.20.206node2.基础环境配置关闭防火墙关闭selinux配置hosts配置master之间的ssh免密登录关闭swap分区sed -i ‘/ swap / s/^$.*$$/#\1/g‘ /etc/fstab配置时钟同步systemctl start chronyd systemctl ena...

Rancher2.0构建kubernetes(K8S)集群【代码】【图】

一、环境准备 1.准备至少3台CentOS7版本的虚拟机# IP地址主机名称192.168.1.160 rancher192.168.1.161 master 192.168.1.162 node1 192.168.1.163 node2 192.168.1.164 node3 2.每台虚拟机上需要做的配置1.安装Docker$ yum install -y docker 2.设置hostname$ hostnamectl --static set-hostname rancher #根据下面名称进行对应配置 3.设置host$ vim /etc/hosts# 粘贴配置192.168.1.160 rancher192.168.1.161 master...

首页 / K8S / k8s 集群部署prometheus + alertmanager + grafana

k8s 集群部署prometheus + alertmanager + grafana

内容导读

内容图文

准备k8s 集群

部署prometheus

内容总结

内容备注

内容手机端

【k8s 集群部署prometheus + alertmanager + grafana】教程文章相关的互联网学习教程文章

K8S - 相关标签

集群 - 相关标签

K8S - 最新教程

K8S - 最热教程