|
Getting started | Prometheus
Configuration | PrometheusDownload | PrometheusDownload Grafana | Grafana Labs - # prometheus<br>mkdir -m=777 -p /data/{download,app_logs,app/prometheus}
- cd /data/download
- wget https://github.com/prometheus/prometheus/releases/download/v2.45.0-rc.0/prometheus-2.45.0-rc.0.linux-amd64.tar.gz
- tar xvfz prometheus-*.tar.gz<br>ln -s /data/download/prometheus-2.45.0-rc.0.linux-amd64/prometheus /usr/bin/prometheus<br>cp /data/download/prometheus-2.45.0-rc.0.linux-amd64/prometheus.yml /data/app/prometheus/prometheus.yml <br>prometheus --config.file=/data/app/prometheus/prometheus.yml --web.listen-address=:9090 --web.enable-lifecycle --storage.tsdb.path=/data/app/prometheus/data >>/data/app_logs/prometheus.log 2>&1 &<br><br># node_exporter 在需要监控的服务器里安装<br>mkdir -m=777 -p /data/{download,app_logs,app/prometheus}<br>cd /data/download<br>wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz<br><em>tar xvfz node_exporter*<br></em>ln -s /data/download/node_exporter-1.6.0.linux-amd64/node_exporter /usr/bin/node_exporter<br># 启动node_exporter,服务器暴露的端口是8080,同时服务器里有其他服务占用了8080端口,可以使用nginx将node_exporter获取指标的api暴露出去<br># location /metrics {<br># proxy_pass http://127.0.0.1:9000/metrics;<br># }<br>node_exporter --web.listen-address 127.0.0.1:9000 >>/data/app_logs/node_exporter.log 2>&1 &<br><em><em><em># 添加node_exporter之后,需要更新prometheus.xml添加targets,然后运行:curl -X PUT http://server_address:port/-/reload重新加载配置文件<br><br># </em></em></em><em>alert_manager可以和prometheus安装到同一台服务器<br></em>cd /data/download<br>wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz<br>tar xvfz alertmanager*<br>ln -s /data/download/alertmanager-0.25.0.linux-amd64/alertmanager /usr/bin/alertmanager<br>cp /data/download/alertmanager-0.25.0.linux-amd64/alertmanager.yml /data/app/prometheus/alertmanager.yml<br>alertmanager --config.file=/data/app/prometheus/alertmanager.yml --web.listen-address 127.0.0.1:9001 >>/data/app_logs/node_exporter.log 2>&1 &<br># 将alert_manager的地址添加到prometheus.yml里的alertmanagers的targets里,然后<em>运行:curl -X PUT http://server_address:port/-/reload重新加载配置文件</em>
复制代码 测试报警邮件功能:设置如果安装exporter的服务器内存占用率超过50%或者tcp timewait超过10的时候就发邮件(在实际工作中需要设置一个合适的条件):
prometheus.yml里添加rule_files的路径:- # my global config
- global:
- scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
- evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
- # scrape_timeout is set to the global default (10s).
- # Alertmanager configuration
- alerting:
- alertmanagers:
- - static_configs:
- - targets:
- - 127.0.0.1:9001
- # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
- rule_files:
- # - "first_rules.yml"
- # - "second_rules.yml"
- - "/data/app/prometheus/alert.rules.yml"
- # A scrape configuration containing exactly one endpoint to scrape:
- # Here it's Prometheus itself.
- scrape_configs:
- # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- - job_name: "prometheus"
- # metrics_path defaults to '/metrics'
- # scheme defaults to 'http'.
- scrape_interval: 5s
- static_configs:
- - targets: ["node1_ip:8080"]
- - targets: ["node2_ip:8080"]
- labels:
- groups: 'container'
复制代码 alert.rules.yml里添加具体的rule,node_socket_TCP_tw这些具体的指标通过http://node_exporter_ip:port/metrics可以获取到- groups:
- - name: tcp-alert-group
- rules:
- - alert: TcpTimeWait
- expr: node_sockstat_TCP_tw > 10
- for: 10m
- labels:
- severity: warning
- annotations:
- summary: tcp time wait more than 10
- description: please check node_sockstat_TCP_tw metric
- - alert: MemoryUse
- expr: (node_memory_MemTotal_bytes-node_memory_MemFree_bytes-node_memory_Buffers_bytes-node_memory_Cached_bytes)/node_memory_MemTotal_bytes > 0.5
- for: 10m
- labels:
- severity: warning
- annotations:
- summary: memory use more than 50% for 10 min
- description: please check memory use
复制代码 alertmanager.yml里配置告警邮件的信息:- global:
- resolve_timeout: 5m
- smtp_smarthost: your_smpt_host:port
- smtp_from: alertmanager@your_email_domain
- smtp_require_tls: false
- route:
- group_by: ['alertname']
- group_wait: 30s
- group_interval: 5m
- repeat_interval: 10m
- receiver: 'email'
- receivers:
- - name: 'email'
- email_configs:
- - to: 'receiver_email'
- send_resolved: true
复制代码 yml文件一旦更新,需要重新加载配置:curl -X PUT http://server_address:port/-/reload
在Prometheus的界面可以看到添加的alert:
当alert的条件满足后,alertmanager就会发邮件
grafana的安装和启动:- # grafana可以和prometheus里安装到同一台服务器
- yum install -y https://dl.grafana.com/enterprise/release/grafana-enterprise-10.0.0-1.x86_64.rpm
- # grafana默认启动的端口号是3000,如果服务器没有暴露3000端口的话,需要修改grafana的配置文件
- sed -i 's/3000/8080/g' /usr/share/grafana/conf/defaults.ini
- grafana server >> /data/app_logs/grafana.log 2>&1 &
- # grafana数据保存地址:/var/lib/grafana.db
复制代码 grafana启动之后就可以在浏览器上打开对应的地址,初次登录用户名和密码:admin/admin
Data sources里添加prometheus,grafana和prometheus启动在同一台服务器里的话,地址就可以用localhost
添加dashboard,在Explore里可以查询指标并且添加到dashboard
cpu使用率:avg(1-irate(node_cpu_seconds_total{mode="idle"}[1m])) by(instance)
内存使用率:(node_memory_MemTotal_bytes-node_memory_MemFree_bytes-node_memory_Buffers_bytes-node_memory_Cached_bytes)/node_memory_MemTotal_bytes
tcp连接数:node_sockstat_TCP_alloc
dashboard:
注意点:- 1.prometheus启动的时候添加--web.enable-lifecycle才允许通过调用/-/reload接口重新加载配置文件<br>2.prometheus启动的时候指定一个固定的数据存放位置--storage.tsdb.path=/data/app/prometheus/data,如果数据存放位置不一致,启动后查不到历史数据,历史数据做备份的话,prometheus启动的服务器还可以变更<br>3.grafana的数据保存地址:/var/lib/grafana.db,定期做备份,服务器发生系统错误无法使用的时候,在新的服务器里同步/var/lib/grafana.db文件之后,启动grafana之前的配置不会丢失
复制代码 来源:https://www.cnblogs.com/huizit1/p/17492003.html
免责声明:由于采集信息均来自互联网,如果侵犯了您的权益,请联系我们【E-Mail:cb@itdo.tech】 我们会及时删除侵权内容,谢谢合作! |
本帖子中包含更多资源
您需要 登录 才可以下载或查看,没有账号?立即注册
x
|