使用Curator清理es时间索引

前言
​ Curator 是 Elastic 官方发布的一个管理 Elasticsearch 索引的工具，可以完成许多索引生命周期的管理工作，例如清理创建时间超过7天的索引、每天定时备份指定的索引、定时将索引从热节点迁移至冷节点等等。

官网
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

安装
通过pip 安装
pip install elasticsearch-curator -i https://pypi.douban.com/simple
通过yum 安装

添加yum源，centos7为例

[curator-5]
name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages
baseurl=https://packages.elastic.co/curator/5/centos/7
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

安装

yum clean all
yum makecache fast
yum install elasticsearch-curator -y
离线安装
rpm -ivhU https://packages.elastic.co/curator/5/centos/7/Packages/elasticsearch-curator-5.8.4-1.x86_64.rpm
安装完会生成 2 个工具

curator # 主角
curator_cli
cli 主要用与调试， 写法比较反人类，不推荐使用，主要用 curator 工具

使用方法
curator 以配置文件的形式运行

这2个文件，叫什么名字，放在什么地方无关紧要，但运行时需制定绝对路径。

curator --config=/etc/curator/config.yml /etc/curator/action.yml
主要就是两个配置文件，

config.yml
主要配置es 的连接信息

使用时清理掉中文， 如果es 没有账号、密码，直接添加hosts 和port 即可，

多个es 地址，使用, 分割。

---

Remember, leave a key empty if there is no value. None will be a string,

not a Python “NoneType”

client:
hosts:
- 10.23.20.157，10.23.20.155
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False

logging:
loglevel: INFO
logfile: /var/log/curator.log
logformat: default
blacklist: [‘elasticsearch’, ‘urllib3’]
action.yml
此文件主要用于执行连接es 之后的执行动作

通常适用于以时间来区分的索引。

详情见：https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/actions.html

本案例只测试了按照时间格式清理索引。

---

Remember, leave a key empty if there is no value. None will be a string,

not a Python “NoneType”

Also remember that all examples have ‘disable_action’ set to True. If you

want to use this action as a template, be sure to set this to False after

copying it.

actions:
1:
action: delete_indices
description: >-
delete indices over 7 days # 此action 主要匹配 containers- 开头名称中有 %Y-%m-%d 格式的索引，超过7天的索引则被清理掉，
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: containers-
- filtertype: age
source: name
direction: older
timestring: ‘%Y-%m-%d’
unit: days
unit_count: 7
2:
action: delete_indices
description: >-
delete indices over 30 days
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
- filtertype: age
source: name
direction: older
timestring: ‘%Y-%m-%d’
unit: days
unit_count: 30
运行
运行很简单

curator --config=/etc/curator/config.yml /etc/curator/action.yml
定时运行

*/10 * * * * curator --config=/etc/curator/config.yml /etc/curator/action.yml
实战
docker运行es
docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e “discovery.type=single-node” harbor.ats.io/docker.io/library/elasticsearch:6.8.15
检测es状态
[root@dce-10-23-20-157 opt]# curl 127.0.0.1:9200
{
“name” : “8DZoKE3”,
“cluster_name” : “docker-cluster”,
“cluster_uuid” : “-nqhHRPkSGSSNBtfR1rJFw”,
“version” : {
“number” : “6.8.15”,
“build_flavor” : “default”,
“build_type” : “docker”,
“build_hash” : “c9a8c60”,
“build_date” : “2021-03-18T06:33:32.588487Z”,
“build_snapshot” : false,
“lucene_version” : “7.7.3”,
“minimum_wire_compatibility_version” : “5.6.0”,
“minimum_index_compatibility_version” : “5.0.0”
},
“tagline” : “You Know, for Search”
}
创建索引
for i in seq 1 30; do curl -X PUT 127.0.0.1:9200/containers-2021-10-$i; done
修改config.yml 配置
上面已列出，此处省略。

查看es 索引
[root@dce-10-23-20-157 clean_es]# curl 127.0.0.1:9200/_cat/indices | sort -k 3
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4543 100 4543 0 0 77156 0 --:–:-- --:–:-- --:–:-- 77000
yellow open containers-2021-10-10 -AFnOlBzT-eRM2BAoTttmQ 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-11 s_YYL13URCy7-BTWIhDHEw 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-12 ZoUFGVuRSOSoz-ZkzuYLcA 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-13 8iJ67LOCSSijNyNpswJXIw 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-14 tYsrR66wQj2CjnTDn9OHcQ 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-15 6OsVTMuvSf6KLx0vo21L9Q 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-16 qRaeW12iT3yvpGJIzk1vGw 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-17 6IT3G9wxTGmk2SkgsAmU1Q 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-1 7th-wticRB2sMPg6Vu2fOg 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-18 qHuZAuoWRoCj9V5x4sDiLA 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-19 af9yc-I2T8e4olqoC-RvbQ 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-20 hHuwYLX9Q7KTOVtk5Bn5hQ 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-21 Yb1z9r8-SrynNAnqitgtuQ 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-22 Y9hGZiE_SIWOxvaka1h2yw 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-23 nJzWSnMXQ0q6u06evFZzDg 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-24 00uc0SunSXmiV1gawgLceA 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-25 hYlHwyfmR0CRR1-_CttrgA 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-26 RPCuVN38QbC11UIp6ge0xA 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-27 kusZdEcFRUSIMjY3fTl7MQ 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-28 N9Avyxf-RX-5eiRJwAvkpQ 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-29 Rd4j19iCSVyNYIxa4FunDA 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-2 VlYD9NflSfupe4PXFXuSbQ 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-30 yRLAhGsqTOG7Jb_N9pP9-g 5 1 0 0 1.1kb 1.1kb
yellow open containers-2021-10-3 JKmEvJ0YSam60fXAePCnSQ 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-4 F9UmkVaCTsWnNTMR_O9FbQ 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-5 -Gd-a8q8TqiRANNorG_Eiw 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-6 ecri1g7XSIW67cDRfMW_Bw 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-7 rqsLUxN5R_OO3G3kft15og 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-8 GwhbKTh4QGG0Ql3q8GYuXg 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-9 4UI3D4WtQ9KfVdaglORLAQ 5 1 0 0 1.2kb 1.2kb
清理日志
[root@dce-10-23-20-155 opt]# curator --config=/opt/config.yaml /opt/action.yaml
查看结果
[root@dce-10-23-20-157 clean_es]# curl 127.0.0.1:9200/_cat/indices | sort -k 3
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2926 100 2926 0 0 43108 0 --:–:-- --:–:-- --:–:-- 44333
yellow open containers-2021-10-1 7th-wticRB2sMPg6Vu2fOg 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-2 VlYD9NflSfupe4PXFXuSbQ 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-3 JKmEvJ0YSam60fXAePCnSQ 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-4 F9UmkVaCTsWnNTMR_O9FbQ 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-5 -Gd-a8q8TqiRANNorG_Eiw 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-6 ecri1g7XSIW67cDRfMW_Bw 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-7 rqsLUxN5R_OO3G3kft15og 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-8 GwhbKTh4QGG0Ql3q8GYuXg 5 1 0 0 1.2kb 1.2kb
yellow open containers-2021-10-9 4UI3D4WtQ9KfVdaglORLAQ 5 1 0 0 1.2kb 1.2kb
为啥 1-9 的没有删除掉， 查看日志

Index “containers-2021-10-1” does not meet provided criteria. Removing from list.
规则不匹配，检测%Y-%m-%d

[root@dce-10-23-20-157 ~]# date +%Y-%m-%d
2021-11-15
[root@dce-10-23-20-157 ~]# date -d “10 days ago” +%Y-%m-%d
2021-11-05
原来是测试案例问题，导致。

直接重建es， 再重新批量创建输入，

[root@dce-10-23-20-157 ~]# for i in seq 1 30; do curl -X PUT 127.0.0.1:9200/containers-2021-10-$(printf “%02d\n” $i); done
{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-01”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-02”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-03”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-04”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-05”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-06”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-07”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-08”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-09”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-10”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-11”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-12”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-13”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-14”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-15”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-16”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-17”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-18”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-19”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-20”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-21”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-22”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-23”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-24”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-25”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-26”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-27”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-28”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-29”}{“acknowledged”:true,“shards_acknowledged”:true,“index”:“containers-2021-10-30”}
再此测试 curator 命令

[root@dce-10-23-20-155 opt]# curator --config=/opt/config.yaml /opt/action.yaml
查看es 索引情况

[root@dce-10-23-20-157 clean_es]# curl 127.0.0.1:9200/_cat/indices | sort -k 3
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
测试已发现 索引 已全部清理完成。

结论
使用 curator 比较方便的就完成了对es 索引的清理。使用逻辑比较清晰，简洁。