背景介绍
业务原有ES
集群因证书到期,需要进行证书更换操作。由于更新证书后需重启整个集群后才能生效,且业务敏感度较高,无法容忍整个集群停止对外服务。
解决方案
-
OP
负责部署一套新ES
集群,版本配置同原集群一致 -
RD
负责开发程序- 将历史数据导入新集群
- 业务服务实现双写两套
ES
集群功能
-
两套集群数据一致后,
OP
和RD
配合切换业务流量 -
OP
将原有集群更新证书
ElasticSearch 集群
准备
ES
版本 5.5.1- Search Guard
JAVA
版本 jdk1.8.0_161
集群配置
3台 Master Node (56核/128G/3.7T普通硬盘)
10.90.104.133
10.90.105.133
10.90.106.133
5台 Data Node (56核/128G/3.7T普通硬盘)
10.90.107.132
10.90.108.132
10.90.109.133
10.90.110.133
10.90.111.133
端口: 9201/9301
复制代码
系统调整
临时生效
$ sudo sysctl -w vm.max_map_count=262144
永久生效
$ grep vm.max_map_count /etc/sysctl.conf
$ echo vm.max_map_count=262144 >> /etc/sysctl.conf
or
$ vi /etc/sysctl.conf
vm.max_map_count=262144
复制代码
安装JAVA
下载 jdk1.8.0_161(若链接404,自行注册账号下载)
$ wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz
创建目录, 解压文件
$ mkdir -p /usr/local/java/jdk1.8.0_161/
$ cp jdk-8u161-linux-x64.tar.gz /usr/local/java/jdk1.8.0_161/
$ cd /usr/local/java/jdk1.8.0_161/
$ tar zxf jdk-8u161-linux-x64.tar.gz && rm -rf jdk-8u161-linux-x64.tar.gz
安装 java,使用 alternatives 切换默认的 java 命令 (command --install <link> <name> <path> <priority>)
$ alternatives --install /usr/bin/java java $JAVA_18_161/bin/java 2
$ alternatives --config java
There are 4 programs which provide 'java'.
Selection Command
-----------------------------------------------
1 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64/jre/bin/java
*+ 2 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64/jre/bin/java
3 /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
4 /usr/local/java/jdk1.8.0_161/bin/java
Enter to keep the current selection[+], or type selection number: 4
安装 jar , javac
alternatives --install /usr/bin/jar jar /usr/local/java/jdk1.8.0_161/bin/jar 2
alternatives --install /usr/bin/javac javac /usr/local/java/jdk1.8.0_161/bin/javac 2
alternatives --set jar /usr/local/java/jdk1.8.0_161/bin/jar
alternatives --set javac /usr/local/java/jdk1.8.0_161/bin/javac
设置环境变量
$ vim /etc/bashrc
export JAVA_HOME=/usr/local/java/jdk1.8.0_161
export JRE_HOME=/usr/local/java/jdk1.8.0_161/jre
export PATH=/root/perl5/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/java/jdk1.8.0_161/bin:/usr/local/java/jdk1.8.0_161/jre/bin
查看java版本
$ java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
复制代码
安装ElasticSearch
创建用户
$ adduser -U -s /sbin/nologin elasticsearch
创建目录
$ mkdir -p /usr/local/elasticsearch
$ mkdir -p /data/elasticsearch/my-project
$ mkdir -p /data/logs/elasticsearch/my-project/{log,pid}
下载es 5.5.1
$ cd /usr/local/elasticsearch
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.tar.gz
解压文件
$ tar zxf elasticsearch-5.5.1.tar.gz && rm -rf elasticsearch-5.5.1.tar.gz
$ mv elasticsearch-5.5.1 my-project-es
复制代码
安装Search Guard
插件
下载插件
$ cd {PROJECT_NAME}
$ wget http://search.maven.org/remotecontent?filepath=com/floragunn/search-guard-5/5.5.1-15/search-guard-5-5.5.1-15.zip
安装插件
$ bin/elasticsearch-plugin install search-guard-5-5.5.1-15.zip
复制代码
生成证书
下载工具(路径没有特殊要求)
$ git clone https://github.com/floragunncom/search-guard-ssl.git
生成证书
$ cd search-guard-ssl/example-pki-scripts/
$ ./clean.sh
$ ./gen_root_ca.sh kubernetes- kubernetes- # COMMAND <CA_PASS> <TS_PASS>
$ ./gen_node_cert.sh k8s-es-node kubernetes- kubernetes- # COMMAND <NODE_NAME> <KS_PASS> <CA_PASS>
$ ./gen_client_node_cert.sh sgadmin kubernetes- kubernetes- # COMMAND <CLIENT_NAME> <KS_PASS> <CA_PASS>
拷贝证书到 config 目录下
$ cp *jks /usr/local/elasticsearch/my-project/config/
复制代码
配置ElasticSearch
备份配置
$ cd /usr/local/elasticsearch/my-project/config/
$ mv elasticsearch.yml{,bak}
Node Master (三台master都按此配置):
主配置: 不做存储,不开启 http
$ vim elasticsearch.yml
cluster.name: my-project-es
node.name: 10.90.104.133
node.master: true
node.data: false
node.ingest: false
path:
data:
- /data/elasticsearch/my-project
logs: /data/logs/elasticsearch/my-project/log
thread_pool.index.queue_size: 1000
thread_pool.bulk.queue_size: 1000
bootstrap.memory_lock: true
network.host: 10.90.104.133
http.port: 9201
transport.tcp.port: 9301
http:
enabled: false
compression: true
cors:
enabled: true
allow-origin: "*"
allow-headers: Authorization
discovery.zen.ping.unicast.hosts: [10.90.104.133:9301,10.90.105.133:9301,10.90.106.133:9301]
discovery.zen.minimum_master_nodes: 2
searchguard.ssl.transport.keystore_filepath: node-k8s-es-node-keystore.jks
searchguard.ssl.transport.keystore_password: kubernetes-
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: kubernetes-
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
- CN=sgadmin,OU=client,O=client,L=test,C=DE
内存配置:
$ vim jvm.options
...
Xms20g
Xmx20g
...
Node Data (后5台数据节点都按此配置):
主配置: 存储数据并开启 http 访问
$ vim elasticsearch.yml
cluster.name: my-project-es
node.name: 10.90.107.132
node.master: false
node.data: true
node.ingest: true
path:
data:
- /data/elasticsearch/my-project
logs: /data/logs/elasticsearch/my-project/log
thread_pool.index.queue_size: 1000
thread_pool.bulk.queue_size: 1000
bootstrap.memory_lock: true
network.host: 10.90.107.132
http.port: 9201
transport.tcp.port: 9301
http:
enabled: true
compression: true
cors:
enabled: true
allow-origin: "*"
allow-headers: Authorization
discovery.zen.ping.unicast.hosts: [10.90.104.133:9301,10.90.105.133:9301,10.90.106.133:9301]
discovery.zen.minimum_master_nodes: 2
searchguard.ssl.transport.keystore_filepath: node-k8s-es-node-keystore.jks
searchguard.ssl.transport.keystore_password: kubernetes-
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: kubernetes-
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
- CN=sgadmin,OU=client,O=client,L=test,C=DE
内存配置:
$ vim jvm.options
...
Xms63g
Xmx63g
...
复制代码
用户权限及密码
$ cd /usr/local/elasticsearch/my-project/plugins/search-guard-5
生成密码
$ sh tools/hash.sh -p ICdsy1B3j68Tr4w2
$2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
修改用户默认密码, 添加用户 myuser
$ vim sgconfig/sg_internal_users.yml
# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
admin:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
logstash:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
kibanaserver:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
kibanaro:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
roles:
- kibanarole
readall:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
#password is: ICdsy1B3j68Tr4w2
myuser:
hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
# password is: ICdsy1B3j68Tr4w2
修改角色映射文件(分配admin权限)
$ vim sgconfig/sg_roles_mapping.yml
...
# 仅将新用户添加到该权限下,其他配置保持默认
sg_all_access:
users:
- admin
- myuser
...
复制代码
添加服务管理脚本
$ vim /usr/lib/systemd/system/es-my-project.service [Unit] Description=Elasticsearch Documentation=http://www.elastic.co Wants=network-online.target After=network-online.target [Service] Environment=ES_HOME=/usr/local/elasticsearch/my-project Environment=CONF_DIR=/usr/local/elasticsearch/my-project/config Environment=PID_DIR=/data/logs/elasticsearch/my-project/pid WorkingDirectory=/usr/local/elasticsearch/my-project User=elasticsearch Group=elasticsearch ExecStartPre=/usr/local/elasticsearch/my-project/bin/elasticsearch-systemd-pre-exec ExecStart=/usr/local/elasticsearch/my-project/bin/elasticsearch \ -p \${PID_DIR}/elasticsearch.pid \ -Edefault.path.conf=\${CONF_DIR} StandardOutput=journal StandardError=inherit LimitNOFILE=65536 LimitNPROC=2048 LimitMEMLOCK=infinity TimeoutStopSec=0 KillSignal=SIGTERM KillMode=process SendSIGKILL=no SuccessExitStatus=143 [Install] WantedBy=multi-user.target 复制代码
修改用户
chown -R elasticsearch:elasticsearch /usr/local/elasticsearch
chown -R elasticsearch:elasticsearch /data/logs/elasticsearch
chown -R elasticsearch:elasticsearch /data/elasticsearch
chown -R elasticsearch:elasticsearch /usr/lib/systemd/system/es-my-project.service
复制代码
启动服务
添加开机自启
sudo systemctl daemon-reload
sudo systemctl enable es-my-project.service
sudo systemctl start es-my-project
复制代码
sguard 初始化密码
sguard配置生效,只需要在集群中的任意节点执行即可(推荐固定一台master操作)
更改sgconfig配置就需要执行本脚本,执行过程中不会重启ES集群。
$ cd /usr/local/elasticsearch/my-project/plugins/search-guard-5/tools
$ ./sgadmin.sh --hostname 10.90.104.133 --port 9301 \
-cd ../sgconfig/ \
-ks /usr/local/elasticsearch/my-project/config/sgadmin-keystore.jks \
-kspass kubernetes- \
-ts ./usr/local/elasticsearch/my-project/config/truststore.jks \
-tspass kubernetes- \
-nhnv --diagnose -cn my-project-es
复制代码
查看集群状态
$ curl -u myuser:ICdsy1B3j68Tr4w2 -XGET http://10.90.107.132:9201/_cluster/health?pretty
{
"cluster_name" : "my-project-es",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 4,
"active_shards" : 9,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
复制代码
Kibana 部署
Dockerfile
FROM docker.xxx.com/library/kibana-searchguard:5.5.1
ENV CLUSTER_NAME=my-project-es \
SERVER_BASEPATH= \
ELASTICSEARCH_URL="http://10.90.107.132:9201" \
XPACK_SECURITY_ENABLED=false \
XPACK_GRAPH_ENABLED=false \
XPACK_ML_ENABLED=false \
ELASTICSEARCH_USERNAME=kibanaserver \
ELASTICSEARCH_PASSWORD=ICdsy1B3j68Tr4w2
# port is 5601
复制代码
基于内部仓库镜像构建
构建镜像+生成容器
$ docker build -t my-project-kibana .
$ docker run -d my-project-kibana
复制代码
登录 kibana
进行初始化设置
更换证书
新生成的证书文件*.jks
拷贝到原集群config
下,重启ES
集群和search guard
即可。