Docker 、jhipster-console 快速elk日志搭建

Ubuntu 14.04 16.04 (使用apt-get进行安装)
血泪教训 : 不要用fish 好多权限拿不到，不解析。
step 0:卸载旧版本docker
全新安装时，无需执行该步骤

sudo apt-get remove docker docker-engine docker.io

step 1: 安装必要的一些系统工具

sudo apt-get update
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common

step 2: 安装GPG证书

sudo curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

Step 3: 写入软件源信息

sudo add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

Step 4: 更新并安装 Docker-CE
sudo apt-get -y update
sudo apt-get -y install docker-ce

安装指定版本的Docker-CE:

# Step 1: 查找Docker-CE的版本:
# sudo apt-cache madison docker-ce
#   docker-ce | 17.03.1~ce-0~ubuntu-xenial | http://mirrors.aliyun.com/docker-ce/linux/ubuntu xenial/stable amd64 Packages
#   docker-ce | 17.03.0~ce-0~ubuntu-xenial | http://mirrors.aliyun.com/docker-ce/linux/ubuntu xenial/stable amd64 Packages
# Step 2: 安装指定版本的Docker-CE: (VERSION 例如上面的 17.03.1~ce-0~ubuntu-xenial)
# sudo apt-get -y install docker-ce=[VERSION]

Step 5: 安装校验

root@aaa:$ sudo docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 07:52:04 2017
 OS/Arch:      linux/amd64
 
Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 07:52:04 2017
 OS/Arch:      linux/amd64
 Experimental: false

1

参考地址：

https://yq.aliyun.com/articles/110806

二、docker-compose安装
转载地址： https://blog.csdn.net/nimoyaoww/article/details/79155489
1、下载docker-compose

sudo curl -L https://github.com/docker/compose/releases/download/1.17.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

2、授权

sudo chmod +x /usr/local/bin/docker-compose

3、查看版本信息

docker-compose --version

显示出版本信息，即安装成功。
docker-compose命令执行错误的解决（Couldn’t connect to Docker daemon at http+unix://var/run/docker.sock）
在命令行中直接执行docker-compose时可能会报出以下错误：

[exec] Couldn’t connect to Docker daemon at http+unix://var/run/docker.sock - is it running?
[exec]
[exec] If it’s at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
[exec] Result: 1

解决方法：

sudo vin /etc/default/docker

修改下面的参数：

DOCKER_OPTS="-H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock"

问题描述：
doker启动时，报错：docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused “exit status 23"”: unknown.

环境：Ubuntu 14.04

原因
docker的版本和linux的内核版本不兼容
解决办法

升级linux内核，执行下列命令

sudo apt-get install --install-recommends linux-generic-lts-xenial

注意，更新了内核后，需要重启系统。

reboot

重启docker服务：

sudo service docker restart

安装JHipster Console
下载JHipster Console的docker-compose文件：

sudo curl -O https://raw.githubusercontent.com/jhipster/jhipster-console/master/bootstrap/docker-compose.yml

启动：

 sudo docker-compose up -d

停止：

 sudo docker-compose stop

需要注意的是JHipster Console几个相关的项目之间是互相调用的，采用官方的docker-compose文件启动默认是可以互相调用的，如果有问题一般出现在系统上其他服务占用了端口。或者自己调整地址和端口后修改一下配置文件即可。主要的服务和端口有：

jhipster-elasticsearch

es服务端口：9200
jhipster-logstash 

logstash API 端口：9600
tcp 日志收集端口：5000
http 日志收集端口：5001
jhipster-console

web访问端口：5601
zipkin

web访问端口：9411
配置JHipster Console

jhipster:
    metrics:
        jmx.enabled: true
        graphite: # Send metrics to a Graphite server
            enabled: true
            host: localhost
            port: 2003
            prefix: jhipster
        prometheus: # Expose Prometheus metrics on the /prometheusMetrics endpoint
            enabled: true
            endpoint: /prometheusMetrics

root@ubuntu1404:/data# sudo docker-compose up -d
WARNING: The state variable is not set. Defaulting to a blank string.
Starting data_jhipster-alerter_1 ...
Starting data_jhipster-curator_1 ...
Starting data_jhipster-logstash_1 ...
Starting data_jhipster-elasticsearch_1 ...
Starting data_jhipster-zipkin_1 ...
Starting data_jhipster-curator_1 ... error

回出现两个错误 从问题应该是代码拉去 是国外问题，不能完全成功，还有就是docker 于ubuntu 版本不一致问题，目前不影响elk 日志使用 暂时没找到好的解决方式，后期回更新。

ERROR: for data_jhipster-curator_1  Cannot start service jhipster-curator: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:297: copying bootstrap data to pipe Starting data_jhipster-alerter_1 ... error

ERROR: for data_jhipster-alerter_1  Cannot start service jhipster-alerter: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for Starting data_jhipster-console_1 ... error

docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/6602807a7f1a03689a847741b0a10171788f7bed2aa92a4dff35c429b02248d1/log.json: no such file or directory): docker-runc did not terminate sucessfully: docker-runc: symbol lookup error: docker-runc: undefined symbol: seccomp_version

解决方法：

升级系统到7.4，即可解决

执行yum update，时间比较长，耐心等待。

[root@bogon ~]# cat /etc/redhat-release

CentOS Linux release 7.4.1708 (Core)

[root@bogon ~]# docker run -it centos echo haha

还有一个解决方法时安装低版本的docker，docker与操作系统的版本依赖性特别强，在安装时务必要注意！

外国人是这样解释的：

RHEL 7.2 is no longer supported and Docker does not test their release against unsupported versions. You need to upgrade your OS, preferably to 7.4, to resolve this issue.

I’ve seen a few people with similar issues in the past few days that resolved this same error message by upgrading their OS. Some of the older packages no longer work with newer docker releases. One of those problems is an outdated libseccomp: https://github.com/moby/moby/issues/35906

日常维护 数据脱敏

系统日志过长自动截取，修改 logstash 配置文件
1.进入容器

 sudo docker exec -it 容器ID /bin/bash

2.文件位置

 cd /usr/share/logstash/pipeline

3.编辑文件

 vi logstash.conf

找到过滤器filter最下面加入ruby 代码块

filter {
 ruby {
        code => "
        msg = event.get('message'); # 获取 message
        count = 20000;
        REPLACE ='******'
        MOBILE_PATTERN = /(13[0-9]|14[01456879]|15[0-3,5-9]|16[2567]|17[0-8]|18[0-9]|19[0-3,5-9])\d{8}/; # 手机号正则
        EMAIL_PATTERN = /([A-Za-z0-9_\-\.])+\@([A-Za-z0-9_\-\.])+\.([A-Za-z]{2,4})/; # 邮箱正则
        INTERNET_URL= /((http|ftp|https|sftp):\/\/[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:~+#])?)/; # 域名
        INTERNET_ALl_URL= /(http|ftp|https):\/\/[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:\/~\+#]*[\w\-\@?^=%&\/~\+#])?/; # 网址
        ID_CARD =/(\d{6})(\d{4})(\d{2})(\d{2})(\d{3})([0-9]|X)/; # 身份证号
        ID =/[1-8][1-7]\d{4}(?:19|20)\d{2}(?:0[1-9]|1[0-2])(?:0[1-9]|[12]\d|3[01])\d{3}[\dX]/; # 身份证强检验
        ##
        # 超长截取
        #  @author liuf
        # @param msg 原始值
        # @param count 截取长度

        def superLongIntercept(msg,count)
          if msg.length > count
            msg = msg[0,count].concat('...超长自动截取...'); # 自动截取日志
          end
          return msg
        end
        msg = superLongIntercept(msg,count)

        ##
        # 敏感词汇
        # @author liuf
        # @param msg 原始值

        def sensitiveWordRecognize(msg)
           return msg.gsub(EMAIL_PATTERN, REPLACE).gsub(MOBILE_PATTERN, REPLACE).gsub(INTERNET_ALl_URL, REPLACE).gsub(ID, REPLACE);
        end
        msg =  sensitiveWordRecognize(msg); 
        event.set('message', msg); # 存入 message
        "
    }
}

4.保存

:wq! (强制性保存)

全部配置文件

input {
    udp {
        port => "${INPUT_UDP_PORT}"
        type => syslog
        codec => json
    }
    tcp {
        port => "${INPUT_TCP_PORT}"
        type => syslog
        codec => json_lines
    }
    http {
        port => "${INPUT_HTTP_PORT}"
        codec => "json"
    }
}

filter {
    if [logger_name] =~ "metrics" {
        kv {
            source => "message"
            field_split_pattern => ", "
            prefix => "metric_"
        }
        mutate {
            convert => { "metric_value" => "float" }
            convert => { "metric_count" => "integer" }
            convert => { "metric_min" => "float" }
            convert => { "metric_max" => "float" }
            convert => { "metric_mean" => "float" }
            convert => { "metric_stddev" => "float" }
            convert => { "metric_median" => "float" }
            convert => { "metric_p75" => "float" }
            convert => { "metric_p95" => "float" }
            convert => { "metric_p98" => "float" }
            convert => { "metric_p99" => "float" }
            convert => { "metric_p999" => "float" }
            convert => { "metric_mean_rate" => "float" }
            convert => { "metric_m1" => "float" }
            convert => { "metric_m5" => "float" }
            convert => { "metric_m15" => "float" }
            # No need to keep message field after it has been parsed
            remove_field => ["message"]
        }
    }
    if [type] == "syslog" {
        mutate {
            add_field => { "instance_name" => "%{app_name}-%{host}:%{app_port}" }
        }
    }
    mutate {
        # workaround from https://github.com/elastic/logstash/issues/5115
        add_field => { "[@metadata][LOGSTASH_DEBUG]" => "${LOGSTASH_DEBUG:false}" }
    }
 ruby {
        code => "
        msg = event.get('message'); # 获取 message
        count = 20000;
        REPLACE ='******'
        MOBILE_PATTERN = /(13[0-9]|14[01456879]|15[0-3,5-9]|16[2567]|17[0-8]|18[0-9]|19[0-3,5-9])\d{8}/; # 手机号正则
        EMAIL_PATTERN = /([A-Za-z0-9_\-\.])+\@([A-Za-z0-9_\-\.])+\.([A-Za-z]{2,4})/; # 邮箱正则
        INTERNET_URL= /((http|ftp|https|sftp):\/\/[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:~+#])?)/; # 域名
        INTERNET_ALl_URL= /(http|ftp|https):\/\/[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:\/~\+#]*[\w\-\@?^=%&\/~\+#])?/; # 网址
        ID_CARD =/(\d{6})(\d{4})(\d{2})(\d{2})(\d{3})([0-9]|X)/; # 身份证号
        ID =/[1-8][1-7]\d{4}(?:19|20)\d{2}(?:0[1-9]|1[0-2])(?:0[1-9]|[12]\d|3[01])\d{3}[\dX]/; # 身份证强检验
        ##
        # 超长截取
        #  @author liuf
        # @param msg 原始值
        # @param count 截取长度

        def superLongIntercept(msg,count)
          if msg.length > count
            msg = msg[0,count].concat('...超长自动截取...'); # 自动截取日志
          end
          return msg
        end
        msg = superLongIntercept(msg,count)

        ##
        # 敏感词汇
        # @author liuf
        # @param msg 原始值

        def sensitiveWordRecognize(msg)
           return msg.gsub(EMAIL_PATTERN, REPLACE).gsub(MOBILE_PATTERN, REPLACE).gsub(INTERNET_ALl_URL, REPLACE).gsub(ID, REPLACE);
        end
        msg =  sensitiveWordRecognize(msg); 
        event.set('message', msg); # 存入 message
        "
    }
}

output {
    if [logger_name] =~ "metrics" {
        elasticsearch {
            hosts => ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
            index => "metrics-%{+YYYY.MM.dd}"
        }
    } else {
        elasticsearch {
            hosts => ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
            index => "logs-%{+YYYY.MM.dd}"
        }
    }
    if [@metadata][LOGSTASH_DEBUG] == "true" {
        stdout {
            codec => rubydebug
        }
    }

}