系列导航
- Istio Mixer Adapter开发 （一）K8S环境搭建
- Istio Mixer Adapter开发 （二）Istio环境搭建
- Istio Mixer Adapter开发 （三）自定义Mixer Grpc Adapter部署
概述
针对Istio Mixer Adapter的开发，后面会有文章专门讲述源码分析及开发调试的过程，这里仅使用前两篇文章讲述的K8S+Istio环境部署我已经开发好的Istio Mixer Grpc Adapter并配置K8S使其生效
gatewaymetric简介
gatewaymetric是笔者借鉴Istio官网指导开发的Istio Mixer Grpc Adapter，他的功能主要是处理Mixer上报的http访问日志，收集南北向流量（指通过istio-ingressgateway从集群外进入到集群）的访问日志，通过调用promethues官方go程序库，内存生成metrics并暴露exporter端口给promethues采集数据，它的功能目前还很不完善，但是作为学习说明已经足够
本文主要参考官网Wiki进行开发，里面也讲到了部分原理
下面是代码结构
➜ gatewaymetric git:(v1.0.5) ✗ tree
.
├── cmd
│ └── main.go
├── config
│ ├── adapter.gatewaymetric.config.pb.html
│ ├── config.pb.go
│ ├── config.proto
│ ├── config.proto_descriptor
│ └── gatewaymetric.yaml
├── gatewaymetric.go
├── sample_operator_cfg.yaml
└── testdata
├── attributes.yaml
├── gatewaymetric.yaml
├── sample_operator_cfg.yaml
└── template.yaml
复制代码上述文件中cmd/main.go为程序入口，其暴露两个端口
1）http:9145为promethues的数据抓取端口
2）grpc:41543为GrpcAdapter的通信端口，也就是与Mixer建立连接的端口，这个后面会用到，要把它显示注册到Mixer当中，使Mixer感知它
构建自定义Mixer Grpc Adapter（gatewaymetric）
我们第一步需要打包这个Adapter并制作成Docker镜像发布到Docker Hub中
➜ cmd git:(v1.0.5) ✗ cd /home/ubuntu/go/src/istio.io/istio/mixer/adapter/gatewaymetric/cmd
➜ cmd git:(master) CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o gatewaymetric
➜ cmd git:(v1.0.5) ✗ CGO_ENABLE➜ cmd git:(v1.0.5) ✗ ./gatewaymetric
adapter listening on "[::]:41543"
promethues listening on ":9145"
D=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o gatewaymetric
复制代码通过上述命令，我们基于源码构建了一个我们定制的GrpcAdapter并且成功启动了它，如果读者无法正常执行结束go build命令，则可能是go环境配置的问题，可以绕开这里直接使用源码cmd下面的gatewaymetric文件即可，笔者已经构建好了二进制文件，接下来我们编写Dockerfile
➜ cmd git:(v1.0.5) ✗ touch Dockerfile
➜ cmd git:(v1.0.5) ✗ vim Dockerfile
FROM scratch
ADD gatewaymetric /
ENTRYPOINT ["/gatewaymetric"]
复制代码该Dockerfile使用scratch基础镜像并将gatewaymetirc（我们自定义的Mixer Grpc Adapter后面都会以此称呼）加入到镜像当中，启动入口为"/gatewaymetric"
下面开始构建
➜ cmd git:(v1.0.5) ✗ sudo docker build -f Dockerfile .
Sending build context to Docker daemon 17.87MB
Step 1/3 : FROM scratch
--->
Step 2/3 : ADD gatewaymetric /
---> 82ca9cdf54ab
Step 3/3 : ENTRYPOINT ["/gatewaymetric"]
---> Running in 2630beb8da5d
Removing intermediate container 2630beb8da5d
---> 3747dfc14d90
Successfully built 3747dfc14d90
➜ cmd git:(v1.0.5) ✗ sudo docker run -p 9145:9145 -p 41543:41543 3747dfc14d90
adapter listening on "[::]:41543"
promethues listening on ":9145"
复制代码通过模拟程序测试通过（这里使用的是前面提到的Wiki当中描述的本地模拟测试的办法），说明打的Docker镜像没有问题，若读者还没有了解这块的知识，忽略这里即可，后面文章会详细描述如何在本地开发及调试Adapter
接下来，我们将该镜像提交到Docker Hub（这里讲述了开发构建并推送镜像的过程，读者可以跳过此步骤直接使用笔者已经构建好并发布到docker hub上的镜像即可）
➜ cmd git:(v1.0.5) ✗ sudo docker tag 3747dfc14d90 zrbcool/gatewaymetric:v1.0
➜ cmd git:(v1.0.5) ✗ sudo docker push zrbcool/gatewaymetric:v1.0
The push refers to repository [docker.io/zrbcool/gatewaymetric]
115a048ee88f: Pushed
v1.0: digest: sha256:616fc9e1d91ad963a5ace4b5210caecbde21aca8ea5f2bfe8f98ac20e33a6cfa size: 528
复制代码部署Adapter到K8S + Istio集群
到K8S + Istio集群的VM中，部署已经制作好的镜像
➜ gatewaymetric cat deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gatewaymetric
namespace: istio-system
labels:
app: gatewaymetric
spec:
replicas: 1
selector:
matchLabels:
app: gatewaymetric
template:
metadata:
labels:
app: gatewaymetric
spec:
containers:
- name: gatewaymetric
image: "zrbcool/gatewaymetric:v1.0"
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 9145
- containerPort: 41543
复制代码然后创建Service
➜ gatewaymetric cat service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: gatewaymetric
name: gatewaymetric
namespace: istio-system
spec:
selector:
app: gatewaymetric
ports:
type: ClusterIP
ports:
- name: promethues
port: 9145
targetPort: 9145
- name: grpc
port: 41543
targetPort: 41543
复制代码简单验证，端口已通，promethues的metrics也可以获取
➜ gatewaymetric kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
gatewaymetric ClusterIP 10.104.182.140 <none> 9145/TCP,41543/TCP
➜ gatewaymetric telnet 10.104.182.140 41543
Trying 10.104.182.140...
Connected to 10.104.182.140.
Escape character is '^]'.
➜ gatewaymetric curl 10.104.182.140:9145/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
...
复制代码至此，我们的Mixer Grpc Adapter已经部署到了我们的K8S集群当中，接下来我们需要配置CRD，使mixer能够感知到我们的Grpc Adapter
部署下面文件的内容，其包含了自定义Adapter的运行时实例handler，template->logentity的运行时实例instance，以及mixer的匹配规则rule指定handler处理instance的Report数据，如果上面这句话看的有点蒙，不要紧这个都是Mixer的设计者抽象出来的概念，目的就是让Mixer适配器的开发者跟运维者分离并使Mixer Adapter的开发更加灵活，后面会有专门章节详细阐述其各个模型的意义，如果读者想提前了解，请参考下面链接：
# handler for adapter gatewaymetric
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
name: h1
namespace: istio-system
spec:
adapter: gatewaymetric
connection:
address: "gatewaymetric.istio-system.svc.cluster.local:41543" #replaces at runtime by the test
params:
file_path: "out.txt"
---
# instance for template metric
apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
name: i1metric
namespace: istio-system
spec:
template: logentry
params:
severity: '"warning"'
timestamp: request.time
variables:
host: request.host
method: request.method | ""
url: request.path | ""
responseCode: response.code | 0
responseBodySize: response.size | 0
responseTotalSize: response.total_size | 0
latency: response.duration | "0ms"
monitored_resource_type: '"UNSPECIFIED"'
---
# rule to dispatch to handler h1
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: r1
namespace: istio-system
spec:
actions:
- handler: h1.istio-system
instances:
- i1metric
---
复制代码在Kubernetes中部署它们
➜ gatewaymetric kubectl apply -f attributes.yaml
attributemanifest.config.istio.io/istio-proxy created
attributemanifest.config.istio.io/kubernetes configured
➜ gatewaymetric kubectl apply -f template.yaml
template.config.istio.io/logentry created
➜ gatewaymetric kubectl apply -f gatewaymetric.yaml
adapter.config.istio.io/gatewaymetric created
➜ gatewaymetric kubectl apply -f operator_cfg.yaml
handler.config.istio.io/h1 created
instance.config.istio.io/i1metric created
rule.config.istio.io/r1 created
复制代码使用浏览器访问http://192.168.101.6:31380/productpage，并查看Mixer的日志，看到
{"level":"info","time":"2019-02-21T03:05:40.292749Z","instance":"accesslog.logentry.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"unknown","destinationApp":"","destinationIp":"10.32.0.34","destinationName":"","destinationNamespace":"","destinationOwner":"","destinationPrincipal":"","destinationServiceHost":"productpage.default.svc.cluster.local","destinationWorkload":"","httpAuthority":"192.168.101.6:31380","latency":"25.101758ms","method":"GET","protocol":"http","receivedBytes":488,"referer":"","reporter":"source","requestId":"405ed7ca-c988-4cfa-8e6a-e808e2cbeab8","requestSize":0,"requestedServerName":"","responseCode":200,"responseSize":5719,"responseTimestamp":"2019-02-21T03:05:40.317790Z","sentBytes":5858,"sourceApp":"","sourceIp":"0.0.0.0","sourceName":"","sourceNamespace":"istio-system","sourceOwner":"","sourcePrincipal":"","sourceWorkload":"","url":"/productpage","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36","xForwardedFor":"10.32.0.1"}
复制代码查看我们自定义的Grpc Adapter日志，看到
2019-02-21T03:05:41.002442Z info received request {[&InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:34.547407ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5719,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5858,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:39.955797905Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:39.579628ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5723,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5862,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.126271976Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:24.084598ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5719,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5858,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.293294914Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:30.813597ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:4415,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:4554,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.55426223Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,}] &Any{TypeUrl:type.googleapis.com/adapter.gatewaymetric.config.Params,Value:[10 7 111 117 116 46 116 120 116],} 7051031778836507932}
2019-02-21T03:05:41.002468Z info host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002473Z info url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002475Z info method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002477Z info responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002479Z info responseBodySize: %!(EXTRA int64=5719)
2019-02-21T03:05:41.002483Z info responseTotalSize: %!(EXTRA int64=5858)
2019-02-21T03:05:41.002487Z info latency: %!(EXTRA float64=0.034547407)
2019-02-21T03:05:41.002500Z info host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002506Z info url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002508Z info method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002510Z info responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002512Z info responseBodySize: %!(EXTRA int64=5723)
2019-02-21T03:05:41.002514Z info responseTotalSize: %!(EXTRA int64=5862)
2019-02-21T03:05:41.002516Z info latency: %!(EXTRA float64=0.039579628)
2019-02-21T03:05:41.002520Z info host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002522Z info url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002524Z info method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002526Z info responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002528Z info responseBodySize: %!(EXTRA int64=5719)
2019-02-21T03:05:41.002530Z info responseTotalSize: %!(EXTRA int64=5858)
2019-02-21T03:05:41.002532Z info latency: %!(EXTRA float64=0.024084598)
2019-02-21T03:05:41.002535Z info host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002537Z info url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002538Z info method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002540Z info responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002542Z info responseBodySize: %!(EXTRA int64=4415)
2019-02-21T03:05:41.002544Z info responseTotalSize: %!(EXTRA int64=4554)
2019-02-21T03:05:41.002547Z info latency: %!(EXTRA float64=0.030813597)
复制代码再看下promethues暴露的9145端口是否有数据呢
➜ gatewaymetric curl 10.104.182.140:9145/metrics
...
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.00099"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.00089"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0007899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0006899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0005899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0004899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0003899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0002899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0001899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-8.999999999999979e-05"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="1.0000000000000216e-05"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.00011000000000000022"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.00021000000000000023"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0003100000000000002"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0004100000000000002"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0005100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0006100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0007100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0008100000000000004"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0009100000000000004"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="+Inf"} 34
http_request_seconds_histogram_sum{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200"} 1.1806039740000003
http_request_seconds_histogram_count{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200"} 34
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.00099"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.00089"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0007899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0006899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0005899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0004899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0003899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0002899999999999998"} 0
...
复制代码说明一切已经生效，have fun！