Graylog 日志多行合并

多行合并

配置示例：

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["10.0.107.111:5044"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data #filebeat数据目录
  logs: C:\Program Files\Graylog\sidecar\logs  #sidecar日志目录
tags:
 - windows
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - F:\CODE\arrow-log\logs\*.log # 日志路径
  fields:
   app_name: example_glef_graylog
   environment: local
   log_type: Java
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}' # 匹配以时间格式为yyyy-MM-dd HH:mm:ss.SSS格式开头的日志
  multiline.negate: true 
  multiline.match: after

根据时间戳合并多行

2022-11-08 08:30:45.934 [http-nio-8081-exec-1] ERROR c.a.g.controller.LogGenerateController - 
java.lang.NullPointerException: null
	at cn.arrow.graylogglefexample.controller.LogGenerateController.createNpe(LogGenerateController.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
  multiline.negate: true
  multiline.match: after

此配置使用negate: true和match: after设置来指定任何不符合指定模式的行都属于上一行。

参考资料

1. https://www.jianshu.com/p/da818db33a22
2. Graylog收集多行日志（Java Error Log）
3. filebeat+kafka+graylog+es+mongodb可视化日志详解 - 掘金
4. filebeat合并多行日志示例 - 腾讯云开发者社区-腾讯云
5. filebeat · ELKstack 中文指南