如何在弹性APM中查看来自OpenTelemeter的跟踪日志

Question

我收到弹性APM日志结构中来自opentelemeter收集器的日志：

“{Timestamp:HH:mm:ss}{Level:u3}trace.id={TraceId}事务。id={SpanId}{NewLine}{Message:lj}{NewLine}{Exception}”

例子：

08:27:47 INF trace.id=898a7716358b25408d4f193f1cd17831 transaction.id=4f7590e4ba80b64b SOME MSG

我试着使用管道

POST _ingest/pipeline/_simulate {   "pipeline": {   "description" : "parse multiple patterns",   "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:logtime} %{LOGLEVEL:loglevel} \\[trace.id=%{TRACE_ID:trace.id}(?: transaction.id=%{SPAN_ID:transaction.id})?\\] %{GREEDYDATA:message}"],
        "pattern_definitions": {
          "TRACE_ID": "[0-9A-Fa-f]{32}",
          "SPAN_ID": "[0-9A-Fa-f]{16}"
        }
      },
      "date": { "field": "logtime", "target_field": "@timestamp", "formats": ["HH:mm:ss"] }
    }   ] } }

我的目标是查看Elastic APM中的日志

 {
        "@timestamp": 2021-01-05T10:10:10",
    
        "message":  "Protocol Port MIs-Match",
        "trace": {
            "traceId": "898a7716358b25408d4f193f1cd17831",
            "spanId": "4f7590e4ba80b64b"
        }
    }

Answer 1

到目前为止干得不错。你的管道几乎是好的，但是，格罗克模式需要一些修复，你有一些孤儿卷发背带。以下是一个有效的例子：

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "parse multiple patterns",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            """%{TIME:logtime} %{WORD:loglevel} trace.id=%{TRACE_ID:trace.id}(?: transaction.id=%{SPAN_ID:transaction.id})? %{GREEDYDATA:message}"""
          ],
          "pattern_definitions": {
            "TRACE_ID": "[0-9A-Fa-f]{32}",
            "SPAN_ID": "[0-9A-Fa-f]{16}"
          }
        }
      },
      {
        "date": {
          "field": "logtime",
          "target_field": "@timestamp",
          "formats": [
            "HH:mm:ss"
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "08:27:47 INF trace.id=898a7716358b25408d4f193f1cd17831 transaction.id=4f7590e4ba80b64b SOME MSG"
      }
    }
  ]
}

回应：

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "trace" : {
            "id" : "898a7716358b25408d4f193f1cd17831"
          },
          "@timestamp" : "2021-01-01T08:27:47.000Z",
          "loglevel" : "INF",
          "message" : "SOME MSG",
          "logtime" : "08:27:47",
          "transaction" : {
            "id" : "4f7590e4ba80b64b"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-03-30T11:07:52.067275598Z"
        }
      }
    }
  ]
}

请注意，缺少确切的日期，因此@timemark字段将解析为今年1月1日。