Elasticsearch：按id选择多个不同的字段

Question

我有ES索引，其中包含以下格式的文档，这些文档使用Fluentd解析器从日志中解析，并在ES中索引。记录格式：

{"id"："id1"，"field 1"："f1_val"，"消息"："XXXX"，"时间"："XXXX"}

{"id"："id1"，"field 2"："f2_val"，"消息"："XXXX"，"时间"："XXXX"}

{"id"："id1"，"field 3"："f3_val"，"field 4"："f4_val"，"消息"："XXXX"，"时间"："XXXX"}

我想按id字段分组，并将字段分组在一起，以便将数据可视化为Kibana仪表板中的表格，如下所示：

{"id"："id1"，"field 1"："f1_val"，"field 2"："f2_val"，"field 3"："f3_val"，"field 4"："f4_val"}

在Kibana：

    Id     Field1     Field2     Field3     Field4
    id1    f1_val     f2_val     f3_val     f4_val       

如何按文档对id进行分组并在Elasticsearch中选择不同的字段值。谢谢！

Answer 1

由于Elasticsearch不支持那么好的连接，在扩展Kibana中，我建议您在将文档放入索引之前将文档连接到应用程序中。如果不可能，我会按照以下建议进行转换：

https://discuss.elastic.co/t/combine-multiple-document-into-one-document-with-limited-fields-merging-of-documents/231758

使用这个，我可以在我的仪表板中实现这样的东西：结果图像

复制步骤：

1. 创建日志索引

    PUT log_index

    POST log_index/_doc/ {"id": "1", "field1": "The"}
    
    POST log_index/_doc/ {"id": "1", "field2": "quick"}
    
    POST log_index/_doc/ {"id": "1", "field3": "brown", "field4": "fox"}

    POST log_index/_doc/ {"id": "2", "field1": "jumped"}
    
    POST log_index/_doc/ {"id": "2", "field2": "over"}
    
    POST log_index/_doc/ {"id": "2", "field3": "the"}

    POST log_index/_doc/ {"id": "2", "field4": "lazy"}

    PUT _transform/join_logs
    {
      "source": {
        "index": [ 
          "log_index"
        ]
      },
      "pivot": {
        "group_by": {
          "id.keyword": {
            "terms": {
              "field": "id.keyword"
            }
          }
        },
        "aggregations": { 
          "field1": {
            "scripted_metric": {
              "init_script": "state.docs = []",
              "map_script": "state.docs.add(new HashMap(params['_source']))",
              "combine_script": "for (t in state.docs) { if(t.get('field1') != null){ return t.get('field1')}}  return null",
              "reduce_script": "states"
            }
          },
          "field2": {
            "scripted_metric": {
              "init_script": "state.docs = []",
              "map_script": "state.docs.add(new HashMap(params['_source']))",
              "combine_script": "for (t in state.docs) { if(t.get('field2') != null){ return t.get('field2')}}  return null",
              "reduce_script": "states"
            }
          },
          "field3": {
            "scripted_metric": {
              "init_script": "state.docs = []",
              "map_script": "state.docs.add(new HashMap(params['_source']))",
              "combine_script": "for (t in state.docs) { if(t.get('field3') != null){ return t.get('field3')}}  return null",
              "reduce_script": "states"
            }
          },
          "field4": {
            "scripted_metric": {
              "init_script": "state.docs = []",
              "map_script": "state.docs.add(new HashMap(params['_source']))",
              "combine_script": "for (t in state.docs) { if(t.get('field4') != null){ return t.get('field4')}}  return null",
              "reduce_script": "states"
            }
          }
        }
      },
      "dest": { 
        "index": "joined_index"
      }
    }

我的假设是，这些字段在每个具有指定id的文档中只出现一次。不知道如果字段在文档之间重叠会发生什么。

Answer 2

{
    "size": 0,
    "aggs": {
        "id_agg": {
            "terms": {
                "field": "id.keyword"
            },
            "aggs": {
                "by_field1": {
                    "terms": {
                        "field": "field1.keyword"
                    }
                },
               "by_field2": {
                    "terms": {
                        "field": "field2.keyword"
                    }
                },
               "by_field3": {
                    "terms": {
                        "field": "field3.keyword"
                    }
                }
            }
        }
    }
}