无法理解elasticsearch分析器正则表达式

Question

有人能帮我理解为什么我对elasticsearch分析器的理解是错误的吗？

我有一个包含各种字段的索引，特别是：

"categories": {
    "type": "text",
    "analyzer": "words_only_analyser",
    "copy_to": "all",
    "fields": {
         "tokens": {
             "type": "text",
             "analyzer": "words_only_analyser",
             "term_vector": "yes",
             "fielddata" : True
          }
      }
}

words_only_analyser如下所示：

"words_only_analyser":{
    "type":"custom",
    "tokenizer":"words_only_tokenizer",
    "char_filter" : ["html_strip"],
    "filter":[ "lowercase", "asciifolding", "stop_filter", "kstem" ]
},

并且words\u only\u标记器看起来像：

"tokenizer":{
    "words_only_tokenizer":{
    "type":"pattern",
    "pattern":"[^\\w-]+"
    }
}

我对标记器中的模式[^\\w-]的理解是，它将标记一个句子，以便在出现任意数量的\或w或-时拆分它们。例如，给定模式，以下句子：

seasonal-christmas-halloween this is a description about halloween

我希望看到：

[seasonal, christmas, hallo, een this is a description about hallo, een]

我可以从https://regex101.com/

然而，当我对上面的句子运行words\u only\u分析器时：

curl -XGET localhost:9200/contextual/_analyze?pretty -H 'Content-Type: application/json' -d '{"analyzer":"words_only_analyser","text":"seasonal-christmas-halloween this is a description about halloween"}'

我得到，

{
  "tokens" : [
    {
      "token" : "seasonal-christmas-halloween",
      "start_offset" : 0,
      "end_offset" : 28,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "description",
      "start_offset" : 39,
      "end_offset" : 50,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : "halloween",
      "start_offset" : 57,
      "end_offset" : 66,
      "type" : "word",
      "position" : 6
    }
  ]
}

这告诉我句子被标记为：

[seasonal-christmas-halloween, description, halloween]

在我看来，标记器模式没有得到实现？有人能解释一下我的理解哪里不正确吗？

Answer 1

很少有东西会改变分析器生成的最终标记，首先是标记器，然后是标记过滤器（例如：您有stop_filter删除停止字，如this，is，a）。

您也可以使用analyze API来测试您的标记器，我创建了您的配置，它生成以下标记。

邮政_analyze

{
    "tokenizer": "words_only_tokenizer", // Note `tokenizer` here
    "text": "seasonal-christmas-halloween this is a description about halloween"
}

后果

{
    "tokens": [
        {
            "token": "seasonal-christmas-halloween",
            "start_offset": 0,
            "end_offset": 28,
            "type": "word",
            "position": 0
        },
        {
            "token": "this",
            "start_offset": 29,
            "end_offset": 33,
            "type": "word",
            "position": 1
        },
        {
            "token": "is",
            "start_offset": 34,
            "end_offset": 36,
            "type": "word",
            "position": 2
        },
        {
            "token": "a",
            "start_offset": 37,
            "end_offset": 38,
            "type": "word",
            "position": 3
        },
        {
            "token": "description",
            "start_offset": 39,
            "end_offset": 50,
            "type": "word",
            "position": 4
        },
        {
            "token": "about",
            "start_offset": 51,
            "end_offset": 56,
            "type": "word",
            "position": 5
        },
        {
            "token": "halloween",
            "start_offset": 57,
            "end_offset": 66,
            "type": "word",
            "position": 6
        }
    ]
}

您可以注意到，仍然存在停止词，因为它只是打断了空格上的标记，而没有考虑-。

现在，如果您在同样具有过滤器的分析器上运行相同的操作，它将减少停止字并为您提供以下标记。

邮政_analyze

{
    "analyzer": "words_only_analyser",
    "text": "seasonal-christmas-halloween this is a description about halloween"
}

后果

{
    "tokens": [
        {
            "token": "seasonal-christmas-halloween",
            "start_offset": 0,
            "end_offset": 28,
            "type": "word",
            "position": 0
        },
        {
            "token": "description",
            "start_offset": 39,
            "end_offset": 50,
            "type": "word",
            "position": 4
        },
        {
            "token": "about",
            "start_offset": 51,
            "end_offset": 56,
            "type": "word",
            "position": 5
        },
        {
            "token": "halloween",
            "start_offset": 57,
            "end_offset": 66,
            "type": "word",
            "position": 6
        }
    ]
}