在Lucene中如何将查询项tfidf作为文档相似度计算的一个因子

Question

我试图通过Lucene实现显式语义分析(ESA)。

在匹配文档时，如何考虑查询中的术语TFIDF？

null

有更好的办法吗？

Answer 1

Lucene已经支持TF/IDF评分，当然，默认情况下，所以我不太确定你在找什么。

实际上，这听起来有点像您希望根据查询本身中的TF/IDF来衡量查询术语。因此，让我们考虑其中的两个要素：

>

tf:Lucene将每个查询项的得分相加。如果同一个查询项在查询中出现两次（如字段:(aab))，加倍后的项将获得更重的权重，相当于（但绝不等于）增加2。

• a b a匹配四个查询项(a b a a，但不匹配C D)
• a b c匹配五个查询项(a b a c a，但不匹配D)

因此，这个特定的得分元素将更强烈地得分第二个文档。

下面是docab aexplain（请参见indexsearcher.explain)的输出：

0.26880693 = (MATCH) product of:
  0.40321037 = (MATCH) sum of:
    0.10876686 = (MATCH) weight(text:a in 0) [DefaultSimilarity], result of:
      0.10876686 = score(doc=0,freq=2.0 = termFreq=2.0
), product of:
        0.25872254 = queryWeight, product of:
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.435168 = queryNorm
        0.42039964 = fieldWeight in 0, product of:
          1.4142135 = tf(freq=2.0), with freq of:
            2.0 = termFreq=2.0
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.5 = fieldNorm(doc=0)
    0.07690979 = (MATCH) weight(text:b in 0) [DefaultSimilarity], result of:
      0.07690979 = score(doc=0,freq=1.0 = termFreq=1.0
), product of:
        0.25872254 = queryWeight, product of:
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.435168 = queryNorm
        0.29726744 = fieldWeight in 0, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.5 = fieldNorm(doc=0)
    0.10876686 = (MATCH) weight(text:a in 0) [DefaultSimilarity], result of:
      0.10876686 = score(doc=0,freq=2.0 = termFreq=2.0
), product of:
        0.25872254 = queryWeight, product of:
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.435168 = queryNorm
        0.42039964 = fieldWeight in 0, product of:
          1.4142135 = tf(freq=2.0), with freq of:
            2.0 = termFreq=2.0
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.5 = fieldNorm(doc=0)
    0.10876686 = (MATCH) weight(text:a in 0) [DefaultSimilarity], result of:
      0.10876686 = score(doc=0,freq=2.0 = termFreq=2.0
), product of:
        0.25872254 = queryWeight, product of:
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.435168 = queryNorm
        0.42039964 = fieldWeight in 0, product of:
          1.4142135 = tf(freq=2.0), with freq of:
            2.0 = termFreq=2.0
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.5 = fieldNorm(doc=0)
  0.6666667 = coord(4/6)

0.43768594 = (MATCH) product of:
  0.52522314 = (MATCH) sum of:
    0.07690979 = (MATCH) weight(text:a in 1) [DefaultSimilarity], result of:
      0.07690979 = score(doc=1,freq=1.0 = termFreq=1.0
), product of:
        0.25872254 = queryWeight, product of:
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.435168 = queryNorm
        0.29726744 = fieldWeight in 1, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.5 = fieldNorm(doc=1)
    0.07690979 = (MATCH) weight(text:b in 1) [DefaultSimilarity], result of:
      0.07690979 = score(doc=1,freq=1.0 = termFreq=1.0
), product of:
        0.25872254 = queryWeight, product of:
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.435168 = queryNorm
        0.29726744 = fieldWeight in 1, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.5 = fieldNorm(doc=1)
    0.07690979 = (MATCH) weight(text:a in 1) [DefaultSimilarity], result of:
      0.07690979 = score(doc=1,freq=1.0 = termFreq=1.0
), product of:
        0.25872254 = queryWeight, product of:
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.435168 = queryNorm
        0.29726744 = fieldWeight in 1, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.5 = fieldNorm(doc=1)
    0.217584 = (MATCH) weight(text:c in 1) [DefaultSimilarity], result of:
      0.217584 = score(doc=1,freq=1.0 = termFreq=1.0
), product of:
        0.435168 = queryWeight, product of:
          1.0 = idf(docFreq=1, maxDocs=2)
          0.435168 = queryNorm
        0.5 = fieldWeight in 1, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          1.0 = idf(docFreq=1, maxDocs=2)
          0.5 = fieldNorm(doc=1)
    0.07690979 = (MATCH) weight(text:a in 1) [DefaultSimilarity], result of:
      0.07690979 = score(doc=1,freq=1.0 = termFreq=1.0
), product of:
        0.25872254 = queryWeight, product of:
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.435168 = queryNorm
        0.29726744 = fieldWeight in 1, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          0.5945349 = idf(docFreq=2, maxDocs=2)
          0.5 = fieldNorm(doc=1)
  0.8333333 = coord(5/6)

但是，也请注意第二个文件中“C”一词在coord和idf中的不同之处。这些分数的影响只是抹去了你从增加同一项的倍数中获得的提升。如果向查询添加足够的，它们最终会交换位置。对C上的匹配结果进行了计算，认为它是一个更有意义的结果。