Elasticsearch-50-most_fields策略进行cross-fields搜索的弊端

发表于 | 分类于

cross-field搜索

就是我们搜索一个唯一标识的时候跨越了多个field,比如一个人,标识是姓名,一个建筑的标识是地址. 姓名可以散落在多个field中,比如first_name和last_name中,地址可以散落在country,province,city中.
跨多个field搜索一个标识,就是cross-fields搜索

这个情况下用most-fields搜索就比较合适了,因为best-fields是优先搜索单个field最匹配的结果,cross-fields本身就不是一个field的问题了

案例

首先,准备数据

1
2
3
4
5
6
7
8
9
10
11
POST /forum/article/_bulk
{ "update": { "_id": "1"} }
{ "doc" : {"author_first_name" : "Peter", "author_last_name" : "Smith"} }
{ "update": { "_id": "2"} }
{ "doc" : {"author_first_name" : "Smith", "author_last_name" : "Williams"} }
{ "update": { "_id": "3"} }
{ "doc" : {"author_first_name" : "Jack", "author_last_name" : "Ma"} }
{ "update": { "_id": "4"} }
{ "doc" : {"author_first_name" : "Robbin", "author_last_name" : "Li"} }
{ "update": { "_id": "5"} }
{ "doc" : {"author_first_name" : "Tonny", "author_last_name" : "Peter Smith"} }

然后来查询一下Peter Smith

1
2
3
4
5
6
7
8
9
10
GET /forum/article/_search
{
  "query": {
    "multi_match": {
      "query": "Peter Smith",
      "fields": ["author_first_name","author_last_name"],
      "type": "most_fields"
    }
  }
}

返回值:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
{
  "took": 28,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 3,
    "max_score": 0.6931472,
    "hits": [
      {
        "_index": "forum",
        "_type": "article",
        "_id": "2",
        "_score": 0.6931472,
        "_source": {
          "articleID": "KDKE-B-9947-#kL5",
          "userID": 1,
          "hidden": false,
          "postDate": "2017-01-02",
          "tag": [
            "java"
          ],
          "tag_cnt": 1,
          "view_cnt": 50,
          "title": "this is java blog",
          "content": "i think java is the best programming language",
          "sub_title": "learned a lot of course",
          "author_first_name": "Smith",
          "author_last_name": "Williams"
        }
      },
      {
        "_index": "forum",
        "_type": "article",
        "_id": "1",
        "_score": 0.5753642,
        "_source": {
          "articleID": "XHDK-A-1293-#fJ3",
          "userID": 1,
          "hidden": false,
          "postDate": "2017-01-01",
          "tag": [
            "java",
            "hadoop"
          ],
          "tag_cnt": 2,
          "view_cnt": 30,
          "title": "this is java and elasticsearch blog",
          "content": "i like to write best elasticsearch article",
          "sub_title": "learning more courses",
          "author_first_name": "Peter",
          "author_last_name": "Smith"
        }
      },
      {
        "_index": "forum",
        "_type": "article",
        "_id": "5",
        "_score": 0.51623213,
        "_source": {
          "articleID": "DHJK-B-1395-#Ky5",
          "userID": 3,
          "hidden": false,
          "postDate": "2018-12-03",
          "tag": [
            "elasticsearch"
          ],
          "tag_cnt": 1,
          "view_cnt": 10,
          "title": "this is spark blog",
          "content": "spark is best big data solution based on scala ,an programming language similar to java",
          "sub_title": "haha, hello world",
          "author_first_name": "Tonny",
          "author_last_name": "Peter Smith"
        }
      }
    ]
  }
}

来看一下返回值, id是2的document被排在了第一位,为什么?
因为IDF分数高, document2的author_first_name 是Smith,在所有的doc中只出现过一次,出现的频率低
再来看下document1 和 document5 这两个的author_last_name都出现了,所以导致document1的分数要比document2的分数要低

大概来说是这样的,es的算法很复杂,这些都是可能影响分数的.

cross-fields问题

  • 只是找到尽可能多的field匹配到的document,而不是某个field完全匹配的document
  • most-fields没办法使用minimum_should_match去掉长尾数据,就是匹配特别少的结果
  • TF/IDF算法,比如上面搜索中的Peter Smith和Smith Williams,搜索Peter Smith的时候,由于first_name中很少有Smith的,所以在query中所有document中的频率很低,得到的分数很高,可能Smith Williams反而会排在Peter Smith的前面