Elasticsearch-51-cross-fields搜索问题解决方案

发表于 | 分类于

解决方案一:使用copy_to

用copy_to可以将多个field组合成一个field.

之前说的问题,其实就是出在了有多个field,那么我们只要把这些field合并成一个field即可,比如搜索一个人名,有first_name和last_name,将这两个field合并成一个full_name就可以解决了

示例

首先,创建三个field: new_author_first_name , new_author_last_name , new_author_full_name

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
PUT /forum/_mapping/article
{
  "properties": {
      "new_author_first_name": {
          "type":     "string",
          "copy_to":  "new_author_full_name" 
      },
      "new_author_last_name": {
          "type":     "string",
          "copy_to":  "new_author_full_name" 
      },
      "new_author_full_name": {
          "type":     "string"
      }
  }
}

new_author_first_name和new_author_last_name都设置copy_to到new_author_full_name中去,用了这个copy_to语法之后,就可以将多个字段的值拷贝到一个字段中,并建立倒排索引

添加数据

1
2
3
4
5
6
7
8
9
10
11
POST /forum/article/_bulk
{ "update": { "_id": "1"} }
{ "doc" : {"new_author_first_name" : "Peter", "new_author_last_name" : "Smith"} }
{ "update": { "_id": "2"} }	
{ "doc" : {"new_author_first_name" : "Smith", "new_author_last_name" : "Williams"} }	
{ "update": { "_id": "3"} }
{ "doc" : {"new_author_first_name" : "Jack", "new_author_last_name" : "Ma"} }		
{ "update": { "_id": "4"} }
{ "doc" : {"new_author_first_name" : "Robbin", "new_author_last_name" : "Li"} }	
{ "update": { "_id": "5"} }
{ "doc" : {"new_author_first_name" : "Tonny", "new_author_last_name" : "Peter Smith"} }

添加完毕后,这时候可以去查询一下全部的数据,返现并没有new_author_full_name这个field,因为这个field就类似于之前有说过的 _all元数据,是不在_source中显示的

接着来查询名称是Peter Smith的数据

1
2
3
4
5
6
7
8
GET forum/article/_search
{
  "query": {
    "match": {
      "new_author_full_name": "Peter Smith"
    }
  }
}

返回值:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 3,
    "max_score": 0.62191015,
    "hits": [
      {
        "_index": "forum",
        "_type": "article",
        "_id": "2",
        "_score": 0.62191015,
        "_source": {
          "articleID": "KDKE-B-9947-#kL5",
          "userID": 1,
          "hidden": false,
          "postDate": "2017-01-02",
          "tag": [
            "java"
          ],
          "tag_cnt": 1,
          "view_cnt": 50,
          "title": "this is java blog",
          "content": "i think java is the best programming language",
          "sub_title": "learned a lot of course",
          "author_first_name": "Smith",
          "author_last_name": "Williams",
          "new_author_last_name": "Williams",
          "new_author_first_name": "Smith"
        }
      },
      {
        "_index": "forum",
        "_type": "article",
        "_id": "1",
        "_score": 0.51623213,
        "_source": {
          "articleID": "XHDK-A-1293-#fJ3",
          "userID": 1,
          "hidden": false,
          "postDate": "2017-01-01",
          "tag": [
            "java",
            "hadoop"
          ],
          "tag_cnt": 2,
          "view_cnt": 30,
          "title": "this is java and elasticsearch blog",
          "content": "i like to write best elasticsearch article",
          "sub_title": "learning more courses",
          "author_first_name": "Peter",
          "author_last_name": "Smith",
          "new_author_last_name": "Smith",
          "new_author_first_name": "Peter"
        }
      },
      {
        "_index": "forum",
        "_type": "article",
        "_id": "5",
        "_score": 0.5063205,
        "_source": {
          "articleID": "DHJK-B-1395-#Ky5",
          "userID": 3,
          "hidden": false,
          "postDate": "2018-12-03",
          "tag": [
            "elasticsearch"
          ],
          "tag_cnt": 1,
          "view_cnt": 10,
          "title": "this is spark blog",
          "content": "spark is best big data solution based on scala ,an programming language similar to java",
          "sub_title": "haha, hello world",
          "author_first_name": "Tonny",
          "author_last_name": "Peter Smith",
          "new_author_last_name": "Peter Smith",
          "new_author_first_name": "Tonny"
        }
      }
    ]
  }
}

这里的搜索结果还是和之前一样的,因为es的算法的原因,没法实现这个场景,但是copy_to已经把前一节提到的问题解决了

之前的问题一被合并成一个field了,就不存在了,而且这里的查询可以使用minimum_should_match来去长尾,第三个问题Smith和Peter在一个field里面了,所以在所有document中出现的次数是均匀的,不会有极端的偏差

解决方案二:原生cross-fields

语法

1
2
3
4
5
6
7
8
9
10
11
GET forum/article/_search
{
  "query": {
    "multi_match": {
      "query": "Peter Smith",
      "fields": ["author_first_name","author_last_name"],
      "type": "cross_fields",
      "operator": "and"
    }
  }
}

这种方法也可以解决上文提到的那三个问题

  • cross_fields是要求每个term都必须在任何一个field中出现

这就解决了第一个问题,举个例子:
搜索条件还是Peter Smith,按照cross_fields来搜索的话
要求Peter必须在author_first_name或author_last_name中出现
要求Smith必须在author_first_name或author_last_name中出现
Peter Smith可能是横跨在多个field中的,所以必须要求每个term都在某个field中出现,组合起来才能组成我们想要的标识,比如一个完整的人名

原来most-fields搜索的时候,可能像Smith Williams也可能会出现,因为most-fields要求只是任何一个field匹配了就可以,匹配的field越多,分数就越高

第二个问题是most-fields没办法去长尾的问题,用cross_fields的时候,每个term都要求出现,那长尾肯定被干掉了
举个例子现在有一个搜索条件是java Hadoop spark 那么这三个term都必须在任何一个filed中出现了
比如有的document中,只有一个field中包含一个java,那就被干掉了,作为长尾就没了.

第三个问题,在使用cross-fields查询的时候,es在计算IDF的时候会将每个query在每个filed中的IDF都取出来,取最小值,就不会出现极端情况下的最大值了

举个例子,还是查询Peter Smith
Smith,在author_first_name这个field中,在所有document的这个field中,出现的频率很低,导致IDF分数很高;Smith在所有doc的author_last_name field中的频率算出一个IDF分数,因为一般来说last_name中的Smith频率都较高,所以IDF分数是正常的,不会太高;然后对于Smith来说,会取两个IDF分数中较小的那个分数.就不会出现IDF分过高的情况.