Elasticsearch-64-IK中文分词器的安装和基本使用

发表于 | 分类于

ik中文分词器安装

从github上拉取ik分词器

1
git clone https://github.com/medcl/elasticsearch-analysis-ik

切换分支

1
git checkout tags/v5.2.0

编译

1
mvn package

将target/releases/elasticsearch-analysis-ik-5.2.0.zip拷贝到es/plugins/ik目录下

在es/plugins/ik下对elasticsearch-analysis-ik-5.2.0.zip进行解压缩,然后删除压缩包

最后重启es

或者说直接这里下载压缩包即可

基础知识

ik分词器中,包含了两种analyzer,可以根据自己的需要自己选,一般用ik_max_word

ik_max_word

会将文本做最细粒度的拆分,比如说会将”中华人民共和国国歌”拆分为”中华人民共和国,中华人民,中华,华人,人民共和国,人民,人,民,共和国,共和,和,国国,国歌”,会穷尽各种可能的组合

ik_smart

会做最粗粒度的拆分,比如将”中华人民共和国国歌”拆分为”中华人民共和国,国歌”

基本的使用

创建索引的时候指定使用ik分词器

1
2
3
4
5
6
7
8
9
10
11
12
13
PUT /my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "text":{
          "type": "text",
          "analyzer": "ik_max_word"
        }
      }
    }
  }
}

创建完成后添加几条测试数据

1
2
3
4
5
6
7
8
9
10
11
12
13
POST /my_index/my_type/_bulk
{ "index": { "_id": "1"} }
{ "text": "男子偷上万元发红包求交女友 被抓获时仍然单身" }
{ "index": { "_id": "2"} }
{ "text": "16岁少女为结婚“变”22岁 7年后想离婚被法院拒绝" }
{ "index": { "_id": "3"} }
{ "text": "深圳女孩骑车逆行撞奔驰 遭索赔被吓哭(图)" }
{ "index": { "_id": "4"} }
{ "text": "女人对护肤品比对男票好？网友神怼" }
{ "index": { "_id": "5"} }
{ "text": "为什么国内的街道招牌用的都是红黄配？" }
``` 
来看下这个分词器的效果

GET my_index/_analyze
{
“text”: “男子偷上万元发红包求交女友 被抓获时仍然单身”,
“analyzer”: “ik_max_word”
}

1
返回值:

{
“tokens”: [
{
“token”: “男子”,
“start_offset”: 0,
“end_offset”: 2,
“type”: “CN_WORD”,
“position”: 0
},
{
“token”: “偷上”,
“start_offset”: 2,
“end_offset”: 4,
“type”: “CN_WORD”,
“position”: 1
},
{
“token”: “上万”,
“start_offset”: 3,
“end_offset”: 5,
“type”: “CN_WORD”,
“position”: 2
},
{
“token”: “万元”,
“start_offset”: 4,
“end_offset”: 6,
“type”: “CN_WORD”,
“position”: 3
},
{
“token”: “万”,
“start_offset”: 4,
“end_offset”: 5,
“type”: “CN_WORD”,
“position”: 4
},
{
“token”: “元”,
“start_offset”: 5,
“end_offset”: 6,
“type”: “CN_CHAR”,
“position”: 5
},
{
“token”: “发红包”,
“start_offset”: 6,
“end_offset”: 9,
“type”: “CN_WORD”,
“position”: 6
},
{
“token”: “发红”,
“start_offset”: 6,
“end_offset”: 8,
“type”: “CN_WORD”,
“position”: 7
},
{
“token”: “发”,
“start_offset”: 6,
“end_offset”: 7,
“type”: “CN_WORD”,
“position”: 8
},
{
“token”: “红包”,
“start_offset”: 7,
“end_offset”: 9,
“type”: “CN_WORD”,
“position”: 9
},
{
“token”: “求”,
“start_offset”: 9,
“end_offset”: 10,
“type”: “CN_CHAR”,
“position”: 10
},
{
“token”: “交”,
“start_offset”: 10,
“end_offset”: 11,
“type”: “CN_CHAR”,
“position”: 11
},
{
“token”: “女友”,
“start_offset”: 11,
“end_offset”: 13,
“type”: “CN_WORD”,
“position”: 12
},
{
“token”: “抓获”,
“start_offset”: 15,
“end_offset”: 17,
“type”: “CN_WORD”,
“position”: 13
},
{
“token”: “获”,
“start_offset”: 16,
“end_offset”: 17,
“type”: “CN_WORD”,
“position”: 14
},
{
“token”: “时”,
“start_offset”: 17,
“end_offset”: 18,
“type”: “CN_CHAR”,
“position”: 15
},
{
“token”: “仍然”,
“start_offset”: 18,
“end_offset”: 20,
“type”: “CN_WORD”,
“position”: 16
},
{
“token”: “单身”,
“start_offset”: 20,
“end_offset”: 22,
“type”: “CN_WORD”,
“position”: 17
}
]
}

1
2
3
他会把所有可能的词语都拆分出来    
  
然后我们来测试搜索一下

GET /my_index/my_type/_search
{
“query”: {
“match”: {
“text”: “16岁少女结婚好还是单身好？”
}
}
}

1
返回:

{
“took”: 11,
“timed_out”: false,
“_shards”: {
“total”: 5,
“successful”: 5,
“failed”: 0
},
“hits”: {
“total”: 3,
“max_score”: 3.603062,
“hits”: [
{
“_index”: “my_index”,
“_type”: “my_type”,
“_id”: “2”,
“_score”: 3.603062,
“_source”: {
“text”: “16岁少女为结婚“变”22岁 7年后想离婚被法院拒绝”
}
},
{
“_index”: “my_index”,
“_type”: “my_type”,
“_id”: “4”,
“_score”: 1.3862944,
“_source”: {
“text”: “女人对护肤品比对男票好？网友神怼”
}
},
{
“_index”: “my_index”,
“_type”: “my_type”,
“_id”: “1”,
“_score”: 0.2699054,
“_source”: {
“text”: “男子偷上万元发红包求交女友 被抓获时仍然单身”
}
}
]
}
}
`