Как избежать URL для поиска Elasticsearch?

Question

В одном из моих полей в Elasticsearch Я хранения URL моих документов (например, http://techcrunch.com/something-great)

Когда я не избежать URL, чтобы документ был правильно - но я получаю ошибку EOF на некоторых URL-адресов ,

Когда я избежать URL с:

String escapedString = QueryParser.escape(e.getKey().getUrl()); 

Документ не найден - я получаю ноль хитов.

Итак, как это сделать?

---

{ 
    _index: "crawlbot", 
    _type: "article", 
    _id: "AVFaaFu4w49jUzVInKS5", 
    _score: 1, 
    _source: { 
     job: { 
      id: 65, 
      name: "wikipedia_en", 
      max_pages: 300000, 
      crawl_depth: 0, 
      processing_patterns: "-Category,-User,-Wikipedia:,-Topic,-Special:,-Talk:,-Portal:,-MOS", 
      status: 0, 
      days: 0, 
      url: [ 
       "https://en.wikipedia.org" 
      ], 
      ajax: false, 
      min_description: 0 
     }, 
     article: { 
      url: "https://en.wikipedia.org/w/index.php?action=history&feed=atom&title=Parliament_of_Romania", 
      provider_url: "https://en.wikipedia.org", 
      provider_name: "", 
      provider_display: "en.wikipedia.org", 
      favicon_url: "http://www.google.com/s2/u/0/favicons?domain=https://en.wikipedia.org", 
      language: "en", 
      metadata: { 
       authors: [] 
      }, 
      entities: [], 
      keywords: [], 
      videos: [], 
      unfilteredKeywords: [], 
      published: "", 
      published_long: 0 
     } 
    } 
} 

И я хотел бы, чтобы получить документ за article.url

Это запрос:

SearchRequestBuilder requestBuilder = client.prepareSearch("crawlbot").setSearchType(SearchType.DFS_QUERY_THEN_FETCH); 
      BoolQueryBuilder queryBuilder = new BoolQueryBuilder(); 
      String escapedString = QueryParser.escape(e.getKey().getUrl()); 
      queryBuilder.must(QueryBuilders.queryStringQuery(escapedString).defaultField("article.url")); 
      queryBuilder.must(QueryBuilders.queryStringQuery(e.getKey().getJob().getId() + "").defaultField("job.id")); 

Ошибка, если я не избежать:

Exception in thread "main" org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed; shardFailures {[9_T8APppReyWKppSNZWmXw][crawlbot][0]: SearchParseException[[crawlbot][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query":{"bool":{"must":[{"query_string":{"query":"http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4","default_field":"article.url"}},{"query_string":{"query":"70","default_field":"job.id"}}]}}}]]]; nested: QueryParsingException[[crawlbot] Failed to parse query [http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4]]; nested: ParseException[Cannot parse 'http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4': Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; nested: TokenMgrError[Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; }{[9_T8APppReyWKppSNZWmXw][crawlbot][1]: SearchParseException[[crawlbot][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query":{"bool":{"must":[{"query_string":{"query":"http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4","default_field":"article.url"}},{"query_string":{"query":"70","default_field":"job.id"}}]}}}]]]; nested: QueryParsingException[[crawlbot] Failed to parse query [http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4]]; nested: ParseException[Cannot parse 'http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4': Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; nested: TokenMgrError[Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; }{[9_T8APppReyWKppSNZWmXw][crawlbot][2]: SearchParseException[[crawlbot][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query":{"bool":{"must":[{"query_string":{"query":"http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4","default_field":"article.url"}},{"query_string":{"query":"70","default_field":"job.id"}}]}}}]]]; nested: QueryParsingException[[crawlbot] Failed to parse query [http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4]]; nested: ParseException[Cannot parse 'http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4': Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; nested: TokenMgrError[Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; }{[9_T8APppReyWKppSNZWmXw][crawlbot][3]: SearchParseException[[crawlbot][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query":{"bool":{"must":[{"query_string":{"query":"http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4","default_field":"article.url"}},{"query_string":{"query":"70","default_field":"job.id"}}]}}}]]]; nested: QueryParsingException[[crawlbot] Failed to parse query [http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4]]; nested: ParseException[Cannot parse 'http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4': Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; nested: TokenMgrError[Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; }{[9_T8APppReyWKppSNZWmXw][crawlbot][4]: SearchParseException[[crawlbot][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query":{"bool":{"must":[{"query_string":{"query":"http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4","default_field":"article.url"}},{"query_string":{"query":"70","default_field":"job.id"}}]}}}]]]; nested: QueryParsingException[[crawlbot] Failed to parse query [http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4]]; nested: ParseException[Cannot parse 'http://www.zeit.de/wirtschaft/2015-11/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4': Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; nested: TokenMgrError[Lexical error at line 1, column 111. Encountered: <EOF> after : "/griechenland-reparationszahlung-bayern-rentner-jahresrueckblick-2?page=4"]; } 
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237) 
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:183) 
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 

Answer 1

Предлагаю вам изменить картографию г article.url поле:

url: { 
    "type": "string", 
    "index": "not_analyzed" 
} 

Неспособность сделать это сделает ваше поле проанализирован и очень трудно запросить учитывая то, как стандартный анализатор будет разбить URL на несколько лексем.

Затем, вместо запроса query_string, вы можете использовать запрос term, чтобы запросить ваши документы.

SearchRequestBuilder requestBuilder = client.prepareSearch("crawlbot").setSearchType(SearchType.DFS_QUERY_THEN_FETCH); 
BoolQueryBuilder queryBuilder = new BoolQueryBuilder(); 
queryBuilder.must(QueryBuilders.termQuery("article.url", e.getKey().getUrl())); 
...        ^
            | 
         use a term query instead 

UPDATE

Вслед за комментарием EVALDAS' (престижность Evaldas!), В конце концов, идея заключается в том, чтобы создать пользовательский анализатор для того, чтобы убедиться, что URL будет строчными, а также ,

При создании индекса, вы можете добавить новый анализатор в settings, а затем использовать его в качестве анализатора Вашего article.url поля:

PUT /crawlbot 
{ 
    "settings": { 
     "analysis": { 
      "analyzer": { 
       "url_analyzer": { 
        "type":   "custom", 
        "tokenizer": "keyword", 
        "filter":  [ "lowercase" ] 
       } 
      } 
     } 
    }, 
    "mappings": { 
     "article": { 
      "properties": { 
       "article": { 
        "url": { 
         "type": "string", 
         "analyzer": "url_analyzer" 
        } 
       } 
      } 
     } 
    } 
}