Apache Solr DataImportHandler не индексирует данные из базы данных

Question

Чтобы импортировать данные из моей базы данных в Apache Solr, я использую DataImportHandler. Интеграция DataImportHandler прошла успешно, так как индексирование кажется таким. Но «Итоговые обработанные документы» - 0, и я не могу найти ошибки в журналах.

Я использую Apache Solr 3.5 с Drupal, и многие элементы уже проиндексированы. Итак, schema.xml уже имеет много полей. С DataImportHandler я хочу добавить еще немного контента и, следовательно, еще несколько полей. Для тестирования у меня есть тестовая база данных с одним столбцом «id» (первичный ключ) и другим столбцом «имя_программы» (текст).

В других сообщениях я часто видел подобные проблемы, но ничто не могло мне помочь, поэтому, надеюсь, вы можете. Я не знаю, что-то не так с моим полем, или если это еще одна проблема.

Это мой dataconfig.xml:

<dataConfig> 
    <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" encoding="UTF-8" 
        url="jdbc:mysql://localhost/test_db" 
         user="user" password="password" batchSize="-1"/> 
    <document> 
    <entity pk="id" query="SELECT `id` AS ID,`test_name` AS TEST_NAME FROM `test`"> 
     <field column="ID" name="id" /> 
     <field column="TEST_NAME" name="test_name" /> 
    </entity> 
    </document> 
</dataConfig> 

Мой schema.xml:

<?xml version="1.0" encoding="UTF-8" ?> 

<schema name="drupal-3.0-beta9-solr3" version="1.3"> 

    <types> 

    <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/> 

    <!-- boolean type: "true" or "false" --> 
    <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" omitNorms="true"/> 
    <!--Binary data type. The data should be sent/retrieved in as Base64 encoded Strings --> 
    <fieldtype name="binary" class="solr.BinaryField"/> 


    <!-- numeric field types that can be sorted, but are not optimized for range queries --> 
    <fieldType name="integer" class="solr.TrieIntField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> 
    <fieldType name="float" class="solr.TrieFloatField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> 
    <fieldType name="long" class="solr.TrieLongField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> 
    <fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> 

    <fieldType name="sint" class="solr.SortableIntField" sortMissingLast="true" omitNorms="true"/> 
    <fieldType name="slong" class="solr.SortableLongField" sortMissingLast="true" omitNorms="true"/> 
    <fieldType name="sfloat" class="solr.SortableFloatField" sortMissingLast="true" omitNorms="true"/> 
    <fieldType name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true" omitNorms="true"/> 

    <fieldType name="tint" class="solr.TrieIntField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> 
    <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> 
    <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> 
    <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> 


    <fieldType name="date" class="solr.DateField" sortMissingLast="true" omitNorms="true"/> 

    <!-- A Trie based date field for faster date range queries and date faceting. --> 
    <fieldType name="tdate" class="solr.TrieDateField" omitNorms="true" precisionStep="6" positionIncrementGap="0"/> 


    <!-- A text field that only splits on whitespace for exact matching of words --> 
    <fieldType name="text_ws" class="solr.TextField" omitNorms="true" positionIncrementGap="100"> 
     <analyzer> 
     <tokenizer class="solr.WhitespaceTokenizerFactory"/> 
     <filter class="solr.LowerCaseFilterFactory"/> 
     </analyzer> 
    </fieldType> 

    <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> 
     <analyzer type="index"> 
     <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/> 
     <tokenizer class="solr.WhitespaceTokenizerFactory"/> 

     <filter class="solr.StopFilterFactory" 
       ignoreCase="true" 
       words="stopwords.txt" 
       enablePositionIncrements="true" 
       /> 
     <filter class="solr.WordDelimiterFilterFactory" 
       protected="protwords.txt" 
       generateWordParts="1" 
       generateNumberParts="1" 
       catenateWords="1" 
       catenateNumbers="1" 
       catenateAll="0" 
       splitOnCaseChange="1" 
       preserveOriginal="1"/> 
     <filter class="solr.LowerCaseFilterFactory"/> 
     <filter class="solr.DictionaryCompoundWordTokenFilterFactory" dictionary="german-common-nouns.txt" minWordSize="5" minSubwordSize="2" maxSubwordSize="15" onlyLongestMatch="true"/> 

     <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="35" side="front"/> 
     <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> 
     <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> 
     </analyzer> 
     <analyzer type="query"> 
     <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/> 
     <tokenizer class="solr.WhitespaceTokenizerFactory"/> 
     <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> 
     <filter class="solr.StopFilterFactory" 
       ignoreCase="true" 
       words="stopwords.txt" 
       enablePositionIncrements="true" 
       /> 
     <filter class="solr.LowerCaseFilterFactory"/> 
     <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> 
     <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> 
     </analyzer> 
    </fieldType> 

    <!-- An unstemmed text field - good if one does not know the language of the field --> 
    <fieldType name="text_und" class="solr.TextField" positionIncrementGap="100"> 
     <analyzer type="index"> 
     <tokenizer class="solr.WhitespaceTokenizerFactory"/> 
     <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" /> 
     <filter class="solr.WordDelimiterFilterFactory" 
       protected="protwords.txt" 
       generateWordParts="1" 
       generateNumberParts="1" 
       catenateWords="1" 
       catenateNumbers="1" 
       catenateAll="0" 
       splitOnCaseChange="0"/> 
     <filter class="solr.LowerCaseFilterFactory"/> 
     </analyzer> 
     <analyzer type="query"> 
     <tokenizer class="solr.WhitespaceTokenizerFactory"/> 
     <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> 
     <filter class="solr.StopFilterFactory" 
       ignoreCase="true" 
       words="stopwords.txt" 
       enablePositionIncrements="true" 
       /> 
     <filter class="solr.WordDelimiterFilterFactory" 
       protected="protwords.txt" 
       generateWordParts="1" 
       generateNumberParts="1" 
       catenateWords="0" 
       catenateNumbers="0" 
       catenateAll="0" 
       splitOnCaseChange="0"/> 
     <filter class="solr.LowerCaseFilterFactory"/> 
     </analyzer> 
    </fieldType> 

    <fieldType name="edge_n2_kw_text" class="solr.TextField" omitNorms="true" positionIncrementGap="100"> 
    <analyzer type="index"> 
     <tokenizer class="solr.KeywordTokenizerFactory"/> 
     <filter class="solr.LowerCaseFilterFactory"/> 
     <filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="25" /> 
    </analyzer> 
    <analyzer type="query"> 
     <tokenizer class="solr.KeywordTokenizerFactory"/> 
     <filter class="solr.LowerCaseFilterFactory"/> 
    </analyzer> 
    </fieldType> 
    <!-- Setup simple analysis for spell checking --> 

    <fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100"> 
    <analyzer> 
     <tokenizer class="solr.StandardTokenizerFactory" /> 
     <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> 
     <filter class="solr.LengthFilterFactory" min="4" max="20" /> 
     <filter class="solr.LowerCaseFilterFactory" /> 
     <filter class="solr.RemoveDuplicatesTokenFilterFactory" /> 
    </analyzer> 
    </fieldType> 

    <fieldType name="sortString" class="solr.TextField" sortMissingLast="true" omitNorms="true"> 
     <analyzer> 
     <tokenizer class="solr.KeywordTokenizerFactory"/> 

     <filter class="solr.LowerCaseFilterFactory" /> 
     <!-- The TrimFilter removes any leading or trailing whitespace --> 
     <filter class="solr.TrimFilterFactory" /> 

     </analyzer> 
    </fieldType> 

    <!-- A random sort type --> 
    <fieldType name="rand" class="solr.RandomSortField" indexed="true" /> 

    <fieldtype name="ignored" stored="false" indexed="false" class="solr.StrField" /> 

    <fieldType name="point" class="solr.PointType" dimension="2" subFieldType="tdouble"/> 

    <fieldType name="location" class="solr.LatLonType" subFieldType="tdouble"/> 

    <fieldtype name="geohash" class="solr.GeoHashField"/> 

</types> 


<fields> 

    <field name="id" type="string" indexed="true" stored="true" required="true" /> 
    <!-- entity_id is the numeric object ID, e.g. Node ID, File ID --> 
    <field name="entity_id" type="long" indexed="true" stored="true" /> 
    <!-- entity_type is 'node', 'file', 'user', or some other Drupal object type --> 
    <field name="entity_type" type="string" indexed="true" stored="true" required="true" /> 
    <!-- bundle is a node type, or as appropriate for other entity types --> 
    <field name="bundle" type="string" indexed="true" stored="true"/> 
    <field name="bundle_name" type="string" indexed="true" stored="true"/> 

    <field name="site" type="string" indexed="true" stored="true"/> 
    <field name="hash" type="string" indexed="true" stored="true"/> 
    <field name="url" type="string" indexed="true" stored="true"/> 
    <!-- label is the default field for a human-readable string for this entity (e.g. the title of a node) --> 
    <field name="label" type="text" indexed="true" stored="true" termVectors="true" omitNorms="true"/> 
    <!-- The string version of the title is used for sorting --> 
    <copyField source="label" dest="sort_label"/> 
    <!-- content is the default field for full text search - dump crap here --> 
    <field name="content" type="text" indexed="true" stored="true" termVectors="true"/> 
    <field name="teaser" type="text" indexed="false" stored="true"/> 

    <field name="path" type="string" indexed="true" stored="true"/> 
    <field name="path_alias" type="text" indexed="true" stored="true" termVectors="true" omitNorms="true"/> 

    <!-- new fields for grouping pdfs --> 
    <field name="article_group" type="string" indexed="true" stored="true" termVectors="true" multiValued="false"/> 
    <field name="pdf_path" type="string" indexed="true" stored="true"/> 
    <copyField source="path" dest="article_group"/> 

    <field name="tid" type="long" indexed="true" stored="true" multiValued="true"/> 

    <field name="test_name" type="text" indexed="true" stored="true" termVectors="true"/> 

    <field name="taxonomy_names" type="text" indexed="true" stored="false" termVectors="true" multiValued="true" omitNorms="true"/> 
    <!-- Copy terms to a single field that contains all taxonomy term names --> 
    <copyField source="tm_vid_*" dest="taxonomy_names"/> 

    <!-- Here, default is used to create a "timestamp" field indicating 
     when each document was indexed.--> 
    <field name="timestamp" type="tdate" indexed="true" stored="true" default="NOW" multiValued="false"/> 

    <!-- This field is used to build the spellchecker index --> 
    <field name="spell" type="textSpell" indexed="true" stored="true" multiValued="true"/> 

    <!-- copyField commands copy one field to another at the time a document 
     is added to the index. It's used either to index the same field differently, 
     or to add multiple fields to the same field for easier/faster searching. --> 
    <copyField source="label" dest="spell"/> 
    <copyField source="content" dest="spell"/> 


    <!-- A set of fields to contain text extracted from HTML tag contents which we 
     can boost at query time. --> 
    <dynamicField name="tags_*" type="text" indexed="true" stored="false" omitNorms="true"/> 

    <!-- For 2 and 3 letter prefix dynamic fields, the 1st letter indicates the data type and 
     the last letter is 's' for single valued, 'm' for multi-valued --> 

    <!-- We use long for integer since 64 bit ints are now common in PHP. --> 
    <dynamicField name="is_*" type="long" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="im_*" type="long" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="iss_*" type="slong" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="ism_*" type="slong" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="ss_*" type="string" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="sm_*" type="string" indexed="true" stored="true" multiValued="true"/> 
    <!-- Normal text fields are for full text - the relevance of a match depends on the length of the text --> 
    <dynamicField name="ts_*" type="text" indexed="true" stored="true" multiValued="false" termVectors="true"/> 
    <dynamicField name="tm_*" type="text" indexed="true" stored="true" multiValued="true" termVectors="true"/> 
    <!-- Unstemmed text fields for full text - the relevance of a match depends on the length of the text --> 
    <dynamicField name="tus_*" type="text_und" indexed="true" stored="true" multiValued="false" termVectors="true"/> 
    <dynamicField name="tum_*" type="text_und" indexed="true" stored="true" multiValued="true" termVectors="true"/> 
    <!-- These text fields omit norms - useful for extracted text like taxonomy_names --> 
    <dynamicField name="tos_*" type="text" indexed="true" stored="true" multiValued="false" termVectors="true" omitNorms="true"/> 
    <dynamicField name="tom_*" type="text" indexed="true" stored="true" multiValued="true" termVectors="true" omitNorms="true"/> 
    <!-- Special-purpose text fields --> 
    <dynamicField name="tes_*" type="edge_n2_kw_text" indexed="true" stored="true" multiValued="false" omitTermFreqAndPositions="true" /> 
    <dynamicField name="tem_*" type="edge_n2_kw_text" indexed="true" stored="true" multiValued="true" omitTermFreqAndPositions="true" /> 
    <dynamicField name="tws_*" type="text_ws" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="twm_*" type="text_ws" indexed="true" stored="true" multiValued="true"/> 
    <!-- trie dates are preferred, so give them the 2 letter prefix --> 
    <dynamicField name="ds_*" type="tdate" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="dm_*" type="tdate" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="bm_*" type="boolean" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="bs_*" type="boolean" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="its_*" type="tlong" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="itm_*" type="tlong" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="fts_*" type="tfloat" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="ftm_*" type="tfloat" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="pts_*" type="tdouble" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="ptm_*" type="tdouble" indexed="true" stored="true" multiValued="true"/> 
    <!-- Binary fields can be populated using base64 encoded data. Useful e.g. for embedding 
     a small image in a search result using the data URI scheme --> 
    <dynamicField name="xs_*" type="binary" indexed="false" stored="true" multiValued="false"/> 
    <dynamicField name="xm_*" type="binary" indexed="false" stored="true" multiValued="true"/> 
    <!-- In rare cases a sfloat rather than tfloat is needed for sortMissingLast --> 
    <dynamicField name="fss_*" type="sfloat" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="fsm_*" type="sfloat" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="pss_*" type="sdouble" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="psm_*" type="sdouble" indexed="true" stored="true" multiValued="true"/> 
    <!-- In rare cases a date rather than tdate is needed for sortMissingLast --> 
    <dynamicField name="dds_*" type="date" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="ddm_*" type="date" indexed="true" stored="true" multiValued="true"/> 
    <!-- In case a 32 bit int is really needed, we provide these fields. 'h' is mnemonic for 'half word', i.e. 32 bit on 64 arch --> 
    <dynamicField name="hs_*" type="integer" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="hm_*" type="integer" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="hss_*" type="sint" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="hsm_*" type="sint" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="hts_*" type="tint" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="htm_*" type="tint" indexed="true" stored="true" multiValued="true"/> 

    <!-- Begin added fields to use features in Solr 3.4+ --> 
    <dynamicField name="points_*" type="point" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="pointm_*" type="point" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="locs_*" type="location" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="locm_*" type="location" indexed="true" stored="true" multiValued="true"/> 
    <dynamicField name="geos_*" type="geohash" indexed="true" stored="true" multiValued="false"/> 
    <dynamicField name="geom_*" type="geohash" indexed="true" stored="true" multiValued="true"/> 
    <!-- End added fields for Solr 3.4+ --> 
    <!-- Sortable version of the dynamic string field --> 
    <dynamicField name="sort_*" type="sortString" indexed="true" stored="false"/> 
    <copyField source="ss_*" dest="sort_*"/> 
    <!-- A random sort field --> 
    <dynamicField name="random_*" type="rand" indexed="true" stored="true"/> 
    <!-- This field is used to store access information (e.g. node access grants), as opposed to field data --> 
    <dynamicField name="access_*" type="integer" indexed="true" stored="false" multiValued="true"/> 


    <!-- The following causes solr to ignore any fields that don't already match an existing 
     field name or dynamic field, rather than reporting them as an error. 
     Alternately, change the type="ignored" to some other type e.g. "text" if you want 
     unknown fields indexed and/or stored by default --> 
    <dynamicField name="*" type="ignored" multiValued="true" /> 

</fields> 

<!-- Field to use to determine and enforce document uniqueness. 
     Unless this field is marked with required="false", it will be a required field 
    --> 
<uniqueKey>id</uniqueKey> 

<!-- field for the QueryParser to use when an explicit fieldname is absent --> 
<defaultSearchField>content</defaultSearchField> 

<!-- SolrQueryParser configuration: defaultOperator="AND|OR" --> 
<solrQueryParser defaultOperator="AND"/> 

</schema> 

меня есть и uniquekey (ID) и defaultSearchField.

Мой выход:

<response> 
<lst name="responseHeader"> 
<int name="status">0</int> 
<int name="QTime">8</int> 
</lst> 
<lst name="initArgs"> 
<lst name="defaults"> 
<str name="config">dataconfig.xml</str> 
</lst> 
</lst> 
<str name="command">full-import</str> 
<str name="status">idle</str> 
<str name="importResponse"/> 
<lst name="statusMessages"> 
<str name="Total Requests made to DataSource">1</str> 
<str name="Total Rows Fetched">3</str> 
<str name="Total Documents Skipped">0</str> 
<str name="Full Dump Started">2013-11-14 14:24:04</str> 
<str name="Total Documents Processed">0</str> 
<str name="Total Documents Failed">3</str> 
<str name="Time taken ">0:0:0.45</str> 
</lst> 
<str name="WARNING"> 
This response format is experimental. It is likely to change in the future. 
</str> 
</response> 

Я уже пробовал много разных вещей, и ничего не работало .. Пожалуйста, помогите мне!

Answer 1

В файле schema.xml под тегом поля вам нужно сохранить только те поля, которые присутствуют в вашей конфигурации данных.

<fields> 
    <field name="id" type="string" indexed="true" stored="true" required="true" /> 
    <field name="test_name" type="string" indexed="true" stored="true" required="true" /> 
</fields> 

И держать его просто, попробуйте изменить часть вашего данных-конфигурации, как этот

<document> 
    <entity pk="id" query="SELECT `id`,`test_name` FROM `test`"> 
     <field column="id" name="id" /> 
     <field column="test_name" name="test_name" /> 
    </entity> 
    </document> 

Answer 2

Пожалуйста, проверьте необходимые поля в вашем schema.xml потому что Solr не обрабатывать документ, который . отсутствует необходимый fieald (именно поэтому вы получаете строк, извлекаемых = # а всего документы обрабатываются = 0)

Так ваша ошибка в том, что вы определяете два обязательных поля (ID, ENTITY_TYPE), как показано ниже:

<field name="id" type="string" indexed="true" stored="true" **required="true"** /> 
<field name="entity_type" type="string" indexed="true" stored="true" **required="true"** /> 

и здесь, в вашей организации, вы просто с помощью поля «ID», но «ENTITY_TYPE» не использовалось:

<entity pk="id" query="SELECT `id`,`test_name` FROM `test`"> 
    <field column="id" name="id" /> 
    <field column="test_name" name="test_name" /> 
    <!-- the other required field (entity_type) is missing here --> 
</entity> 

так как вы индекс этого поля «ENTITY_TYPE» в вашей организации ,

или в вашем случае, вы должны удалить нужный вариант из его определения в schema.xml

Answer 3

вы пропустили необходимые поля. Обязательные поля должны всегда использоваться/назначаться.