Version 2

    Summary

    The following error may periodically happen when rebuilding the content search index:

    #### items failed during the last content index rebuild.

     

    Screen+Shot+2016-03-15+at+3.55.44+PM.png

     

     

    The error is caused by orphaned data in the database and can be resolved by locating and deleting the orphaned rows directly from the database.

     

    This is an action that can only be done by On-Premise customers who have direct access to the application database. If you are using a Hosted or Cloud site and you are seeing this error then you will need to file a new support case. A support engineer will then assist in resolving the issue for you.

     

    Versions

    • Verified for version: All

     

    Details

    Content is processed in batches when rebuilding the search index, and if a piece of content fails to be indexed at any point within the batch then the entire batch is marked as bad and is reported at the end of the rebuild.

     

    The system property of search.rebuild.gather.blockSize controls the size of the batch. By default this is set to 400. In large sites, the search.rebuild.* system properties are often changed to speed up a large rebuild. Therefore, if the batch size has been increased you will see higher numbers in the "items failed..." error message.

     

    The index rebuild gatherer task asks all content types for a range of ids to index. In the case of comments and bookmarks the source content is also queried and indexed. e.g. objectID for documents and markedID for bookmarks. There is no referential integrity that guarantees the source document to exist, which can result in an exception bring thrown during the gather phase.

     

    To locate the failures, search sbs.log for this line:

    $ grep  -i "INFO com.jivesoftware.community.impl.search.RebuildIndexTask - marking failed rebuild entity" sbs.log | wc -l

    800

     

    These entries will show the objectType and the objectID

    2016-03-16 19:01:38,283 [IndexRebuildGatherer-8] [1:admin:REGULAR] INFO com.jivesoftware.community.impl.search.RebuildIndexTask - marking failed rebuild entity: EntityDescriptor{objectID=35401, objectType=105}

    objectType=105 is a document and there is 800 of these INFO statements, we can probably assume that there one or many pieces of content that are triggering the RebuildIndexTask to fail.

     

    Searching for RebuildIndexTask ERRORs further down in the logs,

    2016-03-16 19:01:38,448 [IndexRebuildGatherer-8] [1:admin:REGULAR] ERROR com.jivesoftware.community.impl.search.RebuildIndexTask - Could not find a document backchannel's source document: 15595

    java.lang.IllegalStateException: Could not find a document backchannel's source document: 15595

            at com.jivesoftware.community.DocumentBackChannel.getDocument(DocumentBackChannel.java:151)

            at com.jivesoftware.community.DocumentBackChannel.getContainerType(DocumentBackChannel.java:131)

            at com.jivesoftware.community.impl.search.provider.CommentIndexInfoProvider.loadContainerInfo(CommentIndexInfoProvider.java:376)

            at com.jivesoftware.community.impl.search.provider.CommentIndexInfoProvider.getContent(CommentIndexInfoProvider.java:262)

            at com.jivesoftware.community.impl.search.ContentSearchIndexManagerImpl$RebuildDataGatherer.gather(ContentSearchIndexManagerImpl.java:533)

            at com.jivesoftware.community.impl.search.RebuildIndexTask$GathererTask.doRun(RebuildIndexTask.java:458)

            at com.jivesoftware.community.impl.search.RebuildIndexTask$GathererTask.run(RebuildIndexTask.java:410)

            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

            at java.util.concurrent.FutureTask.run(FutureTask.java:266)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

            at java.lang.Thread.run(Thread.java:745)

    So, RebuildIndexGatherer had a problem gathering comments on document backchannel with id 15595, and potentially others.

     

    The general process is to collect the IDs and confirm that those documents don't exist in jiveDocument:

    select count(*) from jiveDocument where internaldocid IN (15595, 3602, 4588, 4906, 8479);

     

    If confirmed, then go ahead and delete the document comments from jiveComment where that belong to the offending documents.

    delete FROM jiveComment WHERE objectType = 129 AND objectID = 15595;

     

    The quick way to do this for orphaned content:

    DELETE FROM jiveComment WHERE objectType = 129 AND objectID NOT IN (select internaldocid from jiveDocument);
    DELETE FROM jiveBookmark WHERE markedType = 1 AND markedID NOT IN (SELECT threadID FROM jiveThread);
    DELETE FROM jiveBookmark WHERE markedType = 38 AND markedID NOT IN (SELECT blogpostid FROM jiveBlogPost);
    DELETE FROM jiveBookmark WHERE markedType = 102 AND markedID NOT IN (SELECT internaldocid FROM jiveDocument);
    DELETE FROM jiveBookmark WHERE markedType = 3227383 AND markedID NOT IN (SELECT ideaid FROM jiveIdea);
    DELETE FROM jiveBookmark WHERE markedType = 18 AND markedID NOT IN (SELECT pollid FROM jivePoll);
    DELETE FROM jiveBookmark WHERE markedType = 1100 AND markedID NOT IN (SELECT vidid FROM jiveVideo);
    DELETE FROM jiveBookmark WHERE markedType = 801 AND markedID NOT IN (SELECT id FROM jiveexternalurl);
    DELETE FROM jiveBookmark WHERE markedType = 1464927464 AND markedID NOT IN (SELECT userstatusid FROM jiveUserStatus);
    DELETE FROM jiveBookmark WHERE markedType = 96891546 AND markedID NOT IN (SELECT eventid FROM jiveEvent);

     

    Once the orphaned records are removed you will need to do a full context index rebuild.

     

    You may be presented with the option to do a partial content index rebuild, but this will not successfully complete and will continue to result in errors:

    2016-03-17 01:37:38,242 [IndexRebuildGatherer-8] [1:admin:REGULAR] ERROR com.jivesoftware.community.impl.search.PartialRebuildIndexTask - Could not find a document backchannel's source document: 15595java.lang.IllegalStateException: Could not find a document backchannel's source document: 15595