Crawl doesn't work and Query server is not responding; Search works only for old (crawled) contents

We had this search issue in January' 2008 and here is the description,

We have following three servers with MOSS 2007 STD deployed in SharePoint server farm,

App01- Query Server
App02 - Index Server (We are using this server as dedicated web front end computer for crawling)
SPDBServer - SQL 2005

Search was working fine till 01/05/2008 and it stopped working after this date only for new contents meaning search is still working fine for the old contents which were added before 01/05/2008 but not for new contents added after 01/05/2008.

We have been working to find out reasons why did that happen? We have found following event ids on Query server,

Event ID:10042 - A new query machine, 'App01', has been added to the query rotation based on changes to farm topology.

Event ID:27745 - The following information is part of the event: #50071: Unable to connect to the database SharePoint_Config on SPDBServer. Check the database connection information and make sure that the database server is running..

Event ID: 7888 - A runtime exception was detected. Details follow.
Message: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 0 - Not enough memory is available to complete this request)

We had some database backup client issue on database server but after fixing that issue and rebooting all the three SharePoint servers, we noticed that crawl was not responding. Last incremental crawl, started on 01/05/2008 did not stop even on Monday (01/07/2008).

We have incremental crawl scheduled at every 6 hrs. everyday and full crawl scheduled on every saturday.

After coming back to work on Monday 01/07/2008, we tried search and noticed that it was working only for existing contents(added before 01/05/2008) and not for all the new contents, which were added after 01/05/2008.

When we check Search settings, it says "Query server not responding" and we are unable to stop/pause the incremental crawl, which was started on 01/05/2008. Only way to stop that crawl is to stop Office SharePoint Server Search service from services.msc on both App01 and App02 server.When we stop the crawl this way and start it again, it goes on and on and never stops. We have tried to start both full and incremental crawl this way but that didn't help.

Only resolution to this issue, I have found is to readd App01 - Query server to the farm from SharePoint Central Administration however I am still wondering, if that will fix our "Query Server not responding" error. As per our research, it seems few other people also have noticed this issue for some unknown reasons and if SharePoint doesn't have any operational errors, we can safely ignore them. Microsoft is aware of this isse and they are expecting to get this resolved in SharePoint 2007 Service Pack 1.

I was also thinking of resetting all crawled contents and rebuilding the content index. I was affraid, if resetting the crawled contents will not regenerate/rebuild the new content index. If crawl will not work after resetting the crawled contents then SharePoint search will be completely down and it will stop working even for old contents.

Has anybody tried this KB?
http://support.microsoft.com/kb/925609

Can we move content index(search catalogs) from shared location to some other location on Query server and then try resetting crawled contents? Can we move content index back to its original location if resetting crawled contents will not generate a new content index or it will not propagate it to Query server?

If above KB article gives us the flexibility of reusing content index in case of resetting crawled contents fails to build a new content index or propagating it to Query server, we would like to try it first before we can try readding Query server to the SharePoint server farm (Stop and then restart Office SharePoint Server Search service for Query server from SharePoint Central Admin)

Does anybody know an appropriate solution to fix this issue? Search works only for old (Crawled) contents and not for new contents, Crawl never stops (hangs) and there is "Query server not reponding" message on search settings so basically it is not propagating the content index.

Solution:

Here is how we fixed this issue in our environment. This issue might be specific to our environment and this resolution may not work for similar issue in the other environment.

- We noticed few memory leaks in SQL Server 2005. We(our Windows Server team with SQL Administrator) fixed those issues first however fixing memory leaks in SQL didn't help us to fix search issues in SharePoint and we still had "Query Server not responding" error in SharePoint and Search was not working.

- We created a new location for content index propagation on Frontend Server, which is also a Query Server.
Note: Do not move existing crawled contents from old propagation location to a new location manually. In fact, SharePoint does it automatically once you create a new location for content index propagation with the following command. It is strongly recommended that you start a "Full Crawl" after this to rebuild content index for updated contents.

- Ran following command to propagate search index to the new location,

stsadm.exe –o osearch –propagationlocation index file path

Please allow some time to move existing content index from previous location to a new location. You may see few known events in event viewer while SharePoint is moving content index to a new location however you can safely ingnore them. Please check status on Search Administration (SSP => Search Settings) page and for Query Server response. You can start a Full Crawl after few minutes once you see Query Server starts responding in Search Settings.
This has fixed our SharePoint search issue and it works fine now.

- Hope this will help others to resolve similar issue. Please make sure you do not have any SQL Server issues before you troubleshoot it in SharePoint.

References:
http://support.microsoft.com/?kbid=925609http://blogs.msdn.com/pdesai/archive/2007/09/27/moss-2007-search-scope-error-query-index-server-event-log-error-how-to-fix-it.aspx

Technet Post on this issue:
http://forums.microsoft.com/technet/ShowPost.aspx?postid=2706585&siteid=17