Why the Moss crawler might not working?

This week I had to fight against our indexing and querying server in our new SharePoint farm. It is a farm environment consisting of several web-front ends and this poor little guy :). At the beginning, as usual, everything seemed to work as expected and we started to create our first content sources for indexing. You know what… my long and apparently endless journey begun exactly at this moment:

The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured

At the beginning this error was misleading. I double-checked the firewall settings and found no issues. The machines are communicating with each other without issues. Now you might be thinking… what else?

Before I start to tell you how I solved this and another issue, I would like to give you an advice:

Advice: before banging your head against the wall or trying to kill some developers near you, do following:

1) use RDC to connect on your indexing and crawling server

2) open a browser and try to open the SharePoint site you want to crawl by using the credentials of your search account

Most of the issues will popup during this moment:

1) You see if the search account has not enough rights to access your page

2) You see if there are missing assemblies or errors in the web.config

Only because your site is working on your web-front ends, this does not mean that it works on your indexing and querying machine. If some of the above cases occur, probably, you know what are your next steps.

In my case it was a custom WebPart developed in our team. The WebPart was working like a charm when it was not crawled. When the pages were crawled, an exception was caught causing the misleading error message you saw before. Probably, you are using some kind of customizations that are breaking the indexer. Try to remove them from your site for testing purposes or debug the code with Visual Studio.  

Still here? Damn… . After the correction of the bug I was also thinking that everything should work. But I was wrong. And voila’, the next error occurred:

Error in the site data web service

Interestingly, the improved WebPart was not throwing the exception and the pages worked when browsed on the indexing server. After hours of research I finally found the solution for my problem. It was again a customization of our development team. It is a HTTPModule that is responsible for the URL rewriting of our SharePoint pages. The indexer seemed to hate our redirections with the Response.Redirect command. I simply removed the automatic redirection on the indexing server. Afterwards, everything worked fine.

Naturally, this special case occurred only because of our custom code. But I think that we can learn from it anyway. If some of the errors above are occurring, you probably forgot to update some code or you are missing some dlls on your indexing server. So try to debug and test your pages on the server to see what your issue might be.

Hope this helps. Happy eastern

Patrick