Database Migration breaks WSS/SQL Server Full Text Search

So here’s a little-known issue with SharePoint and Full Text Search:

On Joel Oleson’s blog (if you don’t know who this guy is and you’re in SharePoint Operations, find out quickly. Aside from Bill English [the man, his blog], he’s the other Man in SharePoint Managment/Operations – both of these guys regularly present at TechEd conventions), I found a blog entry about how, if your Full Text Search (that’s the one that works in Windows SharePoint Services, and is provided through the back-end SQL Server) isn’t working, and you migrated the database from a different server, the reason would be, possibly/probably, that you migrated the database from a different server.

Yes, I mean either with SPSBackup.exe (which should be your friend by now if you’ve been doing a lot of this and you want to keep the SharePoint Portal Search Index between migrations) or normal SQL Backup/Restore and file operations.

So anyway, there are a couple of stored procedures in the blog you should use to restore your Full Text Search. As far as I can tell, these are either SQL standard stored procedures, or, more likely, SharePoint-installed stored procedures.

Also, rather like part of my previous blog entry, the fix is essentially “turn it off, turn it on”, with some curve-balls in there if things don’t go as expected. It turned out to be thorough enough for us, so maybe it’ll be thorough enough for you.

Since I had to puzzle the blog entry out a little bit, I’ll write what I understood of how to do this here.

  1. Mr. Oleson recommends restarting your SQL Services, but we didn’t find this completely necessary.
  2. Run the following stored procedure: exec proc_DisableFullTextSearch
    1. If you get an error about there not being a Full Text Catalog, then run the following in SQL Query Analyzer and start Step 2 again:

      USE [databasename]
      sp_fulltext_database enable

      Where [databasename] is the name of the _SITE database you’re having the issue with. [But don’t actually type the [] brackets in there or your geek compatriots will laugh at you.]

  3. Run the following stored procedure: exec proc_EnableFullTextSearch

So that should be it. The procedure, as I said, is mostly just turn off, turn on again.

Full Text Search and Account Permissions

This is a more extended writeup of running Windows SharePoint Services 2003 and SQL Full Text Search on a Database box where Local Administrators (BUILTIN\Administrators) don’t have System Admin access in SQL Server 2000. (I mentioned this briefly in the Changing SharePoint Service Accounts article.)

Essentially, you’ll run up against this security policy requirement in some environments. It’s a sensible policy to make in situations/operations where the Local Administrators (of whom many are also Domain Administrators) are folks who are different from the folks who own, run and are responsible for the SQL Servers.

Part of the motivation for this separation is, of course, political. In some organizations you’ll find that folks in one team don’t want to share permissions/rights with other teams who aren’t directly responsible for the upkeep or maintenance of the bit of the sandbox they have dominion over.

The Sensible Computer Security Policy reason is the principle of Least Privileges. When the question, “Do these people/does this group need permissions to this resource?” is answered “No.”, then the principle of Least Privileges dictates that they not be given the access they don’t need. This Security Principle falls under the overall category of Risk Management. The fewer potential risks (i.e. fewer accounts sitting around waiting to be hacked that have permissions they don’t necessarily need), the fewer potential security vulnerabilities sit around waiting to be exploited by Joe Q. Attacker.

It should be noted that in the annals of computer attackers, the long-neglected account that just happens to be a local or domain administrator and just happens to have a really easy to guess password is the holy grail, and almost every computer system has at least one. So do what you can to manage your risks and reduce the number of holy grails that attackers can use to compromise your system.

Anyway, so for whatever reasons, you’ve decided that you wish to implement the policy that Local Administrators on the SQL Server are not allowed to be System Administrators (aka sa) within the SQL Server/Application itself. Note that while it appears that Microsoft “supports” this configuration, it’s not specifically allowed for in Microsoft’s relevant Knowledge Base articles, so if you do go this way, be on the lookout for potential complications. See that other article I mentioned and linked to above for an example of an unexpected consequence.

If you remove BUILTIN\Administrators from your SharePoint 2003 server’s SQL Server Logins, or remove the sa permissions from that group, you will hose up your Full Text Search in SQL Server, which of course (say it with me) will screw up your Full Text Search in your Windows SharePoint Services 2003 sites. (Because Windows SharePoint Services 2003 uses SQL Full Text Search to do its searching.)

How do you fix this?

According to KB Article 317746, if you don’t wish to add BUILTIN\Administrators back to the SQL Server Logins, you still have an out. You must:

  • Add the System Administrators Server Role to the account you are using as the Service Account for SQL Server.
  • Add the Local System account (NT AUTHORITY\System) to the SQL Server Logins.
  • Add the System Administrators Server Role to the Local System account (NT AUTHORITY\System).

You should not have to restart SQL Server after making this change. But you may also need to fix Full Text Search for other reasons, which I will elucidate in a (shortly to follow) article.

Old Security Articles

It’s on my professional site (where my resume would be if I were looking hard for another job), but I wrote a bunch of articles (Security notes, Best Practices, etc.) for Adobe, back before it was Adobe, and before even it was Macromedia, but when it was Allaire, and I was Product Security Manager/Security Response Team Coordinator there.

It was a nice job. Had some drawbacks in that QA/Security reported to Marketing on the Organizational Chart instead of, you know, IT, but it was a good job

I note, looking at these articles now, that the ones that are still credited to me (I wrote a number of security advisories that I’ll try to find too) are credited to me as a Consultant, though at the time I was a salaried employee with the title “Product Security Manager” or “Security Response Team Coordinator” instead. I am still a bit more pleased with being credited as a consultant. It is a title not undeserved.

Anyway, if you want to look at the old writing (from 2001/2002), here are the links (these open in your same window):

Here is a sampling of the Security Bulletins I wrote (None of them are credited) or significantly updated (I was there from 2001 – 2002):

Making Office document open in Office App instead of in an Office ActiveX Control in IE

I just wrote up this little Frequently Asked Question at work:

Assumptions:
  • You have Microsoft Office 2003 installed.
  • You are working with Windows XP Professional as your Operating System.
This is a setting that needs to be made/verified in your Windows settings.
To set the documents to open in their appropriate application:
  1. Open My Computer on your Desktop.
  2. Choose the Tools menu item, then choose Folder Options….
  3. In the Folder Options popup window, choose the File Types tab.
  4. In the Registered File Types list box, choose the file extensions for the application file types you’re interested in verifying/changing (i.e. DOC for Word, XLS for Excel, PPT for PowerPoint, etc.). Once the proper file type is selected, click the Advanced button.
  5. In the Edit File Type pop-up window, uncheck the Browse in same window checkbox.
  6. Click the OK button in the Edit file Type pop-up window.
  7. Click the Close button in the Folder Options pop-up window.

You’re done!

So the flip side is that if you want the document to open up in IE, you check the Browse in same window checkbox.

Sometimes this setting does/does not work. I’m still working on that bit, but also waiting to hear from the users a bit more about the versions of Office/OS they’re working with.

MOSS 2007 and Search

Reading the 7 Development Projects with the 2007 Microsoft Office System and Windows SharePoint Services 2007 eBook (Microsoft Press) yesterday, I was pleased to read that we’re moving off of the sometimes wildly inaccurate results-giving Full Text Search technology that SQL Server provides for Windows SharePoint Services 2003 and using the same technology the Portal is using in the SharePoint 2003 version for both Portal and WSS in 2007.

Just a little note. I’m pleased as Punch. I hope the administration/configuration has been tweaked to be a bit more intuitive, though.

Also, there appear to be a bunch of other books available to buy about 2007 these days. I should hie to a bookstore.

Changing SharePoint Service Accounts, Permissions and Troubles Found and Conquered Therein

(Note: Links to KB Articles open in new windows)

So I have a client who is security conscious enough to ask that we make SharePoint work even when the servers’ Local Administrators (members of BUILTIN\Administrators) are not members of the System Administrators Server Role in SQL Server. This is the default configuration for SQL Server – the BUILTIN\Administrators group should have a checkbox enabled next to the “System Administrators” role in the Server Roles tab of the Login properties for SQL Server.

After this helped break Full Text Search (KB Article 317746) and we fixed it, all was well and good, or so we thought.

It’s not completely obvious, but in SharePoint Portal Server 2003 running on Windows 2003 Enterprise SP1 and above (KB Article 555309), it’s required that the SharePoint Service Account be part of the servers’ (in the farm) Local Administrators. This would appear to convey the idea that SharePoint Service Accout should also be a System Administrator (role) in the SQL Server that hosts the configuration and content databases.

I believe, by running into it head first, we have corroborated this assertion.

Moving from a bad situation (where all three of our environments – Dev/Pilot, QA and Production – are running on the same service account whose password will expire in about 2 months), to a better one (we plan 2 service accounts for each environment), means we have to change the service accounts for both SQL Server services, and for SharePoint services in each environment.

(By the way: It’s actually recommended that if possible, service accounts associated with Enterprise servers like SharePoint be set with passwords that don’t expire. If your security policy requires it, you can then plan to change the password on a regular basis, but you won’t get caught out if you simply don’t have time to do it. The slightly more antagonistic policy is to make the automatic experation/reset loom over your operations people like an unassailable threat that motivates them to change the password regularly, manually, before the deadline. If you live in a more antagonistic sort of place like we do, at least have the common decency not to keep folks from changing the password early. Making the password change be required to happen on a certain day is the kind of sadism that breeds MAD IT CONSULTANTS! MAD, I TELL YOU!)

In doing so, we ran up against the problem of not having the SharePoint service account be a System Administrator in the Server Roles in SQL Server. We followed the KB Articles (SharePoint: KB Article 837813, SQL: KB Article 283811 – For SQL we actually used the SQL Enterprise Manager – it’s the easier way – try SQL Books Online for guides) for changing the service accounts, and when we got to checking the SharePoint Central Administration (to finish up the account changing process), instead of the SharePoint Central Administration, like we thought we’d get, we got a prompt to disconnect from the configuration database.

So, in a fit of troubleshooting pique, we followed the prompt and got ourselves into a deeper hole. Because at the instant that we disconnected, the server we were working on got out of synch from the rest of the server farm.

And how did we get out?

First, the primary fix is to make sure that if you’re not going to give all the Local Administrators System Administrator access in SQL Server, at least make sure that the SharePoint Service Account has System Administrator access. That’s the first thing.

Second, we need to fix the mess. If we hadn’t disconnected from the configuration database when the opportunity presented itself, restoring the System Administrator access for the service account in question would fix the whole kit and kaboodle (requiring, perhaps, closing and opening IE, or forcing a cache refresh [hold down Ctrl and click Refresh button] to see the results in IE).

Instead, since we disconnected the database at probably the worst time, the server that disconnected thinks it’s disconnected, but the rest of the farm doesn’t. Weirdly enough, the broken server acts like it thinks it’s both part and not part of the farm. I managed to get the server to the point where it serves content just fine from WSS, but SharePoint Central Administration thinks it isn’t connected to the configuration database (i.e. it’s broken).

When you try to connect to the existing configuration database, you get a weird and inappropriate error. Usually you get the error, “Unexpected error occurred.” (This error actually tells you a lot. If you google for it, you’ll find that most of the issues associated with this symptom are to do with network connectivity – this isn’t utterly bizarre – from SharePoint’s perspective, a SQL permissions error can look a lot like a network error, and whatever’s bunged up in the broken synchronization is also probably interfering and giving SharePoint weird results, which probably just go in the network bucket, because SharePoint can’t figure out where else to put it.)

Weirder still, it seems like the whole of the problem exists somewhere in the broken server’s configuration, with none of the actual problem on the working parts of the farm at all.

Why do I say this? Because the fix is really simple:

  1. Create a new configuration database with the broken server.
  2. Disconnect from the new configuration database.
  3. Connect to the old configuration database.
  4. You probably want to delete the new, unused configuration database too, but the things are pretty small, storage-wise.

If the synch error really were partly on the original configuration database and partly on the broken server, this shouldn’t fix the problem, because it would still exist in some part on the original configuration database. But in fact, this works.

Now, if, like us, you were in the middle of changing accounts for your farm, just continue on your merry way. Since you only need to make that System Administrator Server Role for the SharePoint Service Account fix once (assuming you’re only using one SQL Server in the farm), you should be good to go with the rest of the servers in the farm.

Since I should be doing this tomorrow in our QA environment (Dev’s down right now), I’ll let you know if there’s more to know, when I’ve figured it out.

Wish me luck!

Introductory Post

Hi folks!

I already have a Livejournal, where posts of a more personal, less technical nature, will go. My current work cycle is very focused on System Engineering/Architecture/Maintenance with SharePoint in mind, and may be devoid of .NET tips for quite some time. I also don’t wish to create posts just to post.

So when things come up that I think are new or niggly enough that they’re not in MSDN (I will have checked) or Technet (same) or on other blogs, I’ll try to post them here. I already have one post in mind, which is about permissions on SharePoint, the back end SQL Server, and the arcane task of changing the service accounts in both while a farm is still up and running.

Having run into some issues, I’m here to tell you that it’s tricky but possible to even recover from a point where Microsoft’s recommendation would be to just throw it all out and reinstall SPS/WSS completely.

Anyhow, this post is to talk about my goals (done), and plans (will do). It’s also to note that being this is a new WordPress blog, it’ll be a bit before I install the really heinously aggressive spam control measures, but I’ve already specified that comments must be held and approved, so if you post a comment, please give me time to approve it before it shows up.

See y’all in the blogosphere!