Getting MOSS 2007 RTM Backup to work when your SQL Database is one you don’t control

In this post I will be somewhat generic about my machine names and account names. This is partly to keep me in form and not thinking too specifically about my special situation and partly for security reasons. I don’t believe in security through obscurity but I also don’t believe in making it extra easy for an attacker to get the first steps of the puzzle for free.

Configuration of my MOSS 2007 RTM install in development:

  • 1 Server that serves all databases to all development environments (Call it SQLDEV1).
    • I have a specific instance (Call it MOSD) in which I can put all my various MOSS databases and my domain user account (Call it DOMAIN\myuser) is a security admin and dbcreator in the Server.
    • Since I used my domain user account (DOMAIN\myuser) to give all the various configuration services and app pools and datbases an ID during initial setup/configuration of MOSS 2007, I (DOMAIN\myuser) am also dbo on all the MOSS databases in the instance.
    • Additionally, it should be noted that on SQLDEV1, the MSSQLSERVER service is running as one domain user account (Call it DOMAIN\dbuser1), and the instance (MOSD) is running as another (Call it DOMAIN\dbuser2).
  • 1 Server that has all the MOSS 2007/.NET 2.0/.NET 3.0 WFX installs (Call it MOSSDEV1).
    • On this machine, my user account (DOMAIN\myuser) is a local administrator. I also have some additional rights that allow me to log into the machine remotely with a terminal services client.
    • It was a precondition to my getting these rights that I set up the MOSS 2007 install so that all configuration aspects that required local admin access be set up with my domain account’s ID (DOMAIN\myuser). I know this isn’t ideal in some respects, but it seems to work okay in this development environment. This will be the main factor in making this blog entry not be entirely helpful in debugging YOUR configuration problem, but I hope it’ll help anyway.

With no additional special steps, I couldn’t get MOSS 2007’s Operations interface to successfully back up the Farm (via the Central Administration: http://your_server:your_port/_admin/Backup.aspx). I kept getting some progress, but each and every actual site collection failed with a failure message ending in “Operating system error 5(Access is denied.). BACKUP DATABASE is terminating abnormally.” [It should be noted that I was previously getting an error 3, which appeared to be tied to using a non-UNC path in the Backup Location field.]

The steps necessary for me to get it all working were:

  1. Create a shared folder on the system for where you want the backups to go. (i.e. \\MOSSDEV1\Farm Backups\).
  2. Add full control permissions to this share for all three accounts: The service account for MSSQLSERVER on the database server (DOMAIN\dbuser1), the service account for the database instance hosting the content/configuration databases (DOMAIN\dbuser2) and the service account running the MOSS 2007 application pool (my guess of the most likely suspect in this operation – DOMAIN\myuser).
  3. Use the share name you created in the Backup Location text box in the 2nd step of the Backup configuration in Central Administration (i.e. \\MOSSDEV1\Farm Backups\)

I had been using the Admin’s traditional UNC path (i.e. \\MOSSDEV1\d$\Farm Backups\) without explicitly sharing a folder, but that tripped me up, because the SQL Server service account IDs were not Local Admins on the MOSS 2007 box (i.e. MOSSDEV1).
Also it took a bit of digging and asking my DBAs about the service account identities for the SQL Server.

My main reference is about SQL Server Backup in the Microsoft KB #255235.

Updated older MOSS 2007 Beta2TR Install article

After finally managing the install and setup correctly in my own deployment last week, I just updated the article on this blog with the newest information I needed to completely the install properly and make everything ship-shape (or close enough for jazz).

I have been negligent – bullet updates, but I’ll get around to the major stuff later

Since I fully expect next month to be a slow month, I should be able to catch up a little.

Anyhow:

  • I am installing the Release bits of Microsoft Office 2007. I don’t know if I’ve already plugged CCleaner but I’m doing so again. I needed it because Office 2007 Beta 2 Technical Refresh didn’t uninstall entirely cleanly. An add-on I’d installed after the original install had to be manually removed, but it didn’t show up in my Add/Remove Programs, so CCleaner was instrumental in my being able to find an uninstall the bugger so I could go ahead with the install of the Release version.
  • It turns out that the extended problems I had properly creating the Shared Service Provider portion of MOSS 2007 were due to two factors:
    • I had neglected to complete the MOSS 2007 Beta 2 TR install properly. I’ll go back to that article and add the details in, but instead of running the configuration wizard right away, I should instead have uninstalled Windows Workflow Framework from Add/Remove Programs, installed the .NET 3.0 Framework RC bits and then run the connfiguration wizard.
    • I wasn’t thinking about permissions and rights properly so was creating the app pool for the Web Application that was to support the SSP with Network Service as the ID, which of course has a different PID/GUID on each machine so wasn’t mapping to the Network Service ID on the database server (2-server setup). What I should have done was create the app pool with a domain account ID that had sufficient perms on both boxes and on the SQL Server itself. It never ceases to amaze me how my mind will just drop stuff. This stuff holds for SharePoint 2003 too and I know that cold, but I just didn’t make the leap to apply it to my knowledge of MOSS 2007. Duh.
  • So I need to blog permissions articles that have been popping up on Technet/MSDN lately.
  • I also need to update on my/my company’s progress in fixing (or trying to fix) the Full Text Search in our production deployment of SharePoint 2003. Client still not interested in calling Microsoft Product Support Services. Now it looks like it might have to do with the Cluster configuration and the FTDATA folder. If it isn’t that, not only am I, but my company is tapped out and it is totally time to stop playing political games and djust call Microsoft PSS.
  • There are some links I found to training materials that I’ll also blog (I’ve been doing research on behalf of my client’s Training department).
  • I’ll be working on customizing my company’s portal soon, and doing a little mini-app with a guy based in the Richmond office, so we’ll see how well the development/customization process on MOSS 2007 collaborates. More updates there, hopefully by next week.

Anyway, been terribly busy, too busy, perhaps, to blog, but I’ll try to return to it, because taking notes is important to me, and putting it here means I can find it whereever I have Net access, and maybe it’ll help out other folks too.

What is wrong with you Microsoft Only people? – Checksum utilities for verifying large files

So today I am downloading the 2007 Office Beta 2 installs (and whoever heard of paying $1.50 for 5 download tries? I sort of understand, but if it were really just covering bandwidth fees, I should think it would be a lot lower). I note that the download listings/product key e-mails do not come with checksums for these large files.

The files are all in the 75 MiB – 250 MiB range. In UNIX-land, people would as part of the normal posting process just provide checksums. But in Windows-land apparently this is not done. Why not?

Checksums are extremely useful for making sure that the bits you expected to transfer over the network are the ones you got. You can see that this would be useful for both file content verification in the sense of “did I lose any bits along the way that would corrupt my install and can I know it before trying to install and have it fail?” But it’s also useful in the sense of making sure that the bits you want me to download are the same ones I want to get, and assuring that no 3rd party attackers did a man-in-the-middle attack, substituting trojan horses and other nasty things into the install instead. Okay, granted, private key encryption technology would be better than a simple checksum, but a checksum would still be better than nothing, which is what I get when I pay $1.50 to download the damned things.

With that in mind, let me introduce you to NullRiver’s winMd5Sum. This is a free and easy to use utility that allows you to create MD5 checksums on files and also to compare pre-generated checksums to the ones you generate on your end to check the download. Go use it. You’ll like it. While you’re at it, tell your download hosts (Microsoft too, please) that you’d like it if they’d start using it or some similar process to help you verify your large file downloads.

For posterity, I’m going to post the MD5 checksums I’ve got so far for my Office 2007 Beta 2 downloads (from Microsoft via the License Technology Group) [This assumes that each binary isn’t especially constructed for each product key – I guess we’ll see]:

  • Microsoft Office Forms Server 2007 – OFS32-EN.IMG – 14,796,544 bytes – MD5: 4ba65c890b6c86158666b41c3652d2bb
  • Microsoft Office Groove 2007 – OG-EN.EXE – 220,111,048 bytes – MD5: ba497c8610ae774b4f3af92755e83bf7 [Works fine]
  • Microsoft Office OneNote 2007 – OON-EN.EXE – 231,814,328 bytes – MD5: 95750f6b8c48c602b39c4b1271913398 [Works fine]
  • Microsoft Office Outlook 2007 with Business Contact Manager – BCM-EN.EXE – 252,769,672 bytes – MD5: 9cb44475cfbbbebb7c84eced9ef6e437 [Works fine]
  • Microsoft Office Professional Plus 2007 – OPPLUS-EN.EXE – 461,881,224 bytes – MD5: 7fc65a38b6bd9dce0563afea2c5b9a93 [Works fine]
  • Microsoft Office Project Professional 2007 – OPP-EN.EXE – 210,237,736 bytes – MD5: 50c1f917637de95c9aa72114e6385acb [Works fine]
  • Microsoft Office SharePoint Designer 2007 – SPD-EN.EXE – 236,994,544 bytes – MD5: 94fe6551b52ef1d38556d76677966073 [Works fine]
  • Microsoft Office SharePoint Server 2007 – Enterprise – SPS32-EN.IMG – 308,555,776 bytes – MD5: 0db4750dd73faca499fc5df95c7f63b3
  • Microsoft Office SharePoint Server 2007 for Search – OSS-EN.IMG – 231,387,136 bytes – MD5: c1c2b5ed9c0a31c48fb59afe3fb29919
  • Microsoft Office Visio Professional 2007 – OVP-EN.EXE – 293,966,312 bytes – MD5: 4259e1f323509e8392143e20416490f5 [Works fine]
  • Microsoft Windows SharePoint Services [v3] – SharePoint_setup.exe – 78,849,224 bytes – MD5: 51cd9f824bb5b6bfc90b96f0de956a1b

This is the complete list of the downloads I paid for.

Also, FYI, here is the link for the Beta 2 Technical Refresh download.

Here’s the file info for that download:

  • Microsoft Office 2007 Beta Two Technical Refresh – office2007b2tr-kb000000-fullfile-en-us.exe – 518,733,856 bytes – MD5: 9ad077c27fb279516b8636e43c3e0463 [Works fine]

I haven’t verified that all of these files work, but I have verified that the total file size is the same as was originally reported when I initiated the download, which is as close as you can get without MD5 or other checksum tools. I’ll note by striking the item out if for some reason the download is corrupt. Also, when I say [Works fine], I mean that it installed fine with all options installed to run on the drive. I won’t say that the actual programs installed worked fine, as they are in Beta.

Product Testimony – Acronis TrueImage 9.x

I don’t do Product plugs/testimonies for kickback, but only because/when products seem to truly deserve accolades.

Anyway, the version of Acronis TrueImage that I own is the Home version, but based on that, I’d definitely consider it for enterprise use too.

TrueImage is a disk imaging tool, and item-by-item recovery tool as well. You can back up an entire partition to a compressed archive and later, if you wish, mount the archive as a logical device, browse it and restore only what you wish. It features data verification, partition information restores, etc. It seems also to handle backup/restore cycles to disparate sizes (including downsizing, assuming the target device has enough room for the actual storage required by the files in the archive) gracefully and well.

Like any utility of this nature, you’re probably not going to use it day to day unless restoring disk images is 25% or more of your job description, but it’s very useful to have when you need it, and a good all-around way to keep a backup of, say, your system drive on, say, an external USB drive, in case you need it.

Since it also does both Full and Incremental operations, you can conceivably update your backups as often as once a day or more often.

This would even, with some training and relatively clueful users, be a good way to manage data/system recovery on a per-user basis, I think.

I’m very happy with it.

Database Migration breaks WSS/SQL Server Full Text Search

So here’s a little-known issue with SharePoint and Full Text Search:

On Joel Oleson’s blog (if you don’t know who this guy is and you’re in SharePoint Operations, find out quickly. Aside from Bill English [the man, his blog], he’s the other Man in SharePoint Managment/Operations – both of these guys regularly present at TechEd conventions), I found a blog entry about how, if your Full Text Search (that’s the one that works in Windows SharePoint Services, and is provided through the back-end SQL Server) isn’t working, and you migrated the database from a different server, the reason would be, possibly/probably, that you migrated the database from a different server.

Yes, I mean either with SPSBackup.exe (which should be your friend by now if you’ve been doing a lot of this and you want to keep the SharePoint Portal Search Index between migrations) or normal SQL Backup/Restore and file operations.

So anyway, there are a couple of stored procedures in the blog you should use to restore your Full Text Search. As far as I can tell, these are either SQL standard stored procedures, or, more likely, SharePoint-installed stored procedures.

Also, rather like part of my previous blog entry, the fix is essentially “turn it off, turn it on”, with some curve-balls in there if things don’t go as expected. It turned out to be thorough enough for us, so maybe it’ll be thorough enough for you.

Since I had to puzzle the blog entry out a little bit, I’ll write what I understood of how to do this here.

  1. Mr. Oleson recommends restarting your SQL Services, but we didn’t find this completely necessary.
  2. Run the following stored procedure: exec proc_DisableFullTextSearch
    1. If you get an error about there not being a Full Text Catalog, then run the following in SQL Query Analyzer and start Step 2 again:

      USE [databasename]
      sp_fulltext_database enable

      Where [databasename] is the name of the _SITE database you’re having the issue with. [But don’t actually type the [] brackets in there or your geek compatriots will laugh at you.]

  3. Run the following stored procedure: exec proc_EnableFullTextSearch

So that should be it. The procedure, as I said, is mostly just turn off, turn on again.

Changing SharePoint Service Accounts, Permissions and Troubles Found and Conquered Therein

(Note: Links to KB Articles open in new windows)

So I have a client who is security conscious enough to ask that we make SharePoint work even when the servers’ Local Administrators (members of BUILTIN\Administrators) are not members of the System Administrators Server Role in SQL Server. This is the default configuration for SQL Server – the BUILTIN\Administrators group should have a checkbox enabled next to the “System Administrators” role in the Server Roles tab of the Login properties for SQL Server.

After this helped break Full Text Search (KB Article 317746) and we fixed it, all was well and good, or so we thought.

It’s not completely obvious, but in SharePoint Portal Server 2003 running on Windows 2003 Enterprise SP1 and above (KB Article 555309), it’s required that the SharePoint Service Account be part of the servers’ (in the farm) Local Administrators. This would appear to convey the idea that SharePoint Service Accout should also be a System Administrator (role) in the SQL Server that hosts the configuration and content databases.

I believe, by running into it head first, we have corroborated this assertion.

Moving from a bad situation (where all three of our environments – Dev/Pilot, QA and Production – are running on the same service account whose password will expire in about 2 months), to a better one (we plan 2 service accounts for each environment), means we have to change the service accounts for both SQL Server services, and for SharePoint services in each environment.

(By the way: It’s actually recommended that if possible, service accounts associated with Enterprise servers like SharePoint be set with passwords that don’t expire. If your security policy requires it, you can then plan to change the password on a regular basis, but you won’t get caught out if you simply don’t have time to do it. The slightly more antagonistic policy is to make the automatic experation/reset loom over your operations people like an unassailable threat that motivates them to change the password regularly, manually, before the deadline. If you live in a more antagonistic sort of place like we do, at least have the common decency not to keep folks from changing the password early. Making the password change be required to happen on a certain day is the kind of sadism that breeds MAD IT CONSULTANTS! MAD, I TELL YOU!)

In doing so, we ran up against the problem of not having the SharePoint service account be a System Administrator in the Server Roles in SQL Server. We followed the KB Articles (SharePoint: KB Article 837813, SQL: KB Article 283811 – For SQL we actually used the SQL Enterprise Manager – it’s the easier way – try SQL Books Online for guides) for changing the service accounts, and when we got to checking the SharePoint Central Administration (to finish up the account changing process), instead of the SharePoint Central Administration, like we thought we’d get, we got a prompt to disconnect from the configuration database.

So, in a fit of troubleshooting pique, we followed the prompt and got ourselves into a deeper hole. Because at the instant that we disconnected, the server we were working on got out of synch from the rest of the server farm.

And how did we get out?

First, the primary fix is to make sure that if you’re not going to give all the Local Administrators System Administrator access in SQL Server, at least make sure that the SharePoint Service Account has System Administrator access. That’s the first thing.

Second, we need to fix the mess. If we hadn’t disconnected from the configuration database when the opportunity presented itself, restoring the System Administrator access for the service account in question would fix the whole kit and kaboodle (requiring, perhaps, closing and opening IE, or forcing a cache refresh [hold down Ctrl and click Refresh button] to see the results in IE).

Instead, since we disconnected the database at probably the worst time, the server that disconnected thinks it’s disconnected, but the rest of the farm doesn’t. Weirdly enough, the broken server acts like it thinks it’s both part and not part of the farm. I managed to get the server to the point where it serves content just fine from WSS, but SharePoint Central Administration thinks it isn’t connected to the configuration database (i.e. it’s broken).

When you try to connect to the existing configuration database, you get a weird and inappropriate error. Usually you get the error, “Unexpected error occurred.” (This error actually tells you a lot. If you google for it, you’ll find that most of the issues associated with this symptom are to do with network connectivity – this isn’t utterly bizarre – from SharePoint’s perspective, a SQL permissions error can look a lot like a network error, and whatever’s bunged up in the broken synchronization is also probably interfering and giving SharePoint weird results, which probably just go in the network bucket, because SharePoint can’t figure out where else to put it.)

Weirder still, it seems like the whole of the problem exists somewhere in the broken server’s configuration, with none of the actual problem on the working parts of the farm at all.

Why do I say this? Because the fix is really simple:

  1. Create a new configuration database with the broken server.
  2. Disconnect from the new configuration database.
  3. Connect to the old configuration database.
  4. You probably want to delete the new, unused configuration database too, but the things are pretty small, storage-wise.

If the synch error really were partly on the original configuration database and partly on the broken server, this shouldn’t fix the problem, because it would still exist in some part on the original configuration database. But in fact, this works.

Now, if, like us, you were in the middle of changing accounts for your farm, just continue on your merry way. Since you only need to make that System Administrator Server Role for the SharePoint Service Account fix once (assuming you’re only using one SQL Server in the farm), you should be good to go with the rest of the servers in the farm.

Since I should be doing this tomorrow in our QA environment (Dev’s down right now), I’ll let you know if there’s more to know, when I’ve figured it out.

Wish me luck!