TFS update error: VS402642 found a backup job running against database

Last week, I was performing upgrade to TFS 2015 Update 2. It all looked well for a very long time, untill one of collection databases failed to upgrade on step 1163 of 1171.

Clicking on error gave me some more information:
VS402642 found a backup job running against tfs_xxx database . Wait for the backup job to complete and the rerun the failed collection job from the Status tab.

From TFS Administrator Console i could see that the collection is offline, and job status is Failed. Trying to rerun it immediately fails with the same error.
For the next 15 minutes we tryed to locate if there is anything taking backup of the database, but there was nothing active - and one failed earlier. I could only assume, that backup job on SQL server kicked in while collection was under upgrade and didn't manage to finish, but also blocked successfull TFS update.
Making it short, we had to : quiesce to stop all TFS activities, detach collection database in SQL Management studio (do not detach in TFS!), killing all connections, attach collection database back, unquiesce.
After that job was able to rerun and finish the upgrade.

The lesson learned - make sure no scheduled backups can kick in while you perform your time consuming upgrade to TFS 2015 Update 2.

Restore database to another name on the same SQL server

I often need to copy a database into another database on the same SQL server instance. So I take a backup and restore from it into another database. And here is the script:

RESTORE DATABASE [Tfs_tfs] FROM DISK=N'C:\Program Files\Microsoft SQL Server\MSSQL11.NATANSQL\MSSQL\Backup\BackupFile.bak'

WITH

   MOVE 'Tfs_MyCollection' TO 'C:\Program Files\Microsoft SQL Server\MSSQL11.NATANSQL\MSSQL\DATA\Tfs_tfs.mdf',

   MOVE 'Tfs_MyCollection_log' TO 'C:\Program Files\Microsoft SQL Server\MSSQL11.NATANSQL\MSSQL\DATA\Tfs_tfs_log.ldf'

GO

Where Tfs_MyCollection is the name of original database,
BackupFile.bak is the backup of it, and
Tfs_tfs is the new database name.

Problem upgrading from TFS 2013 Update 4 to TFS 2015 Update 2

Microsoft announced relase of TFS 2015 Update 2 in the end of March, warning that it is a big one in context of internal data changes. Performing several upgrades from multiple versions to mutiple versions for last years, I can say, that usually it goes smoothly. Some hickups, of course, but no big ones.

This week I had an upgrade from TFS 2013 Update 4 to latest (2015 Update 2), and ran into rather interesting failure.

By the way, many people ask for timing - for a collection of 80 Gb it takes around 1.5 hours (with database on another machine with 2x4 cores and 64 Gb RAM).
On my virtual machine with 2Gb memory and all in one place - empty collection took 2 minutes.

The upgrade process itself went totally smooth. Then we start testing upgraded TFS, and got 2 collections totally fine, and one - unaccessible. While all 3 collection were shown fine and healthy in TFS admin console,  web page for one of them returned:

All the team projects in the "bad" collection were not available from either web or Visual Studio.

Event Log had this:
System.Web.HttpException (0x80004005): Page not found.
   at Microsoft.TeamFoundation.Server.WebAccess.Controllers.ErrorController.NotFound()
   at lambda_method(Closure , ControllerBase , Object[] )
   at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
...and lots of other Page Not Found.

At this moment we decided to rollback to TFS 2013 Update 4, as googling didn't give us any answer what is wrong and how to fix it.

Day after, exploring the error logs, I noticed that some urls looked wrong, which gave me an idea... so I was able to reproduce the failure on a totally clean environment. 

The problem was in the collection name. From TFS 2012 and on it is not allowed to call collection tfs, as it is a reserved word. My customers had their TFS from version 2010, so they managed to call collection "tfs" and strangely, problem never appeared before they got all the way into 2015.

Solution is rather simple. Prior to upgrade to TFS 2015, rename the collection into smth else. And then it all goes nicely.

I hope, Microsoft will add some checks during verification prior to upgrade. Even though case might be not so common.

PS: Microsoft provided us a solution in form of .dll to keep collection name unchanged and also announced that the problem will be fixed in 2015 Update 3. 

PPS: Follow up: we also hit a build controllers and agents issue - the just stopped with "Page not found" error. Copying the .dll under /Application Tier/Message Queue/bin/Plugins folder on app tier machine fixed it.

PS: reproducing error

Image 1: Windows 7 

• Install TFS 2010 (SQL Server 2008 R2 Update 3). Create collection with name tfs. 

Image 2: Windows Server 2012

• Migrate data and upgrade to TFS 2012 Update 4 (SQL Server 2012 Standard)
• Upgrade to TFS 2013 Update 4
• Upgrade to TFS 2015 Update 2
• Open web and try to access any team project in the collection tfs - Page not found. Connecting from Visual Studio doesn't work either.

No reporting, no sharepoint integration.