Thursday, April 14, 2016

Problem upgrading from TFS 2013 Update 4 to TFS 2015 Update 2

Microsoft announced relase of TFS 2015 Update 2 in the end of March, warning that it is a big one in context of internal data changes. Performing several upgrades from multiple versions to mutiple versions for last years, I can say, that usually it goes smoothly. Some hickups, of course, but no big ones.

This week I had an upgrade from TFS 2013 Update 4 to latest (2015 Update 2), and ran into rather interesting failure.

By the way, many people ask for timing - for a collection of 80 Gb it takes around 1.5 hours (with database on another machine with 2x4 cores and 64 Gb RAM).
On my virtual machine with 2Gb memory and all in one place - empty collection took 2 minutes.

The upgrade process itself went totally smooth. Then we start testing upgraded TFS, and got 2 collections totally fine, and one - unaccessible. While all 3 collection were shown fine and healthy in TFS admin console,  web page for one of them returned:

All the team projects in the "bad" collection were not available from either web or Visual Studio.

Event Log had this:
System.Web.HttpException (0x80004005): Page not found.
   at Microsoft.TeamFoundation.Server.WebAccess.Controllers.ErrorController.NotFound()
   at lambda_method(Closure , ControllerBase , Object[] )
   at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
...and lots of other Page Not Found.

At this moment we decided to rollback to TFS 2013 Update 4, as googling didn't give us any answer what is wrong and how to fix it.

Day after, exploring the error logs, I noticed that some urls looked wrong, which gave me an idea... so I was able to reproduce the failure on a totally clean environment. 
The problem was in the collection name. From TFS 2012 and on it is not allowed to call collection tfs, as it is a reserved word. My customers had their TFS from version 2010, so they managed to call collection "tfs" and strangely, problem never appeared before they got all the way into 2015.

Solution is rather simple. Prior to upgrade to TFS 2015, rename the collection into smth else. And then it all goes nicely.

I hope, Microsoft will add some checks during verification prior to upgrade. Even though case might be not so common.

PS: Microsoft provided us a solution in form of .dll to keep collection name unchanged and also announced that the problem will be fixed in 2015 Update 3. 

PPS: Follow up: we also hit a build controllers and agents issue - the just stopped with "Page not found" error. Copying the .dll under /Application Tier/Message Queue/bin/Plugins folder on app tier machine fixed it.

PS: reproducing error
Image 1: Windows 7 
  • Install TFS 2010 (SQL Server 2008 R2 Update 3). Create collection with name tfs
Image 2: Windows Server 2012
  • Migrate data and upgrade to TFS 2012 Update 4 (SQL Server 2012 Standard)
  • Upgrade to TFS 2013 Update 4
  • Upgrade to TFS 2015 Update 2
  • Open web and try to access any team project in the collection tfs - Page not found. Connecting from Visual Studio doesn't work either.
No reporting, no sharepoint integration.

1 comment: