Showing posts with label TFS. Show all posts
Showing posts with label TFS. Show all posts

Thursday, May 26, 2016

How to build .Net Core project with TFS vNext build

So, we created a wonderful .Net Core solution, it looking forward to build it with VSTS or TFS on-premises. And of course we want to use vNext build system :)

First of all - the Hosted agent in Azure CAN NOT build .Net Core projects yet. We will therefore need to use on-premises agent. If you need to know how to configure a vNext build agent - check this manual. Also Martin has a very detailed step-by-step post here.

After build agent is on, make sure you have Visual Studio 2015 Update 2 (ot upgrade to it) and install  .Net Core SDK from here. Now we are all set!

If you just go on the easiest way and do - get the code, build:

it will not work. Reason is that package management is all different in .Net Core, and nuget restore doesn't know how to deal with your .Net Core project packages. So you get:
The dependency Ix-Async >= 1.2.5 could not be resolved.
The dependency Microsoft.AspNetCore.Antiforgery >= 1.0.0-rc2-final could not be resolved.
The dependency Microsoft.AspNetCore.Authorization >= 1.0.0-rc2-final could not be resolved.
... 
and so on. Packages are listet in your project.json file. So unless you restore them with .Net CLI tool prior to build, it ain't work.

Therefore create a packagesRestore.ps1 powershell script like this:
Get-ChildItem -Path $PSScriptRoot\MyWebApp.NetCore -Filter project.json -Recurse | ForEach-Object { & dotnet restore $_.FullName 2>1 }
and save it next next to your .sln file. Check in. 
Pay attention that my project name is MyWebApp.NetCore - so the project.json file is under corresponding directory. 

Now, in you vNext build definition add a new execute powershell before Build solution:

and that's it! Now we can build .Net Core app with VSTS on our on-premises Build agent :)

Thursday, April 28, 2016

TFS update error: VS402642 found a backup job running against database

Last week, I was performing upgrade to TFS 2015 Update 2. It all looked well for a very long time, untill one of collection databases failed to upgrade on step 1163 of 1171.

Clicking on error gave me some more information:
VS402642 found a backup job running against tfs_xxx database . Wait for the backup job to complete and the rerun the failed collection job from the Status tab.

From TFS Administrator Console i could see that the collection is offline, and job status is Failed. Trying to rerun it immediately fails with the same error.
For the next 15 minutes we tryed to locate if there is anything taking backup of the database, but there was nothing active - and one failed earlier. I could only assume, that backup job on SQL server kicked in while collection was under upgrade and didn't manage to finish, but also blocked successfull TFS update.
Making it short, we had to : quiesce to stop all TFS activities, detach collection database in SQL Management studio (do not detach in TFS!), killing all connections, attach collection database back, unquiesce.
After that job was able to rerun and finish the upgrade.

The lesson learned - make sure no scheduled backups can kick in while you perform your time consuming upgrade to TFS 2015 Update 2.

Thursday, April 14, 2016

Problem upgrading from TFS 2013 Update 4 to TFS 2015 Update 2

Microsoft announced relase of TFS 2015 Update 2 in the end of March, warning that it is a big one in context of internal data changes. Performing several upgrades from multiple versions to mutiple versions for last years, I can say, that usually it goes smoothly. Some hickups, of course, but no big ones.

This week I had an upgrade from TFS 2013 Update 4 to latest (2015 Update 2), and ran into rather interesting failure.

By the way, many people ask for timing - for a collection of 80 Gb it takes around 1.5 hours (with database on another machine with 2x4 cores and 64 Gb RAM).
On my virtual machine with 2Gb memory and all in one place - empty collection took 2 minutes.

The upgrade process itself went totally smooth. Then we start testing upgraded TFS, and got 2 collections totally fine, and one - unaccessible. While all 3 collection were shown fine and healthy in TFS admin console,  web page for one of them returned:

All the team projects in the "bad" collection were not available from either web or Visual Studio.

Event Log had this:
System.Web.HttpException (0x80004005): Page not found.
   at Microsoft.TeamFoundation.Server.WebAccess.Controllers.ErrorController.NotFound()
   at lambda_method(Closure , ControllerBase , Object[] )
   at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
...and lots of other Page Not Found.

At this moment we decided to rollback to TFS 2013 Update 4, as googling didn't give us any answer what is wrong and how to fix it.

Day after, exploring the error logs, I noticed that some urls looked wrong, which gave me an idea... so I was able to reproduce the failure on a totally clean environment. 
The problem was in the collection name. From TFS 2012 and on it is not allowed to call collection tfs, as it is a reserved word. My customers had their TFS from version 2010, so they managed to call collection "tfs" and strangely, problem never appeared before they got all the way into 2015.

Solution is rather simple. Prior to upgrade to TFS 2015, rename the collection into smth else. And then it all goes nicely.

I hope, Microsoft will add some checks during verification prior to upgrade. Even though case might be not so common.

PS: Microsoft provided us a solution in form of .dll to keep collection name unchanged and also announced that the problem will be fixed in 2015 Update 3. 

PPS: Follow up: we also hit a build controllers and agents issue - the just stopped with "Page not found" error. Copying the .dll under /Application Tier/Message Queue/bin/Plugins folder on app tier machine fixed it.

PS: reproducing error
Image 1: Windows 7 
  • Install TFS 2010 (SQL Server 2008 R2 Update 3). Create collection with name tfs
Image 2: Windows Server 2012
  • Migrate data and upgrade to TFS 2012 Update 4 (SQL Server 2012 Standard)
  • Upgrade to TFS 2013 Update 4
  • Upgrade to TFS 2015 Update 2
  • Open web and try to access any team project in the collection tfs - Page not found. Connecting from Visual Studio doesn't work either.
No reporting, no sharepoint integration.

Wednesday, February 18, 2015

TeamCity + TFS Git repository: unable to find valid certification path to requested target

Accessing TFS Git repository from TeamCity, for an on-premises TFS installation gives you the following error?

Failed for the root '"XXX Git" {instance id=225, parent internal id=48, parent id=XXX_YYY_Git, description: "https://tfsinstance/tfs/DefaultCollection/TeamProject/_git/GitRepository#refs/heads/master"}: List remote refs failed: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 

Your corporate certificate is not known by TeamCity. This is how you fix the problem.

Save your corporate root certificate into CompanyRoot.cer file

For server side checkout: 
  1. Login to the TeamCity server machine
  2. Add this root certificate to keystore of your TeamCity 
%TeamCityInstallDir%\jre\bin\keytool.exe  -importcert -trustcacerts -file CompanyRoot.cer -alias ca_alias -keystore "%TeamCityInstallDir%\jre\lib\security\cacerts"

(replace %TeamCityInstallDir% with actual path to your TeamCity)

If you haven't ever changed it, password for the keystore is changeit :)

For agent side checkout:
  1. Install Git on agent machine to lets say C:\Programs\Git\
  2. Run on agent machine git config --global http.sslCAInfo D:\Programs\Git\bin\curl-ca-bundle.crt
  3. Edit curl-ca-bundle.crt manually, add your Company root certificate to the end of the file
Mor information can be found here: 


Wednesday, October 8, 2014

Howto: Set up TeamCity to use TFS Proxy

TeamCity usage of TFS as a source provider is not exactly perfect, but setting up fetching data using TFS Proxy worked surprisingly well!

So, we have some TFS server and a configured TFS proxy with address, let's say http://tfsProxy:8081
and now we need to explain TeamCity that it should fetch files using it.

Configure VCS root pointing to TFS server as ususal, do not think about proxy yet.

And now - all you need to do is to set up an environment variable TFSProxy with the value of TFS Proxy address.
Then restart TeamCity agent - if you use agent side checkout. Or restart TeamCity server - if you use server side checkout.

NB! Sending TFSProxy as an environment variable to build will not work. It MUST be an environment variable on a machine which is fetching code.

Testing scenario:
Set up a build to fetch some data from TFS.

1. Clean up TFS Proxy cache by deleting all under: %TFS_Proxy_install_directory%\Version Control Proxy\_tfs_data\Proxy

2. Clean up TeamCity cache:



  • Server side checkout - delete all under %TeamCity Data Directory%\system\caches\sources
  • Agent side checkout - delete all under %localappdata%\Microsoft\Team Foundation\5.0\Cache
3. Clean up build directory from source files, if not empty.
4. Start build and see Cache folder getting populated on TFS Proxy machine :)

Hope that helps!


Wednesday, May 14, 2014

Git problem connecting to TFS on-premise from Bamboo agent

That was a weird case - thus even more interesting to solve.

We needed to connect our on-premise TFS instance with Git projects to build on Bamboo build environment (there will come a full howto-step-by-step post about it later).
And besides everything  was set up correct, build was still failing trying to fetch code from repository. Error looked like:

fatal: https://{buildUser}@tfs.osiris.no/tfs/{Collection}/{TeamProject}/_git/{RepositoryName}/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

More interesting - testing the same stuff against my own VisualStudio repository in Azure went smooth.

Narrowing the scope showed up that the command failing was git-ls-remote:

git ls-remote https://{buildUser}:*****@tfs.osiris.no/tfs/{Collection}/{TeamProject}/_git/{RepositoryName}

The problem was (weird!) that the password for the build user contained ! at the end. After I changed the password - everything went fine. Even more - after I changed the password back it still goes fine...

For info: it is git version 1.8.1.4 running at Amazon elastic cloud build instanse.

Wednesday, November 27, 2013

TFS: Error deleting Team Project

Recently I needed to cleanup a collection, removing a number of team projects from it. Generally all went fine, but one team project refused to be deleted, saying something like that:

Error logs said not much:
Executing step: 'Delete the team project data from Version Control' VersionControl.DeleteTeamProject (2 of 11)
[Error] Violation of PRIMARY KEY constraint 'PK__#yourLoc__FEE84A3672E9B4C6'. Cannot insert duplicate key in object 'dbo.#yourLocalPendingChanges'. The duplicate key value is (51843).

What I understood from it, is that it failed to delete team project from the source control. Oh well, thought I, I can try to delete from command line. So I did and hit:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE>tf destroy "$/MyProject" /collection:http:\\localhost:8080\tfs\defaultcollection
TF246021: An error occurred while processing your request.
Technical information (for administrator):
SQL Server Error: 2627


At this moment things started looking very dark. I tried of course deleting again from Admin console, and of course with no luck.

But indeed I found the solution :)

I used the destroy command to drop things under source control one by one. Like :
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE>tf destroy "$/MyProject/Subfolder1" /collection:http:\\localhost:8080\tfs\defaultcollection
...
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE>tf destroy "$/MyProject/Subfolder2" /collection:http:\\localhost:8080\tfs\defaultcollection
 ..etc 
At first to folders I hit the same error, but I kept going and then, after some folders got successfully deleted, all others, even those wich were giving error - also got deleted. 

So when all subfolders was deleted I ran again:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE>tf destroy "$/MyProject" /collection:http:\\localhost:8080\tfs\defaultcollection
And it all went good!
So, back to admin console and  delete project again - voila!

I am certain I am not the only one who hit the problem and desperately googled without any result :)
So hope this solution will work for others as well!

Wednesday, November 20, 2013

TFS to Git migration: step by step

It has been some posts about migrating to Git from TFS, unfortunately none of them worked for mas expected.

So, my task was: migrate a source from TFS to Git, preserve all history, but do not include dlls or other heavy files. Last part requires naturally not only removing those files from the latest source, but also removing it from the entire history.
On my way I met some hickups, which I will mention, so the road for others after me can be smooth :)

So, lets start!
You will use GitBash and Git-tf:

1. Install Git-TF. Follow this link and download latest release:  http://gittf.codeplex.com/
 Installation instruction is very easy, I chosen not to use Chocolatey, but changed PATH environment variables, worked perfectly.

2. Start GitBash. If you do not have it - can be downloaded from here

Note! All things from now are done from under GitBash.

Check if git-tf was installed successfully - just type "git tf", it should return usage description of git tf command.

Navigate to the place where you have you Git repositories. I use C:\Git\Sources\Repos folder. So my git promt looks like this now:

3. Clone my TFS source to some temporary Git repo.
git tf clone {collection url} {team project path} {temp repo name} --deep

like this:

Note --deep - that means I want to take history as well.

4. Create Git repository which you will use later as your Git source. So far I will not link it to TFS Git repository, just create it.
git init {repo name}
like this:

get into newly create repository by simply
cd {repo name}
you can see you are in a master branch of your newly created repository:

5. Being there, now pull the source from you temp TFS git to your future Git place:
git pull ../{temp repo name} --depth=100000000

teoretically we could push it to our TFS git already now, but first lets cleanup dlls or other unneeded files, so when we put it to TFS git it is all clean and lightweight.

In my example I had 2 installation files located in folder wix/wixDownloads. So I want to leave those behind.

git filter-branch --force --index-filter 'git rm -rf --cached --ignore-unmatch wix/wixdownloads'
Command works only from the root, so paths must be relative from there.
If I would need to delete only one file, I would type:
git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch wix/wixdounloads/wix35.msi' 
It is important it says was rewritten in the end, that means stuff was actually removed. Otherwise - you probably misspell something.

If you type wrong, sometimes you get error "cannot remove directory '..../.git-rewrite" : directory is not empty. Dont panic, just remove it manually.

6. We are almost there! Now just clean up so all deleted files are physically deleted:
git reflog expire --all
git  gc --aggressive --prune
Now you can compare size og original forlder with clone from TFS, and size of cleaned up folder.
If everything is correct, the second one is much smaller!

7. And finally - push up to your TFS git:

git remote add master {your  TFS Git repository address}
git push -u master --all
like this: 

8. Now, when I compare hisoty on my original TFS source to history on my new TFS git source, I see no wixDownloads folder, and even not a corresponding checkin (well, this is because checkin contained JUST this folder, and no other files).



To the left - my new Git repo history, to the right - my original TFS history.

Hope it saves somebodys time!

Update 1: there is a possibility to hit  error when pulling source to the temp repository, if full filenames on temp repository exceed 256 symbols:

 error: unable to create file ...{verylongfullpath}... (Filename too long)

ignoring the error gets us on with being unable to do filter-branch and error:
fatal: Neeeded a single revision

To get around that make sure that the folder name for temp repo is short enough. In my case I just deleted temp repository, created a new one with a very short name and did git pull again.

Monday, November 18, 2013

TFS - customize Features backlog and board

By default Features board shows only work item type Feature. Lets say we have another work item type - Business Request, and we want it to be tretaed same way as Features.

So we need to make changes i categories definition of the process template and include Business Request in Features category.

First, download existing:
witadmin exportcategories /collection:{CollectionAddress} /p:{TeamProjectName} /f:C:\Temp\categories.xml

Now lets edit it:
 

And upload back to server:
witadmin importcategories /collection:{CollectionAddress} /p:{TeamProjectName} /f:C:\Temp\categories.xml

Ok, we can see new work item type available on our Feature backlog:


Next - configure Feature board.
By default it has only 3 columns, and only 2 states to choose from: Active and New. Which was very confusing to me, since our Business Request definitely has more states available.

To make it appear in the list of available States - we need to do changes in process config file. First,
download existing:
witadmin exportprocessconfig /collection:{CollectionAddress} /p:{TeamProjectName} /f:C:\Temp\processConfig.xml

Edit it, so it contains states we want to use:
 
Upload:
witadmin importprocessconfig /collection:{CollectionAddress} /p:{TeamProjectName} /f:C:\Temp\processConfig.xml



Now we can see more States available to map to columns in Features Board:
 

Congratulations!



Friday, October 4, 2013

TFS API 2013 - check access rights of the user

I've got a task which suppose to be really simple - check if user has an access to delete a build in  a Team project.

Seems to be easy, but no, it showed to be really poorly documented!
Therefore I put my code here, so people can find it and reuse it.

So, at first, we create a collection object, same good old way:

var tfs = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(new Uri("YourCollectionAddress"));
tfs.EnsureAuthenticated();


Getting a team project object, good old way:

var versionServer = tfs.GetService<VersionControlServer>();
var teamProject = versionServer.GetAllTeamProjects(false).ToList().First(tp => tp.Name == tpName);


Getting user identity - here comes new - using IIdentityManagementService:

var identityService = tfs.GetService<IIdentityManagementService>();
var userIdentity = identityService.ReadIdentity(IdentitySearchFactor.AccountName, username, MembershipQuery.Expanded, ReadIdentityOptions.None);


Getting SecurityNamespace object for team project - in this object we can see which permissions generally are appliable for the target (in our case - team project). Also new - using ISecurityService (IGroupSecurity is obsolete now):

var ss = (ISecurityService)tfs.GetService(typeof(ISecurityService));
var sn = ss.GetSecurityNamespaces().First(ssn => ssn.Description.Name == "Project");

With a good usage og LINQPad I can check the following access rights for SecurityNamespace for a Team Project:
Ok, so I will check for  UPDATE_BUILD then.

Next step - getting actually access rights list for the user in the Team project:

var securityToken = "$PROJECT:" + teamProject.ArtifactUri;
var access = sn.QueryAccessControlList(securityToken, new List<IdentityDescriptor> {userIdentity.Descriptor}, true);

Almost there!

Now we just need to check if the following access list contains UPDATE_BUILD, either as inhereted or granted directly:

if (access.AccessControlEntries.Any(
      acl => ((acl.ExtendedInfo.EffectiveAllow & 128) == 128 || 
      (acl.ExtendedInfo.InheritedAllow & 128) == 128)))
{
    return true;
}
return false;

Hope that helps :)

More information
Those pages helped me a lot, and actually were only info I found about using TFS API for permissions set\check:
http://blog.johnsworkshop.net/tfs11-api-managing-team-administrators/

http://pascoal.net/2011/11/using-team-foundation-server-registrypart-i-the-concepts/
pascoal.net/2011/11/using-team-foundation-server-registrypart-ii-the-api/

Tuesday, August 13, 2013

TFS 2013 Preview and TF400917: The current configuration is not valid for this feature.

After upgrading to TFS 2013 Preview some of our team projects started to get this error when trying to access backlog:
Some of our projects were successfully upgraded though. So what is to be done in order to fix the error:

1. You need to install Visual Studio 2013. witadmin tool from Visual Studio 2012 can export process config files, but can not import. Make sure you run witadmin from C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE

2. Run
witadmin exportprocessconfig /collection:http://yourTFSserver:8080/tfs/YourCollection /p:YourTeamProject /f:c:\temp\processconfig.xml 

3. Correct processconfig.xml file so for nodes :
  • FeedbackRequestWorkItems, 
  • FeedbackResponseWorkItems
  • RequirementBacklog 
  • TaskBacklog 
there exists attributes pluralName and singularName. Like this, for example:
<RequirementBacklog category="Microsoft.RequirementCategory" pluralName="Backlog items" singularName="User Story">
...
</RequirementBacklog>

4. For RequirementBacklog and TaskBacklog there should exist not empty AddPanel sub-node; like this:
<TaskBacklog category="Microsoft.TaskCategory" pluralName="Tasks" singularName="Task">
    <AddPanel>
      <Fields>
        <Field refname="System.Title" />
      </Fields>
    </AddPanel>

...
</TaskBacklog>

5. Run
witadmin importprocessconfig /collection:http://yourTFSserver:8080/tfs/YourCollection /p:YourTeamProject /f:c:\temp\processconfig.xml

6. It might complain about multiple states with the type Complete. Yes, it should be only one. Make sure that this part of processconfig.xml defines only one state as Complete:
<States>
      <State type="Proposed" value="Analysis" />
      <State type="Proposed" value="Proposed" />
      <State type="Proposed" value="Not Started" />
      <State type="InProgress" value="Active" />
      <State type="InProgress" value="Resolved" />
      <State type="Complete" value="Closed" />
</States>


7. Run witadmin importprocessconfig.... again. That should be it.


Monday, April 29, 2013

Backlog board TF400898: An Internal Error Occurred.

After upgrading to TFS 2012 Update2 our customer start getting "TF400898: An Internal Error Occurred" when trying to view a product backlog board.

Event log at tfs application tier said:
Exception Message: The given key was not present in the dictionary. (type KeyNotFoundException)
Exception Stack Trace:    at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.TeamFoundation.Server.WebAccess.Agile.Models.WorkItemSource.<>c__DisplayClassa.<GetCompleteWorkItemData>b__6(IDataRecord dataRecord)
   at Microsoft.TeamFoundation.Server.WebAccess.Agile.Utility.WorkItemServiceUtils.<GetWorkItems>d__c.MoveNext()...

...etc ..etc

As this is a known issue, solution is described here:
http://blogs.msdn.com/b/visualstudioalm/archive/2013/03/19/how-to-fix-tf400898-an-internal-error-occurred-on-your-kanban-board-in-tfs-2012-update-2.aspx

But... it didn't help. Yes, there was inconsistency and 2 possible work items states mapped to Complete board state, but after fixing CommonProcessConfig.xml and leaving only User Story work item in Requirements categories - error still kept coming.

It looks like if there are work items which have the "invalid" state, the board is not getting rebuilt. So, in this unlucky case the fastest fix is to create a new area, and make it the only and default area for backlog.Which practically makes the backlog (and the board) empty.

So, this time the board gets rebuilt successfully. Go to the board page and verify it. After that, just set default area(s) for the backlog the way it was before. Most likely you will need to correct columns for the board, but the annoying TF400898 will be gone.
Just click on Correct this now, and assign relevant states for the board columns.


Neat!

Tuesday, April 9, 2013

Some features of Team Web Access are not visible to you

Installing TFS, going to Team Web Access and observing something lilke this?
And there is no Backlog and Sprint Planning Tools and neither can you see Request Feedback feature.

You need to enable all features. Somehow by default they are set to "Standard" level, not "Full".
Enter administration page and go to Control Panel:

Select Full and make it default.


That will do!

Thursday, April 4, 2013

TFS 2012 Velocity chart

Here I will explain what is it the velocity chart in TFS 2012 shows. Generally - velocity shows how much a team can do during a sprint.

So it comes to - how do you estimate effort. IN TFS 2012 we use User Stories, and for estimates - Story Points. This is essential to remember. Look here, an example of velocity chart:
First of all, on X-axis - iteration ordered by time. This means you have to set timeframe for your iterations. For this go to admin page and Iterations tab.

Y-axis. See numbers? Those are story points. If your work items do not have story points - they will not appear on this chart.

Now about colours. Green shows work items in Resolved\Completed\Done state. Blue - Active. Other states are not shown.

Hope that clarifies some things.

Thursday, February 28, 2013

TFS + VS: Find branches of a file

Let say, you have a file initially used by one project, and later usage spred widely around other projects.
For us it happend with code analysis rules settings file.

So we decided that the best is to branch this file. Having it in one place get you into relative paths hell, just copying across projects... hmmmm.... no so good - gets out of control sooner or later.
Brnach it - and you know where it is and which version.

What I found not so obvious - is where to find all branches taken from the file?

Locate file in Source control, right-click -> Advanced -> Properties.
Here it is - Branches tab:



Hope it helps somebody :)