Convert order lines to weighted graph

Let's say we have orders history with some products. We need to perform community detection as a part of market basket analysis. Order lines are like OrderId - ProductId.

First thing need to be done is to convert order lines into weighted graph. Where Nodes are products and Edges connect nodes if products were purchased in the same order.
Something like this:

Weighted means that an Edge "weight" between two products is equal to amount of times those products were bought together.

Trying to find existing code for this task and not succeding, I have created a code snippet in R, which does the conversion from order lines to weigthed graph using adjacency matrix. 

For the graph above adjacency matrix can look like this:

The ides is simple - convert order lines into adjacency matrix N x N, where N = number of products (all columns are products, and all rows are products, edges weight = number of times two products bought in the same order). And adjacency matrix is easy convertable to graph.

This approach happens to work relatively fast.

Code is located here: https://github.com/iladan/R/blob/master/codeSnippets/OrdersToGraph.R

Hope that saves somebody's time :)

Error in library(igraph) : there is no package called ‘igraph’

An odd error occured to me today, after
   install.packages("igraph") 
   library(igraph) 
returned error: Error in library(igraph) : there is no package called ‘igraph’
Checked library location for my R (it is, by the way, C:\Program Files\R\R-3.2.5\library) and igraph was indeed not there! The package was downloaded thougth, as my log contained:
   The downloaded binary packages are in C:\Users\iladan\AppData\Local\Temp\Rtmp63c5us\downloaded_packages
And yes, indeed, igraph_1.0.1.zip was there.

Well, if it doesn't want to play by the book, - simply extract the archive into library folder and problem solved.

How to build .Net Core project with TFS vNext build

So, we created a wonderful .Net Core solution, it looking forward to build it with VSTS or TFS on-premises. And of course we want to use vNext build system :)

First of all - the Hosted agent in Azure CAN NOT build .Net Core projects yet. We will therefore need to use on-premises agent. If you need to know how to configure a vNext build agent - check this manual. Also Martin has a very detailed step-by-step post here.

After build agent is on, make sure you have Visual Studio 2015 Update 2 (ot upgrade to it) and install  .Net Core SDK from here. Now we are all set!

If you just go on the easiest way and do - get the code, build:

it will not work. Reason is that package management is all different in .Net Core, and nuget restore doesn't know how to deal with your .Net Core project packages. So you get:
The dependency Ix-Async >= 1.2.5 could not be resolved.
The dependency Microsoft.AspNetCore.Antiforgery >= 1.0.0-rc2-final could not be resolved.
The dependency Microsoft.AspNetCore.Authorization >= 1.0.0-rc2-final could not be resolved.
... 
and so on. Packages are listet in your project.json file. So unless you restore them with .Net CLI tool prior to build, it ain't work.

Therefore create a packagesRestore.ps1 powershell script like this:
Get-ChildItem -Path $PSScriptRoot\MyWebApp.NetCore -Filter project.json -Recurse | ForEach-Object { & dotnet restore $_.FullName 2>1 }
and save it next next to your .sln file. Check in. 
Pay attention that my project name is MyWebApp.NetCore - so the project.json file is under corresponding directory. 

Now, in you vNext build definition add a new execute powershell before Build solution:

and that's it! Now we can build .Net Core app with VSTS on our on-premises Build agent :)

How to use Microsoft R Server (Revolution R) in Visual Studio

First, you will need to install R Tools for Visual Studio followng instructions here. Then Microsoft R Server - the how-to is here.

After R Tools for Visual Studio are installed, open Visual Studio. You will see a new top menu "R Tools" available:

Click R Tools -> Options. Visual Studio options will open a tab for R settings:

Modify R Engine path to where you have your MRO-for-RRE installation (default is C:\Program Files\Microsoft\MRO-for-RRE\8.0\R-3.2.2)

Restart Visual Studio and enjoy using R server from your favourite IDE :)

Installing Microsoft R Server (Revolution R) on Windows

As we know, Microsoft bought Revolutionary R, and now it is available as a part of your MSDN ubscription. This is wonderfull! Installation though, is not still on a level with other Microsoft products, therefore - a short how-to install it post.

1. Download Microsoft R Server from msdn subscribers downloads page:

Thats's the one:

2. Download prerequisite - MRO 3.2.2 for RRE 8.0.0 from here:
https://mran.microsoft.com/install/mro4mrs/8.0.0/MRO-3.2.2-for-RRE-8.0.0-Windows.exe
as that one will not be installed automatically along with other prerequisites.

3. Install MRO for RRE 8.0.0 by running MRO-3.2.2-for-RRE-8.0.0-Windows.exe
4. Unpack en_rre_for_windows_x64_8183330.zip and run Revolution-R-Enterprise-8.0.0-Windows.exe

That's it. Some notes:

1. Yes, I had MRO 3.2.3 and MRO 3.2.4 installed, but Microsoft R server requires exactly MRO-for-RRE.
2. It will install some old components dating back to Visual Studio 2008,

Restore database to another name on the same SQL server

I often need to copy a database into another database on the same SQL server instance. So I take a backup and restore from it into another database. And here is the script:

RESTORE DATABASE [Tfs_tfs] FROM DISK=N'C:\Program Files\Microsoft SQL Server\MSSQL11.NATANSQL\MSSQL\Backup\BackupFile.bak'

WITH

   MOVE 'Tfs_MyCollection' TO 'C:\Program Files\Microsoft SQL Server\MSSQL11.NATANSQL\MSSQL\DATA\Tfs_tfs.mdf',

   MOVE 'Tfs_MyCollection_log' TO 'C:\Program Files\Microsoft SQL Server\MSSQL11.NATANSQL\MSSQL\DATA\Tfs_tfs_log.ldf'

GO

Where Tfs_MyCollection is the name of original database,
BackupFile.bak is the backup of it, and
Tfs_tfs is the new database name.

Windows -> Unix convert line endings with Git

Recently we met a problem, that files we fetch from TFVC to Linux build agents fail the build.

Problem was good old line endings - all our files had CRLF, since they were checked in from Windows.
Problem never appeared before, previously people were using Perforce clients, which can fix line ending for the "host" operating system. With Team Explorer everywhere that was not a case anymore.
So long story short, I ended up with need to fix line ending for the set of files, located in the same directory. Massive convertion of file endings... I need a tool... Git! So I created a script which convert line endings for all files in a specific directory (and subdirectories) and checks in to TFVC.

Configure Git line endings to unix style :

git config --global core.autocrlf input

Go into folder which need to be converted:

cd MYFOLDER

Make Folder a git repo, checkin all files:

git init

git add *

git commit -m "make it a unix, baby..."

At this moment in repository files are with Unix line endings, but locally on disk - still unchanged. Let's fix that.

Delete all files and then undo change so it will fetch back from repository:

git rm *

git reset

git checkout *

Now all files have correct line endings, we need to checkin to TFVC. So, delete .git in  your folder:

rd .git /S /Q

Go up one level and checkout your folder and all under:

cd ..

tf checkout MYFOLDER /recursive

Checkin to TFS forcibely (it does not see the difference if only line endings changed, so we need to force):

tf checkin MYFOLDER /recursive /force /noprompt

Voila :)

What is extra nice,  Visual Studio is clever enougth to deal with Linux style line endings. So I still can build my solution with devenv.exe. 

And even msbuild had nothing against unix style line endings.

PS: some more reading on subject

http://blog.codinghorror.com/the-great-newline-schism/

PPS: Of course, if you want to checkin changes or new files to TFVC, you need to watch line endings. It is possibel to configure your TFVC client to be aware. See more information here:

https://msdn.microsoft.com/en-us/library/gg413272(v=vs.100).aspx

or here:

http://blogs.msdn.com/b/tfsxp/archive/2010/04/26/extended-attributes-in-team-explorer-everywhere.aspx

TeamCity + TFS Git repository: unable to find valid certification path to requested target

Accessing TFS Git repository from TeamCity, for an on-premises TFS installation gives you the following error?

Failed for the root '"XXX Git" {instance id=225, parent internal id=48, parent id=XXX_YYY_Git, description: "https://tfsinstance/tfs/DefaultCollection/TeamProject/_git/GitRepository#refs/heads/master"}: List remote refs failed: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 

Your corporate certificate is not known by TeamCity. This is how you fix the problem.

Save your corporate root certificate into CompanyRoot.cer file

For server side checkout: 

1. Login to the TeamCity server machine
2. Add this root certificate to keystore of your TeamCity 

%TeamCityInstallDir%\jre\bin\keytool.exe  -importcert -trustcacerts -file CompanyRoot.cer -alias ca_alias -keystore "%TeamCityInstallDir%\jre\lib\security\cacerts"

(replace %TeamCityInstallDir% with actual path to your TeamCity)

If you haven't ever changed it, password for the keystore is changeit :)

For agent side checkout:

1. Install Git on agent machine to lets say C:\Programs\Git\
2. Run on agent machine git config --global http.sslCAInfo D:\Programs\Git\bin\curl-ca-bundle.crt
3. Edit curl-ca-bundle.crt manually, add your Company root certificate to the end of the file

Mor information can be found here: 

http://blogs.msdn.com/b/phkelley/archive/2014/01/20/adding-a-corporate-or-self-signed-certificate-authority-to-git-exe-s-store.aspx

Howto: Set up TeamCity to use TFS Proxy

TeamCity usage of TFS as a source provider is not exactly perfect, but setting up fetching data using TFS Proxy worked surprisingly well!

So, we have some TFS server and a configured TFS proxy with address, let's say http://tfsProxy:8081
and now we need to explain TeamCity that it should fetch files using it.

Configure VCS root pointing to TFS server as ususal, do not think about proxy yet.

And now - all you need to do is to set up an environment variable TFSProxy with the value of TFS Proxy address.
Then restart TeamCity agent - if you use agent side checkout. Or restart TeamCity server - if you use server side checkout.

NB! Sending TFSProxy as an environment variable to build will not work. It MUST be an environment variable on a machine which is fetching code.

Testing scenario:
Set up a build to fetch some data from TFS.

1. Clean up TFS Proxy cache by deleting all under: %TFS_Proxy_install_directory%\Version Control Proxy\_tfs_data\Proxy

2. Clean up TeamCity cache:

• Server side checkout - delete all under %TeamCity Data Directory%\system\caches\sources
• Agent side checkout - delete all under %localappdata%\Microsoft\Team Foundation\5.0\Cache

3. Clean up build directory from source files, if not empty.
4. Start build and see Cache folder getting populated on TFS Proxy machine :)

Hope that helps!

3 things to know to set up DC in Azure

I assume some have already configured domain controller on-premise. It is almost the same, when you do it in Azure, accept for few things. But those are important things.

1. Set a static IP for your domain controller
Actually your IP address stay persistent for 45 years if you do not stop your VM. But if you do stop\deallocate your VM - you might get another IP address after start. And that is (softly said) not so good for domain controller and DNS server.
So make sure to confiure a "static" IP for your DC, like this, using powershell:
(If my vm name is myTestDC and corresponding cloud service name is myTestDC)

$dcvm = Get-AzureVM -ServiceName myTestDC -Name myTestDC
Set-AzureStaticVNetIP -VM $dcvm -IPAddress 192.168.0.4 | Update-AzureVM

2. Configure location of NTDS and SYSVOL to be on Data disk, not OS disk
Azure provides us with 2 types of disks - OS and Data. Where as OS disks are using caching by default, and Data - not. So make sure to attach an extra Data disk to your DC and configure your ntds and sysvoll to be placed there.

3. Modify you network DNS to include new DNS server

Existing vm-s will need restart to pickup new DNS settings.

x. And another, rather tiny but important thing :) Administrator username is not allowed to use in Azure, so - say my administrator name is superAdmin, password superPassword, and I created domain superDomain.Test with netbios name superTest.  Logging into my DC after restart is superTest\superAdmin with password superPassword.

PS: Howether, if you demote your DC - suddenly you will need to login with .\administrator and password superPassword. ...Assume it is a bug, not a feture.

TFS to Git migration: step by step

It has been some posts about migrating to Git from TFS, unfortunately none of them worked for mas expected.

So, my task was: migrate a source from TFS to Git, preserve all history, but do not include dlls or other heavy files. Last part requires naturally not only removing those files from the latest source, but also removing it from the entire history.
On my way I met some hickups, which I will mention, so the road for others after me can be smooth :)

So, lets start!
You will use GitBash and Git-tf:

1. Install Git-TF. Follow this link and download latest release:  http://gittf.codeplex.com/
 Installation instruction is very easy, I chosen not to use Chocolatey, but changed PATH environment variables, worked perfectly.

2. Start GitBash. If you do not have it - can be downloaded from here

Note! All things from now are done from under GitBash.

Check if git-tf was installed successfully - just type "git tf", it should return usage description of git tf command.

Navigate to the place where you have you Git repositories. I use C:\Git\Sources\Repos folder. So my git promt looks like this now:

3. Clone my TFS source to some temporary Git repo.
git tf clone {collection url} {team project path} {temp repo name} --deep

like this:

Note --deep - that means I want to take history as well.

4. Create Git repository which you will use later as your Git source. So far I will not link it to TFS Git repository, just create it.
git init {repo name}
like this:

get into newly create repository by simply
cd {repo name}
you can see you are in a master branch of your newly created repository:

5. Being there, now pull the source from you temp TFS git to your future Git place:
git pull ../{temp repo name} --depth=100000000

teoretically we could push it to our TFS git already now, but first lets cleanup dlls or other unneeded files, so when we put it to TFS git it is all clean and lightweight.

In my example I had 2 installation files located in folder wix/wixDownloads. So I want to leave those behind.

git filter-branch --force --index-filter 'git rm -rf --cached --ignore-unmatch wix/wixdownloads'

Command works only from the root, so paths must be relative from there.

If I would need to delete only one file, I would type:
git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch wix/wixdounloads/wix35.msi' 

It is important it says was rewritten in the end, that means stuff was actually removed. Otherwise - you probably misspell something.

If you type wrong, sometimes you get error "cannot remove directory '..../.git-rewrite" : directory is not empty. Dont panic, just remove it manually.

6. We are almost there! Now just clean up so all deleted files are physically deleted:
git reflog expire --all

git  gc --aggressive --prune

Now you can compare size og original forlder with clone from TFS, and size of cleaned up folder.
If everything is correct, the second one is much smaller!

7. And finally - push up to your TFS git:

git remote add master {your  TFS Git repository address}

git push -u master --all

like this: 

8. Now, when I compare hisoty on my original TFS source to history on my new TFS git source, I see no wixDownloads folder, and even not a corresponding checkin (well, this is because checkin contained JUST this folder, and no other files).

To the left - my new Git repo history, to the right - my original TFS history.

Hope it saves somebodys time!

Update 1: there is a possibility to hit  error when pulling source to the temp repository, if full filenames on temp repository exceed 256 symbols:

 error: unable to create file ...{verylongfullpath}... (Filename too long)

ignoring the error gets us on with being unable to do filter-branch and error:
fatal: Neeeded a single revision

To get around that make sure that the folder name for temp repo is short enough. In my case I just deleted temp repository, created a new one with a very short name and did git pull again.

Hyper-V: Clone a machine with SQL Server installed

The first part of this post will describe the cloning process,
the second - how to get SQL server up again after the clone machine is ready.

Part 0 : Before all

If you use Reporting services, backup service key and store separately, preferably on another machine. Remeber the password, ofc.

Part 1 : Hyper-V - clone your virtual machine

Have to say at once - I did have a possibility to stop the original machine I was cloning. If you do not have such an optyion, this is not exactly post for you. 
Disclaimer: The described method does not pretend to be the best, it just worked for me. I have tried different ways, and that one worked all the time.

I didn't turn on the original machine untill i was completely done with cloning and fixing SQL Server.

1. Shut down the original machine and export it to some location.

 

2. Copy exported machine where you want to have your clone and rename the folder accordingly. (it is absolutely safe to do that, your original machine is not influenced)

3. In your exported folder delete all files except for hard drives, which have extension .hvdx

4. Create new virtual machine at your host, giving it a new name and attch existing hard drive - the one which is exported. IMPORTANT: do not turn on the original machine yet!

5. Start your new machine. Start command line tool and run the following:
c:\windows\system32\sysprep\sysprep
in the window check Generalize.

Wait until the process is done and machine rebooted. This takes time, so fetch coffeee or smth :)

6. When the machine is restarted it asks for windows settings, since they were nuked. Provide them. Details of what sysprep does with machine is here - very nice post, helped me to understand a lot!

7. Change machine name to something meaningfull, reboot. Add it to domain if needed, reboot.

8. If you have static ip configuration, do not forget to change that as well.

9. That's it , you have a clone up and running!

And now, lets get SQL server up.

Part 2 : Reconfigure SQL Server on a cloned machine
 
The problem is that the cloned machine after renaming has old SQL server instance name (from the original machine), so we need to change that. Analysis services doesn't look happy and, naturally, Reporting services also needs to be reconfigured.

1. Start SQL Server Management Studio, try to connect to the instance with the new machine name. If it works - lucky, the process was taken care off during machine rename.

1a. If connecting with machine name as SQL Server name didn't work - provide . or localhost as a server name and connect. Create new query as:

sp_dropserver <old_machine_name>;
GO
sp_addserver <new_machine_name>, local;
GO

Restart SQL server. 

Now - Reporting services.

2. Start Reporting Services Configuration Manager. Navigate to Database tab and click Change Database. As you can see it has the name of the old machine, put the new name in, and on Next page select correct Report Server Database (ReportServer). Finish.

3. Restore Encryption Key. Go to Encryption Keys tab and click Restore. Find the key file you saved before (see Part 0 : Before all) and restore the key from there.

 

 

4. That should be it. Navigate to Report Manager URL to make sure it works.

Now - Analysis services.

1. After all that, my analysis service still was not happy. The only thing that helped was to start installation again and repair (just next next next - repair). If somebody knows how to do it the correct\better way - drop me a note, I will be really thankfull!

2. When repairing done, restart the machine. Make sure you can connect to Analysis services, SQL Server instance and Reporting Services.

3. Start your original image.

MSSQL: Shrink transaction log

Common issue, right. Let's do it.

1. Go to your database in Management Studio.
2. Right click on the database and select Tasks -> Backup. Choose Transaction Log as Backup type.

 

3. When backup is taken, right click on database, select Tasks -> Shrink -> Files. Choose File type Log.

In 95% of cases that will do the trick.

In 5% of unlucky events, log file is not getting less, but there is no error displayed. If you have that, click Script on the previous screenshot (Shrinking log file). And then run script to do the job, and click on Messages at Output window.

You will see smth like this: 
Cannot shrink log file 2 because the logical log file located at the end of the file is in use.

Then you need some extra steps.
2a. Right-click your database and select Properties. At Options tab set Recovery model to Simple.
Do Step 3.
3a. Set recovery model back to Full. 

Good luck!

Configure SSL in IIS :The process cannot access the file because it is being used by another process.

Tja... annoying! Obviously someone else is listening on 443, let's find out who.

Command prompt: netstat -ano
Looking for port 443 and memorizing PID.

Troublesome PID = 8952. Open Task manager -> Processes - sort by PID.
Ah, here it is. Skype! Kill it, and you will be able to successfully configure your web site to work over SSL.

HowTo: Run application as a user from another domain

Sometimes (pretty often actually) I need to run application as another user.
What I usually do in this case - SHIFT+right-click

and that works as charm.... unless the account I need to run under comes from another domain.
Then I get this:
 
In that case there is a neat solution from command line:

runas /user:domain\username /netonly "application.exe"

Made my life way more easy :)

The remote server was disconnected because there are no Remote Desktop License Servers...

 And nobody can do anything, since remote desktop is not working.
There is a simple solution:

Run "mstsc /admin"