AzureML supported R packages

To avoiid the situation when you write a custom R code and execute it in AzureML... and suddenly get an error about package is not supported. Bummer!
There is a list of R packages supported : https://msdn.microsoft.com/en-us/library/mt741980.aspx

Save yourself a trouble, have a look first!

Getting started with Machine Learning?

I am often asked what to look at if somebody wants to get started with Machine Learning. Usually I sent people to Coursera "Machine Learning" class by Andrew Ng. It's like a litmus test - after taking that one, people usually get a feeling if Machine Learning is something they want to continue with or not.

However, if you decide to continue, what's next? What kind of knowledge\skills to look at? What are buzzword in all those learning materials?

And than I found this blog post, which IMHO, summarize it all pretty well! Even addressing nesessity to understand Linear Algebra (while I usually take it for granted and never mention, but in fact - one have to know Linear Algebra. It's a corner stone of any engeneering skill) So here it is, enjoy:
http://abhijitannaldas.com/getting-started-with-machine-learning-in-one-hour/

Do not despair if it seems too much to deal at once. The learning approach can be bottom-up i.e. from theory to practice... but top-down works as well! I.e. get yourself a case and work down the limited scope of the theory nesessary to understand and develop the solution.

Understanding AzureML Web Services pricing

So, you have created a predictive experiment in Azure ML Studio, and it is time to go in production.
I am not going to explain how to deploy a web service - enough tutorials for that.
The question you need to ask yourself - how much it is going to cost me? And here come confusions. Most of people just click on deploy and go for default proposed. I haven't found any article explaining differences and options available. So, this post is about it.

At first, if we go to Azure Pricing calulator and add Machine Learning:

There is nothing here about cost of web services deployed from AzureML studio.

But click at the small information button and choose "Machine Learning pricing details" and the curtain lifts up! It redirects us to the page with explanation what is what and how much. Let's focus on the part "Production Web API pricing".

First of all, everybody understands that a web service requires some resources to spin. Those resources need to be allocated - at that is what we pay for. For standart web application those resources are allocated by creating App Sevice Plan. App Service Plans come in many cost options defines by how much compute power and how much disk place it includes. Behind an App Service Plan there is a virtual machine, so be aware - it's gonna cost no matter if you use it or not (unless the choosen plan is "Free"). Same as monthly abonement to a fitness club - you pay for it, and nobody cares if you actually visit it.

When deploy web service in AzureML studio, there is no such thing to choose as "App Service Plan". Those are for standart web services only. For AzureML web services there are 2 options: 

• "classic" Web Service 
• Machine Learning Web Service

What's the difference?
Classic is pretty much "pay-as-you-go" with cost:

And MachineLearning Web Service Plan is basically same as  "App Service Plan" but for AzureML. Prepaid, pre-allocated resources (there is a Free option - with 2 hours and place for 2 services).




Also, when creating a new ML Workspace we must now create a "Machine Learning Web Service Plan". And Free option can be used once per geo-location. So, when creating new Workspace pay attention to what you choose for Web Service Plan:

as chosing default suggestion will create new Web Service plan, and if Free one has already been used, the new one gonna cost.

Few words about scaling. As it says here, 20 concurrent requests are configured by default. If you need more - add more endpoints. Top limit is 200.

And yes, there is a new portal for just managing ML web services now - with the view for both deployment options:

I hope pricing options for AzureML web services are clear now :)

3 things I really miss in Azure Machine Learning

Azure Machine Learning is a handy tool, absolutely. If I need to run some model quickly to justify gut feeling or to have a simple overview over data, it fits really well. Or, for example, set up a web service from a Machine Learning experiment is really easy, so kudos for that!

But there are some things which annoy me time after time, which I really want to be implemented or done differently. Here is my top 3 "wish-list":

1. Navigation inside the experiment

 

mean, honestly... I can kindof accept zoom button, but for navigating inside the experiment window I really expect drag scene to work! As for know it requires moving mouse to a  side every time and scroll up or down.
Update : yey, there is a possibility to drag scene in AzureML! Just need to be enabled by clicking this button

2. Delete several datasets together
I do load and save a lot of datasets, and to delete them as they become irrelevant one-by-one is a time killer.

3. Ability to script the experiment
Yes, if I only could upload  experiment as a script... that opens so many possibilities. Like an essential one - VERSION CONTROL my experiments. Oh, don't even let me started...

Please Microsoft. Christmas is coming, and I was a nice girl :)

PS: do you have such things that annoy hell out of you? Comment!

Convert order lines to weighted graph

Let's say we have orders history with some products. We need to perform community detection as a part of market basket analysis. Order lines are like OrderId - ProductId.

First thing need to be done is to convert order lines into weighted graph. Where Nodes are products and Edges connect nodes if products were purchased in the same order.
Something like this:

Weighted means that an Edge "weight" between two products is equal to amount of times those products were bought together.

Trying to find existing code for this task and not succeding, I have created a code snippet in R, which does the conversion from order lines to weigthed graph using adjacency matrix. 

For the graph above adjacency matrix can look like this:

The ides is simple - convert order lines into adjacency matrix N x N, where N = number of products (all columns are products, and all rows are products, edges weight = number of times two products bought in the same order). And adjacency matrix is easy convertable to graph.

This approach happens to work relatively fast.

Code is located here: https://github.com/iladan/R/blob/master/codeSnippets/OrdersToGraph.R

Hope that saves somebody's time :)

How to use Microsoft R Server (Revolution R) in Visual Studio

First, you will need to install R Tools for Visual Studio followng instructions here. Then Microsoft R Server - the how-to is here.

After R Tools for Visual Studio are installed, open Visual Studio. You will see a new top menu "R Tools" available:

Click R Tools -> Options. Visual Studio options will open a tab for R settings:

Modify R Engine path to where you have your MRO-for-RRE installation (default is C:\Program Files\Microsoft\MRO-for-RRE\8.0\R-3.2.2)

Restart Visual Studio and enjoy using R server from your favourite IDE :)

Installing Microsoft R Server (Revolution R) on Windows

As we know, Microsoft bought Revolutionary R, and now it is available as a part of your MSDN ubscription. This is wonderfull! Installation though, is not still on a level with other Microsoft products, therefore - a short how-to install it post.

1. Download Microsoft R Server from msdn subscribers downloads page:

Thats's the one:

2. Download prerequisite - MRO 3.2.2 for RRE 8.0.0 from here:
https://mran.microsoft.com/install/mro4mrs/8.0.0/MRO-3.2.2-for-RRE-8.0.0-Windows.exe
as that one will not be installed automatically along with other prerequisites.

3. Install MRO for RRE 8.0.0 by running MRO-3.2.2-for-RRE-8.0.0-Windows.exe
4. Unpack en_rre_for_windows_x64_8183330.zip and run Revolution-R-Enterprise-8.0.0-Windows.exe

That's it. Some notes:

1. Yes, I had MRO 3.2.3 and MRO 3.2.4 installed, but Microsoft R server requires exactly MRO-for-RRE.
2. It will install some old components dating back to Visual Studio 2008,

Pivot in U-SQL

In my case I needed to pivot a table with date by a week day. Meaning, having table like:

Transno	Date
1223	11/01/2016
2795	12/01/2016

To get result as:

Transno	1	2	3	4	5	6	7
1223	0	1	0	0	0	0	0
2795	0	0	1	0	0	0	0

Where corresponding week day number column gets value 1, and others - 0... Outputting result to a .csv, of course.

First I select transno, day of week as a number, and some constant for future pivotting.
@res1 =
    SELECT Transno, Convert.ToInt32(Date.DayOfWeek) AS wd, 1 AS a1
    FROM dbo.LocalTransno;

Then use MAP_AGG, which is a base for pivot:
@res2 =
    SELECT Transno,           
           1 AS k1,
           MAP_AGG(wd, (int?) a1) AS mapwd
    FROM @res1
    GROUP BY Transno;

Here I create a dummy table with 1 row, just to select values from 1 to 7 (my weekdays numbers) - as a key values for pivot.
@one = SELECT * FROM (VALUES(1)) AS T(a);
@keys = 
    SELECT 
        1 AS k1, 
        Inmeta.USQLScripts.Helper.InitList(1, 7) AS dkeys
    FROM @one;

Where InitList s a code-behind function returning an object of SqlArray<int>:

  public partial class Helper  {
       public static SqlArray<int> InitList(int lower, int upper) {
            var values = new List<int>();
            for (int i = lower; i <= upper; i++)  {
                values.Add(i);
            }
            var res = new SqlArray<int>(values) ;
            return res;
        }
  }

And finally, pivotting and unrolling to comma-separated format at once:
@res =
    SELECT 
    a.Transno.ToString() + ","  + 
    String.Join(",", b.dkeys.ToList().Select(k => a.mapwd.ContainsKey(k) ? 1 : 0)) 
    AS x
    FROM @res2 AS a
    JOIN @keys AS b ON a.k1 == b.k1;
Note the trick of dummy join, to connect weekdays keys table with my data :)

And finally output - removing the quotes from string output gives a clean csv :
OUTPUT @res
TO @out
USING Outputters.Csv(quoting : false);

It's a bit of running around, but that's the only way I managed to make it work.

The singularity is near!

So... yes, the singularity feels really near now. If you haven't read any of Ray Kurtzweil books, it's about time now. Or rather yesterday.

In fact, think about it - according to all the predictions on this field, we are going to experience singularity already in our lifetime. This is not a science fiction any more, this is reality. It moves into us like a huge locomotive, and it is your choice - get smashed by it or jump on and enjoy the ride.

For 50 years ago, such a thing as cellphone was unheard. Now - nearly everybody uses smartphone as a part of daily routines, hardly imagining being without it. Cell phones technology got integrated into our life. And that's exactly what is happening with Machine Learning nowadays.

Why now? Technology has finally developed to that point where we can build some kind of artificial intelligence on it. And nevertheless - in a programming world tools and languages has evolved, enabling us to create and apply Machine Learning solutions to broad set of problems.

Isn't it exciting? Stay tuned! :)