Monitoring Social Media with Social Cloud

This article is also available on the RedBit Site

Tracking and visualizing social media streams in real-time and across multiple social networks can be a challenge. Three months ago we set out to address this challenge, in partnership with our friends at IdeaNotion and Microsoft.

Today we’re pleased to release as Open Source the Social Cloud project.

social cloud logo-blue

Social Cloud allows users to monitor various social media channels such as Twitter, Instagram and FourSquare, aggregate and process that data (e.g. generate word clouds) and expose real-time streams to web & mobile clients.

The entire solution consists of the following:

  1. NodeJS “emitters” on Linux VMs running concurrently to monitor social media  streams
  2. NodeJS “server” exposing streaming data to front end visualization systems
  3. An admin web dashboard built in NodeJS & Bootstrap
  4. Native iPhone, Android and Windows Phone admin clients built using Xamarin
  5. Email, SMS, push notification & in-app alerts using SendGrid, Twillio & Pusher
  6. HTML5/Canvas visualization engine powered by IdeaCloud

Here is an architectural overview of the system that was built using Windows Azure


On the outside, seems like a pretty easy concept, but internally there are a lot of components to make this an efficient scalable system. Here is a list and description of all of the features used for the system.

Azure Virtual Machines

Windows Azure has the ability to run Virtual Machines or VMs in the cloud. The ability to spin up a VM in a few minutes is great from a development standpoint because we can spin up test boxes with minimal effort and minimal network infrastructure.

Social Cloud leverages this feature and we utilize Linux VMs to run our NodeJS processes. We went with a Linux VM because of WebSockets support and we needed a little more control than what Azure Websites provided. Also, the development team felt more comfortable with Linux VM instead of Windows and allowed us to easily start multiple NodeJS instances to monitor social media streams.


As you can see from the diagram, there are multiple NodeJS instances running each monitoring their own social network. In the case of Twitter, we were able to use their streaming API for realtime results while the others are using a polling techninque.  Once we receive the data, we store it in a mongoDB instance and pass a message off to Azure Service Bus for downstream processing.


Windows Azure Service Bus

Social Cloud uses the Windows Azure Service Bus to detach the process of gathering the various social messages from the processing of said messages to ensure that we can receive the messages as quickly as possible.  One of the different processes that subscribe to the Azure Service Bus topics are the Word Emitter that listens for text based messages and splits them into the words, processes them, then places new messages back onto the Bus.  Another subscription listener is the Socket.IO process that passes incoming messages from the bus to each of the listening Socket.IO clients.  This separation also provides the benefit of being able to withstand one component failing, as the messages are queued until the process is brought back online.


Windows Azure Websites

Social Cloud leverages Windows Azure Websites to run the administration dashboard which is written using NodeJS. This site also provided APIs to be accessed by iPhone, Android and Windows Phone apps. We leveraged the Git deploy feature available and the integration with BitBucket for automated deployments.



Windows Azure Mobile Services

Windows Azure Mobile services are one of the new features that is still currently in preview mode but we still did manage to use some of the features available for the Social Cloud solution.

Some of the features used are

  1. Data – this feature allows you to easily store data within your system from clients accessing the system. Specifically we used it to store device information when implementing push notifications. It also allows you to easily set permissions to read/write the data and also write custom scripts to run when an insert, update, delete or read operation is performed on the data
  2. APIs – APIs is a relatively new feature and essentially allows developers to implement some custom REST APIs required for the system. It also easily allows developers to add support for permissions to any operation performed.
  3. Push – this feature allows developers to create native push notifications to Windows 8, Windows Phone 8, Android and iOS clients. This is great as you can quickly get this infrastructure setup in minutes.
  4. Identity – authenticating users is usually required in mobile apps. Social Cloud required it as we only allowed authorized users to access the data. Using the Identity feature we were able to quickly add authentication to things such as Google, Facebook, Twitter and Microsoft Account.

Essentially, Azure Mobile Services allowed us to quickly implement features such as authentication on mobile clients, and let us build out the features of Social Cloud and not worry about plumbing required for things such as authentication.


Windows Azure Store

For Social Cloud, we leveraged various third party components to add features and functionality to the system. The store allowed us to quickly find and purchase the third party services relatively quickly to be integrated into the Social Cloud system. The following third party components were used in the system

  • MongoLab – store all the streaming data coming in
  • Twillio – send text notifications
  • SendGrid – send email notifications
  • Pusher – send notifications to clients connected on the web dashboard
  • New Relic – monitor the health of the system


Social Cloud is a pretty extensive system requiring many different custom components and third party components to produce the required functionality.  Using Windows Azure and the different features available such as Azure Mobile Services, Virtual Machines and Service Bus allowed us to build a robust system quickly and easily.

Over the next few days we’ll be posting some technical blog posts on the system but as of now all the source code is available on GitHub. Be sure to monitor this blog or follow me or the RedBit team on Twitter to get notified when the code is released.


Wceload Replacement

Have you ever tried to use wceload to install your application and had a complete headache to get it working?  Ever wish you could just get ride of wceload and have more control on your installation process?  If you did, you should definitely check out our new CAB Installer SDK which will help you install your applications with ease and more importantly give you the developer more control.

Check out Chris’ announcement post here and the CAB Installer SDK product page here.

This product is being sold under our new ‘Value Based Pricing Model’ which basically means you pay what you think it’s worth.  See here for more details on the new pricing model.

As always feedback is welcome!

Task Manager 2.0

Back in September 2007 we released our first version of Task Manager (formerly called TaskManCF). 

Last week we released Task Manager 2.0.  A few major things we have added to this version:

  1. Changed the name from TaskManCF to Task Manager.
  2. Improved the UI for Windows Mobile Professional
  3. Ability to switch to application
  4. Ability to find an applications associate process
  5. Ability to kill all running applications

One of the more exciting things is we decided to get Task Manager logo certified and it is now we can proudly say Task Manager is “Designed For Windows Mobile 6”!.

Of course this does not mean it only runs on Windows Mobile 6, it also supports Windows CE, Pocket PC and Windows Mobile 5.  Only requirement is .NET Compact Framework 2.0 which can be downloaded here.

Here is a screen shot of Task Manager running on Windows Mobile 6 Professional device

Currently it does not support Windows Mobile Standard (aka SmartPhone) but if there is a need for it we can add that.

Go ahead and download the Evaluation version and let us know what you think and if you want other features added!

Windows Mobile 6.1 Welcome Center and Compact Framework

For those not familiar, Windows Mobile 6.1 comes with a new application called Welcome Center which basically is a help system for users. 


I was running some hopper tests on Task Manager 2.0 and looking through the hopper.log file I noticed the following line:

STATS: \Windows\welcomecenter.exe -> #NETCF_AGL_PARK_\Windows\welcomecenter.exe: Time in State =          0 secs ( 0.0%); Actions =      2; Visits =      2; Last visit =    196 actions ago

If you noticed #NETCF_AGL_PARK_ then you know that this means the application WelcomeCenter.exe uses Compact Framework and this is a welcoming surprise.  Let’s hope this trend continues with other apps from the Windows Mobile team and other teams within Microsoft!

As for Task Manager 2.0 here is a screen shot of the new version.


And yes we do know that Windows Mobile 6.1 has a Task Manager included but ours does not filter out any processes or applications, allows you to terminate processes, end applications, terminate threads, change thread priorities and view all threads and modules loaded for a process.  The same binary also works on different platforms such as Windows CE, Pocket PC 2003, Windows Mobile 5 and Windows Mobile 6. It’s also written using Compact Framework  and leverages the Smart Device Framework.

ASP.NET on Windows CE

We have just released a new product called Padarn Web Server for Windows CE.  What exactly is it?  Well it’s a very small-footprint ASP.NET Web Server built to run on Windows CE 5.0+.  Think about WIFI routers, set top boxes or any other embedded device that would need some sort of a web UI to configure it.  Or possibly you need to do some reporting or expose some services to configure the device remotely. With Padarn you can leverage your managed code experience and build web apps faster.  For example take a look here to display the device system information. Here is the source to display the device system info. Want to see other examples then check out and check out the product page here.

Popup Menu Control

We have just released an update to our Popup Menu Control which resolves a few bugs our customers have found.  If you have never heard of the control then in short what it allows you to do is display a customized Context Menu in your .NET Compact Framework application.  By customized I mean showing icons, background images, background/foreground colours and owner drawn capabilities.  It’s supported on both Windows Mobile and Windows CE

Here are a screen shot of what can possibly be done.

By setting the PopupMenuItem.OwnerDraw property to true, you as the developer have the option to customize the menu item however you like.  Here is a new sample I just created showing a gradient context menu:


Download the evaluation version and try out the samples.

Media Player Control for Windows Mobile

We recently released Media Player Control for Windows CE and we have got a few requests asking if it works on Windows Mobile.  Unfortunately this product does not work on Windows Mobile. 

Today we have just released Media Player Control for Windows Mobile Beta 1 for all the Windows Mobile developers out there that want to easily add media to their .NET Compact Framework applications.

Here is a screen shot of the application playing a WMV trailer of Rush Hour 3


The product is currently in beta and if you would like to participate in the Beta the sign up instructions can be found here.

Recent Product Releases

We recently released two new products for Windows Mobile and Windows CE developers. 

The first one is TaskManCF which runs on Windows Mobile (no SmartPhone) and Windows CE devices.  It allows you to view all applications running, processes, threads etc.  Here is a screen shot of the product running on a WM5 device.

The second product we released is a Media Player Control for Windows CE devices.  If you are a Compact Framework developer and want to add video media directly into your application then check out the this control.  It allows you to add the video window directly onto a form so no loading up Media Player and hacking it to play video.  We created a ‘Video Kiosk’ sample that highlights this feature.  Here are some screen shots.

Main Screen:


Video Playing:


The Media Player Control for Windows CE is only for Windows CE devices (as the name implies) and not supported on Windows Mobile.  We are working on a version for Windows Mobile devices so stay tuned…

Telephony Library Released

Neil talks about the new TAPI Managed Library OpenNETCF just released.  If you ever wanted to use the Telephony API  within your .NET Compact Framework application and don’t want the hassle of figuring out the P/Invokes then the Telephony Library may be for you.

Give it a try and download the evaluation version here and click here for more information on this and other products available from OpenNETCF.