Archives For NodeJS

This article is also available on the RedBit Site

Tracking and visualizing social media streams in real-time and across multiple social networks can be a challenge. Three months ago we set out to address this challenge, in partnership with our friends at IdeaNotion and Microsoft.

Today we’re pleased to release as Open Source the Social Cloud project.

social cloud logo-blue

Social Cloud allows users to monitor various social media channels such as Twitter, Instagram and FourSquare, aggregate and process that data (e.g. generate word clouds) and expose real-time streams to web & mobile clients.

The entire solution consists of the following:

  1. NodeJS “emitters” on Linux VMs running concurrently to monitor social media  streams
  2. NodeJS “server” exposing streaming data to front end visualization systems
  3. An admin web dashboard built in NodeJS & Bootstrap
  4. Native iPhone, Android and Windows Phone admin clients built using Xamarin
  5. Email, SMS, push notification & in-app alerts using SendGrid, Twillio & Pusher
  6. HTML5/Canvas visualization engine powered by IdeaCloud

Here is an architectural overview of the system that was built using Windows Azure

architecture

On the outside, seems like a pretty easy concept, but internally there are a lot of components to make this an efficient scalable system. Here is a list and description of all of the features used for the system.

Azure Virtual Machines

Windows Azure has the ability to run Virtual Machines or VMs in the cloud. The ability to spin up a VM in a few minutes is great from a development standpoint because we can spin up test boxes with minimal effort and minimal network infrastructure.

Social Cloud leverages this feature and we utilize Linux VMs to run our NodeJS processes. We went with a Linux VM because of WebSockets support and we needed a little more control than what Azure Websites provided. Also, the development team felt more comfortable with Linux VM instead of Windows and allowed us to easily start multiple NodeJS instances to monitor social media streams.

arch-VMs

As you can see from the diagram, there are multiple NodeJS instances running each monitoring their own social network. In the case of Twitter, we were able to use their streaming API for realtime results while the others are using a polling techninque.  Once we receive the data, we store it in a mongoDB instance and pass a message off to Azure Service Bus for downstream processing.

arch-serviceBus

Windows Azure Service Bus

Social Cloud uses the Windows Azure Service Bus to detach the process of gathering the various social messages from the processing of said messages to ensure that we can receive the messages as quickly as possible.  One of the different processes that subscribe to the Azure Service Bus topics are the Word Emitter that listens for text based messages and splits them into the words, processes them, then places new messages back onto the Bus.  Another subscription listener is the Socket.IO process that passes incoming messages from the bus to each of the listening Socket.IO clients.  This separation also provides the benefit of being able to withstand one component failing, as the messages are queued until the process is brought back online.

arch-serviceBusClients

Windows Azure Websites

Social Cloud leverages Windows Azure Websites to run the administration dashboard which is written using NodeJS. This site also provided APIs to be accessed by iPhone, Android and Windows Phone apps. We leveraged the Git deploy feature available and the integration with BitBucket for automated deployments.

dashboard

arch-websites

Windows Azure Mobile Services

Windows Azure Mobile services are one of the new features that is still currently in preview mode but we still did manage to use some of the features available for the Social Cloud solution.

Some of the features used are

  1. Data – this feature allows you to easily store data within your system from clients accessing the system. Specifically we used it to store device information when implementing push notifications. It also allows you to easily set permissions to read/write the data and also write custom scripts to run when an insert, update, delete or read operation is performed on the data
  2. APIs – APIs is a relatively new feature and essentially allows developers to implement some custom REST APIs required for the system. It also easily allows developers to add support for permissions to any operation performed.
  3. Push – this feature allows developers to create native push notifications to Windows 8, Windows Phone 8, Android and iOS clients. This is great as you can quickly get this infrastructure setup in minutes.
  4. Identity – authenticating users is usually required in mobile apps. Social Cloud required it as we only allowed authorized users to access the data. Using the Identity feature we were able to quickly add authentication to things such as Google, Facebook, Twitter and Microsoft Account.

Essentially, Azure Mobile Services allowed us to quickly implement features such as authentication on mobile clients, and let us build out the features of Social Cloud and not worry about plumbing required for things such as authentication.

arch-mobileServices

Windows Azure Store

For Social Cloud, we leveraged various third party components to add features and functionality to the system. The store allowed us to quickly find and purchase the third party services relatively quickly to be integrated into the Social Cloud system. The following third party components were used in the system

  • MongoLab – store all the streaming data coming in
  • Twillio – send text notifications
  • SendGrid – send email notifications
  • Pusher – send notifications to clients connected on the web dashboard
  • New Relic – monitor the health of the system

arch-store

Social Cloud is a pretty extensive system requiring many different custom components and third party components to produce the required functionality.  Using Windows Azure and the different features available such as Azure Mobile Services, Virtual Machines and Service Bus allowed us to build a robust system quickly and easily.

Over the next few days we’ll be posting some technical blog posts on the system but as of now all the source code is available on GitHub. Be sure to monitor this blog or follow me or the RedBit team on Twitter to get notified when the code is released.

 

In two previous articles, I went through a technical overview of Windows 8 Finder App Template and the companion Finder App Server.  In this article, I’ll go through the Hero App Template and Hero App Server which are built on top of the Finder App Template and Server.

Introduction

The Windows 8 Hero App Template is a template developed to allow developers to quickly and easily create an application to collect certain information while out in the field on a Windows 8 device. For example, if a developer wanted to create an app to allow people to submit information on roads needing pothole repairs, the Hero App Template is a good starting point.

The Hero App Template is a derivative of the Windows 8 Finder App Template described in a previous article and The Hero App Server is a derivative of the Finder App Server described in a previous article.

This article will go through some of the technical features of the Hero App Template and accompanying NodeJS server to store all information. It’s assumed you have already gone through Windows 8 Finder App Template and Finder App Server articles. Continue Reading…

I recently released two Windows 8 templates on GitHub with an accompanying quick start guide for the Finder App Template. This article will go through the optional NodeJS backend server and how to get started with it and connect it to the Windows 8 template.

Introduction

The Windows 8 Finder Template is a template developed to allow developers to quickly and easily create a location based type application for the Windows 8 store.  It allows a developer to easily pull existing point of interest data into the app to display on a map. If no point of interest data is available, it is accompanied by a backend server which allows the importing of data which can be used to feed the Windows 8 application.

In Part 1 of this article we went through the Windows 8 template and set it up to consume various feeds to display on a map.

In this part, we will go through some of the features of the NodeJS backend server and how to connect to the Windows Finder App Template.

NOTE: the backend server is optional and not required for the Windows 8 Finder App Template. Continue Reading…