Tracking and visualizing social media streams in real-time and across multiple social networks can be a challenge. Three months ago we set out to address this challenge, in partnership with our friends at IdeaNotion and Microsoft.
Today we’re pleased to release as Open Source the Social Cloud project.
Social Cloud allows users to monitor various social media channels such as Twitter, Instagram and FourSquare, aggregate and process that data (e.g. generate word clouds) and expose real-time streams to web & mobile clients.
The entire solution consists of the following:
- NodeJS “emitters” on Linux VMs running concurrently to monitor social media streams
- NodeJS “server” exposing streaming data to front end visualization systems
- An admin web dashboard built in NodeJS & Bootstrap
- Native iPhone, Android and Windows Phone admin clients built using Xamarin
- Email, SMS, push notification & in-app alerts using SendGrid, Twillio & Pusher
- HTML5/Canvas visualization engine powered by IdeaCloud
Here is an architectural overview of the system that was built using Windows Azure
On the outside, seems like a pretty easy concept, but internally there are a lot of components to make this an efficient scalable system. Here is a list and description of all of the features used for the system.
Azure Virtual Machines
Windows Azure has the ability to run Virtual Machines or VMs in the cloud. The ability to spin up a VM in a few minutes is great from a development standpoint because we can spin up test boxes with minimal effort and minimal network infrastructure.
Social Cloud leverages this feature and we utilize Linux VMs to run our NodeJS processes. We went with a Linux VM because of WebSockets support and we needed a little more control than what Azure Websites provided. Also, the development team felt more comfortable with Linux VM instead of Windows and allowed us to easily start multiple NodeJS instances to monitor social media streams.
As you can see from the diagram, there are multiple NodeJS instances running each monitoring their own social network. In the case of Twitter, we were able to use their streaming API for realtime results while the others are using a polling techninque. Once we receive the data, we store it in a mongoDB instance and pass a message off to Azure Service Bus for downstream processing.
Windows Azure Service Bus
Social Cloud uses the Windows Azure Service Bus to detach the process of gathering the various social messages from the processing of said messages to ensure that we can receive the messages as quickly as possible. One of the different processes that subscribe to the Azure Service Bus topics are the Word Emitter that listens for text based messages and splits them into the words, processes them, then places new messages back onto the Bus. Another subscription listener is the Socket.IO process that passes incoming messages from the bus to each of the listening Socket.IO clients. This separation also provides the benefit of being able to withstand one component failing, as the messages are queued until the process is brought back online.
Windows Azure Websites
Social Cloud leverages Windows Azure Websites to run the administration dashboard which is written using NodeJS. This site also provided APIs to be accessed by iPhone, Android and Windows Phone apps. We leveraged the Git deploy feature available and the integration with BitBucket for automated deployments.
Windows Azure Mobile Services
Windows Azure Mobile services are one of the new features that is still currently in preview mode but we still did manage to use some of the features available for the Social Cloud solution.
Some of the features used are
- Data – this feature allows you to easily store data within your system from clients accessing the system. Specifically we used it to store device information when implementing push notifications. It also allows you to easily set permissions to read/write the data and also write custom scripts to run when an insert, update, delete or read operation is performed on the data
- APIs – APIs is a relatively new feature and essentially allows developers to implement some custom REST APIs required for the system. It also easily allows developers to add support for permissions to any operation performed.
- Push – this feature allows developers to create native push notifications to Windows 8, Windows Phone 8, Android and iOS clients. This is great as you can quickly get this infrastructure setup in minutes.
- Identity – authenticating users is usually required in mobile apps. Social Cloud required it as we only allowed authorized users to access the data. Using the Identity feature we were able to quickly add authentication to things such as Google, Facebook, Twitter and Microsoft Account.
Essentially, Azure Mobile Services allowed us to quickly implement features such as authentication on mobile clients, and let us build out the features of Social Cloud and not worry about plumbing required for things such as authentication.
Windows Azure Store
For Social Cloud, we leveraged various third party components to add features and functionality to the system. The store allowed us to quickly find and purchase the third party services relatively quickly to be integrated into the Social Cloud system. The following third party components were used in the system
- MongoLab – store all the streaming data coming in
- Twillio – send text notifications
- SendGrid – send email notifications
- Pusher – send notifications to clients connected on the web dashboard
- New Relic – monitor the health of the system
Social Cloud is a pretty extensive system requiring many different custom components and third party components to produce the required functionality. Using Windows Azure and the different features available such as Azure Mobile Services, Virtual Machines and Service Bus allowed us to build a robust system quickly and easily.
Over the next few days we’ll be posting some technical blog posts on the system but as of now all the source code is available on GitHub. Be sure to monitor this blog or follow me or the RedBit team on Twitter to get notified when the code is released.
