Do it yourself social media monitoring – not for the faint hearted or the IT department

 In social media

DIYFor the second time in as many months, a potential client has informed me that “the internal IT team are developing and evaluating their own home-grown social media monitoring platform”.

This either points to the fact that the entire social media monitoring industry has failed, or the IT departments think they can build a better/faster/cheaper widget.

Over 12 months ago a client asked me if he thought it was possible to “build your own” social media monitoring tool.  To cut  a long story short, I looked at all of the options for him (including building it internally) and then approached Visible Technologies about bringing their solution to the UK.  The rest they say is history.

That said, I think I am well placed to comment on the buy vs build question when it comes to social media monitoring platforms and the different approaches.

I suppose I am unlikely to believe that an internal IT team can build an enterprise grade social listening platform – and for that reason I know that both organisations concerned use enterprise grade servers, email platforms and desktops.

As far as I know, they have not “built their own” email platform, server farm or computers – instead relying on those companies that have this at the core of their business to develop, and integrate these mission critical platforms.

Even with the invention of cloud computing, I know large corporates see cost efficiencies in moving to open-source and online services, however there are pros and cons in each instance.

Some of the challenges any internal IT team will face when developing and testing their own social listening platform include:

Data capture: Last time I looked, social media doesn’t shut down overnight.  That means that there is a constant stream of information being created (including some 90 million tweets per day according to Shiva Rajarman from Twitter speaking at Ad:Tech London in September) in real time.

I am sure many IT teams are used to collecting data themselves in large quantities, but unless you are geared up specifically to grab and process real time data, then something will be missed.

I am sure IT departments also don’t have a lot of experience about what to capture.  For example, as part of a social media monitoring strategy, you would not normally just collect mentions of a company’s brand, but also relative competitors, executives, market segments, future products, hiring mentions etc. These diverse data sources need to be collected using key search phrases to weed out the spam and irrelevant posts.

You see you will still need to collect a lot more data than you use, and process it to remove spam, duplicates etc and then provide the rest of the organisation something they can actually use on a daily basis – that is actionable insight, not just more raw data to process and interpret.

Data storage: As per the section above, social media literally generates petabytes of data.  Is the IT department looking at a solution that can grow as the social media firehose is opened wider and wider?  Who is going to pay for all of this additional storage?

Spam: Social media is full of spam, and mis-formed data. How can you reliably clean all of the data coming into a home grown tool without having some level of understanding of where to look for the data and what to discard.

Duplication: This is one of the biggest challenges faced by social media monitoring providers is to detect and remove the masses of duplication (or analyse it to see who is repurposing/re-tweeting the same information).

Conversation: It is fairly simple to grab RSS feeds from social media, but how about those blogs, forums and traditional media sites that invite comments.  Here it is important to grab not only the initial post, but also the subsequent comments – in context and where relevant.  Building a system to extract comments in context and re-assemble them for a marketing or customer care team to respond to is not a trivial task.

Sentiment: There is an on-going debate both within the industry and in the broader marketing community about the value of automated vs human sentiment analysis .  The merits of this warrant a complete blog post of their own – but let me look at this from a different angle.

While it is useful to understand if someone “loves” or “hates” our brand – but why is this?  Automated sentiment can only go so far and look at the aggregated level of a post – ie is the whole post good or bad.

Consider this fictitious example post or comment on a blog or forum:

“I really like my new iPhone – I grabbed it from Orange but their signal is pretty poor where I live so I am thinking of getting out of my contract early and moving to Vodafone – people tell me they have a better network.  Having said this they are probably all as rubbish as each other”.

What is the overall sentiment of this post? An academic approach would say mixed.  The important thing here is that there are multiple pieces of information and sentiment.  To break it down into something we could actually use and share with different areas of the company:

iPhone – positive – “I really like my new iPhone”

Orange coverage – negative – “their signal is pretty poor where I live”

Orange retention – negative – “thinking of getting out of my contract early”

Vodafone acquisition – positive – “moving to Vodafone…they have a better network”

Mobile industry – negative – “probably all as rubbish as each other”

So – many levels of sentiment and this information would be of interest to different parts of the organisation.  At Visible, we have been using hybrid human/automated sentiment analysis.

There is a level of sentiment that can be determined automatically, but to allocate it into an actionable form, you still need humans.  See the screenshot above of one of the Visible tools at work – showing how sentiment (good/neutral/bad/good&bad) can be scored against different areas of interest.

assignedWorkflow: If you want to be able to action any of the insight you capture using free/DIY or enterprise grade tools then you will want  to have some sort of workflow management and audit trail built in.  I am sure for many large corporates having an audit trail of messages coming in and out of the company via social media will be mandatory

Insight: Even if the IT team can give you all of the data you think you need – then what?  You need a way of making sense of the data and reporting this insight to the wider business.  I can just imagine the IT boffin showing off their latest DIY monitoring tool to a Marketing head and the question being raised “looks good, but what does this actually mean for the business”.

Data is useless unless it is relevant data and you can gather some actionable insight from it.  At Visible, we offer deep professional services on top of the data to ensure that top management can actually make some decisions based on what is being captured.

Perhaps I am wrong and the two companies in question have all of these bases covered, but I’m prepared a small wager they haven’t.  In fact one of the companies has already sheepishly asked for me to come back in and provide some “social media advice”.

In my opinion social media monitoring is not something that can be completely developed in-house. If you would like to see how a dedicated social media monitoring vendor like Visible Technologies does it, feel free to contact me or tweet me @andrewgrill

While you are here ...

Did you know that Andrew also speaks regularly on topics such as this at conferences and events around the world.

You may wish to view Andrew's extensive speaking portfolio at practicticalfutur.ist.

If you enjoyed this blog post you may like other related posts listed below under You may also like ...

To receive future posts you can subscribe via email or RSS, download the android app, or follow me on twitter @andrewgrill.

About 

Based in London, Practical Futurist and former Global Managing Partner at IBM, Andrew Grill is a popular and sought-after presenter and commentator on issues around digital disruption, workplace of the future and new technologies such as blockchain. Andrew is a multiple TEDx and International Keynote Speaker.