The Mission

Let’s forget Trump, China, the Fed and the markets for a day. We have bigger questions to deal with. Like: what if a powerful Artificial Intelligence program were used to analyze, say, a pathetic blog? What would it think?

Now that people and computers are connecting in ways never before dreamed of something called ‘sentiment analysis’ is all the rage. And because cheap hormones and fake thinking are what this site is all about, how could it not be a perfect subject?

That was the conclusion of techie phenom Keith Stoute. He’s the founder of Visual Antidote, an IT consulting outfit and software manufacturer with a global clientele. He’s also a blog dog who has just AI’d Greater Fool. Imagine. This is like NASA looking for life on a distant, weird, freaky dead star.

“I am an avid fan of your blog (the term ‘addict’ may be more adept) and also learning technologies related to Artificial Intelligence and Machine Learning,” he writes. “I’ve decided to merge the two, and have had IBM’s Watson technology read one year’s worth of posts (including comments) and provide feedback on what it found.  Surprisingly it did not break!

“To do this I scraped all blogs (365) and comments (50,521) from August 1st 2018 to July 31, 2019, stored them in a database, and passed them to Watson one at a time (50,643 records – you’re busy).  Watson returned a measurement of what it analysed in terms of sentiment (overall positive, or negative?), sadness, joy, anger, fear and disgust.   The results are pretty fun.

“This type of technology – sentiment analysis – is one of many that is currently changing the way humans and computers can interact, and how our futures will look.  Big organisations like banks and telcos are already using it to monitor their customers interactions in call logs, chatbots and surveys.    There are implications, both positive and negative, for humanity, which makes me excited to learn and apply it (hopefully for good). I had a lot of fun compiling this and I hope your readers get some enjoyment from it too.”

Well, what follows is Stoute’s summary of Watson’s findings. By the way, this is the same program that won a million bucks beating everyone on Jeopardy. But today it’s parsing the steerage section. Put on your safety googles. Here we go.

For each blog and comment we received a score for overall sentiment (positive or negative), joy, anger, fear, disgust and sadness.  What we identified was that blogs with a ‘negative’ sentiment generally seemed to garner the most comments, which could be assumed to be a proxy for engagement.

However, it does seem that being ‘too negative’ drives comments down – see farthest left bubble.

It should also be noted that Guest-Bloggers in general seemed to be more positive than average, and they post on weekends, which might account for some of the lower comment counts on the positive end of the scale.  For example, guest bloggers account for 19 of the 50 most positive blogs, but only 6 of the 50 most negative.  The 10 most positive blogs had an average of 97 comments, while the 10 most negative blogs had an average of 130 comments.

The most commented-on blog ranked number 20 out of 365 total, as measured by fear (0.59797).

The real Insanity: the Comments Section:

The Greaterfool.ca has a lively comments section, filled with a wide variety of characters who opine on and off topic, often without decorum, taste or manners.  This makes for a compelling challenge for Watson. Luckily we did not break it with the insanity it had to endure!   Sadly, the most prolific commenter has a pseudonym I will not repeat on my blog, but pertains to something that can occupy a crowded elevator you do not want to be in (how’s that for a Jeopardy question?).

Below is a chart of the top 10 most prolific commenters.

Click to enlarge

If you can believe it, “Mr. Elevator” posted 1,577 comments in 12 months, or an average of 4.3 comments every day.  This is just over double  the amount of the 3rd most prolific commenter.

The regular contributer ‘For those about to Flop’ would have been in the top 10 had he posted consistently under one name, but seems to flop back and forth between his long formal name (463 comments) and ‘flop’ (183 comments). So he is aptly named.

Speaking of Flop, who likes to post regularly about ‘how much’, let’s rank our top comments as evaluated by IBM’s Watson.

Just a brief note about ranking: Watson returns a value between -1 and 1 as a measure of a sentiment or emotion.  So an anger value of 0.9 means you are really angry, and -0.9 means you are not angry AT ALL.

Top Positive Comment
“Thank you so much for your excellent answer to my question! I really enjoy your writing and I look forward to every post. Kind of you to share your knowledge with us. (The pictures are cool too!)” – Ian “Blog Dog”, Nov 26, 2018. Sentiment rank: 0.999947

Top Negative Comment
“Yet another pic of a stupid canine failing the IQ test. (Too bad dog owners are equally of lower intelligence, or they could save their poor dumb puppies from their own stupidity.)” – Felix, Sep 23 2018. Sentiment rank: -0.998314

Top Joyful Comment
“Thank you, Garth. Such wonderful stories. Happy New Year to you and your family.” – Chris, Dec 30 2018.  Joy Rank: 0.991983.

Top Angry Comment
“Enjoy life don’t, worry about the small shit. Gamble bitches.” – Smoking Man, Apr 4 2019. Anger rank: 0.935145

We knew SM would appear here somewhere. Frankly, this does not strike me as all that angry, I like rank #10 as the most angry:

“If the lazy parasites communist CONservatives want t No B-20 that is fine but they should DEMAND the shut down of CMHC. They always hate government interference but since it helps those useless communists they dont say boo about CMHC. Proving CONservatives are nothing but CON artists and SHYSTERS who hate the free market but love communism for the rich.” – Communist CONservatives, May 14, 2019. Anger rank: 0.853132

Top Disgust Comment
“Progressives and their “ideals” sure are revolting and disgusting.” – Classical Liberal Millennial, May 5, 2019. Disgust rank: 0.91043

Top Fear Comment
“oh boy I fear for the comments section today. I’m out.” – yorkville renter, Feb 28 2019. Fear rank: 0.947706

Top Sad Comment
“My Dad did the same as Nance:( He won’t tell me but at 88 I’m sure he’s broke now. Just CPP and OAS to live on. Sad AF!” – Honey Dripper, Sep 13 2019. Sadness rank: 0.965764

How we did it:

IBM Watson is “IBM’s suite of enterprise-ready AI services, applications, and tooling.” The area of AI we are using is known as Natural Language Processing and Sentiment Analysis. Basically you give a computer system some phrases in a human language like English, and it will return its interpretation of some aspects of it, such as:

• Overall sentiment – is it positive or negative?
• Emotions – what emotions are involved, and how strong are they?
• Entities – what are the topics or categories being discussed?

Organizations use this to process large volumes of data to look for specific trends.  For example, a call centre at a bank could transcribe call logs and monitor realtime if customers are becoming emotional in one direction or another, or companies can analyze huge volumes of their customer survey results for meaningful insights into how their customers or market is feeling.

To get all this data we built a web scraping tool using Python to retrieve publicly available content from the Greater Fool blog. From there we passed it to Watson’s API and recorded the results in a database for further analysis. From there we did further analysis with SQL, Python and Excel.

We ran our approximately 56,000 records against Watson without tweaking or tailoring Watson to understand the culture, sarcasm or irony prevalent on the Greater Fool blog.   For example, it deemed this post as overwhemingly happy and positive!

“Happy Housing Crash Everyone! The days of SHYSTER lies are over. I can’t wait for a uber type realtor. Could we see $1000-$5000 to sell a home?  Happy Housing Crash Everyone!:-)”

It reads like a biblical pestilence is being lifted… how could that not be happy?  To be fair Happy Housingcrash Everyone’s average sentiment across 21 comments was negative.