≡ Menu

Is big data snake oil?

The business world is hypnotized by the big money promises of big data. But is it just another empty trend? To get some real answers, we need to ask the right questions.



Acclaimed by the Crowned Heads of Europe

There’s a scene in The Wizard of Oz where Dorothy, fresh from the farm, comes across Professor Marvel and his wagon-of-wonders. Marvel, obviously a con man, uses common tricks to convince Dorothy she should return home.

Big Data is like Professor Marvel in the Wizard of Oz: a snake oil salesman with a latent sense of conscience. He hawks empty promises, but he also can use his illusion for a higher purpose.

There is some good in Marvel, but you need to present him with the right situation to uncover it.

Like Professor Marvel, big data also promises to predict the future. “Look into the mighty crystal ball of user behavior” big data seems to say, “and you’ll discover a miraculous gold road leading to an Emerald City of profits.” Unfortunately, Professor Marvel faked it and the world of big data is faking it too. Professor Marvel scammed Dorothy by extrapolating information from a photograph and using gimmicks. Big data fakes it by answering the wrong questions.

Bad Structure and Even Worse Philosophy

Here’s the typical scene in board rooms today: CEO tells the CIO to hire some “data people.” Data people, mostly coming from either a marketing or a programming background, set up shop somewhere in the business or IT department. They use Python, R, Hadoop etc., software and languages meant for the job of handling millions, maybe billions of tiny bits of information. “OK, we’re ready,” the data scientists say. “What do you want to know?”

Oops! We’ve already taken a wrong turn on the yellow-brick road. Did you catch it?

Data science is SCIENCE. It is not meant to be housed under sales or IT. A data science team should report to R&D. If your company outsources its research and development, then it should outsource its data science. Without a well-mapped reporting hierarchy and a centralized research structure, any data team is set up to fall into very common error traps, ones the academic world knows all too well. 

Businesses like AT&T and Proctor & Gamble were founded on research principles, and have, by many reports, succeeded in wrangling big data. This isn’t coincidence. In my career, I’ve hopped between academic research and business/IT; I can tell you science has its own culture that needs protection to eke out trustworthy results.

Good science can’t be conducted in the tornado fields of quarterly goals. If a lab’s culture is dominated by the profit-is-king sentiment, the research is more likely to be tainted by common fallacies and decision errors. The very questions asked will be wrongly posed. In journal publishing, we see this often when a study’s funders “magically” come up with supporting results. Human biases are so predictable that transparency in funding is an explicit and strict tenet in academic research. Academics judge a study’s results on many factors, one of the most important being the source of the study’s financial support.

Knowledge for knowledge’s sake, or even a simple search for truth over illusion, is the purpose of any data inquiry.

But when misplaced data scientists are working under business mores, the path to Oz starts to crack. The data scientists don’t feel the fissures forming because their focus is on the data. Middle management doesn’t notice anything amiss because their eyes and ears are awaiting actionable information. Top management doesn’t look at much besides profit margins and quarterly goals. But these cracks grow and what eventually comes out is an expensive wagon-full of reports that have no more predictive power over the future than Professor Marvel’s crystal ball.

Start simply and give it time

The reporting structure and the philosophy are the keys to successfully wield big data into info you can use. Data science is R&D and the pursuit of truth over fantasy are the core values on which to grow any big data endeavor. Once the team is in the right place with the right attitude, start them off with data gathering. Most companies aren’t even capturing half of the data that’s available to them. Run it off a cloud server service and capture as many points of data as possible. You have to make decisions, obviously, because there exists an infinite amount of data in each user transaction. Still, I’m positive your company isn’t anywhere near exhausting its data mining capability. It will take time to hit the sweet spot, and as the world changes, so will your data points. But data is a living thing, constantly changing and fluid. There will be some basics that will always be relevant (e.g., rudimentary demographics), but be prepared to re-map your route on a regular basis.

No place like home

At the end of The Wizard of Oz, the Wizard (aka Professor Marvel) ends up floating away without Dorothy. Dorothy finds another way back to Kansas, one that had been with her all along. We have it in us to find a truthful and honest way to handle big data. Treat it like the science that it is and integrate it carefully into the business structure, and you’ll discover all you ever needed to know was right there with you the whole time.


Photo credit: Wizard of Oz still, by Insomnia Cured Here on Flickr

wizardofozprofmarvel 0 comments

2 cultural shifts I’ve noticed

a picture of an old typewriter's shift keyThe web changes everything.

This is a bold statement for me. When it comes to networked electronic communication tools, I’m a “the more things change, the more they stay the same” kind of person. People deal with strange new things in predictable ways. But the web speeds up this process like never before.

Over the last several weeks I’ve felt 2 paradigm shifts in the online ether. One shift is in a particular way people are veering from in-person norms to specific norms for online communities. The other is in using the Internet to learn a foreign language.

Silos are bullshit.

I’ve noticed more and more people are openly admitting to blocking family members or unfollowing friends on Facebook and other platforms. This action used to be equated to shunning someone in person but that association is weakening. Users are customizing the tools more to their own personal needs. This means adopting new societal rules for online communications that veer from in-person communications. Soon it will be a normal thing to exclude family members and close friends from your social platform accounts, and no offense will be taken. The excluded persons won’t see this as a slight and will understand they can contact you via other means.

This is a giant leap away from imprinting established social customs onto new communication tools. This next step, one that usually takes many decades to implement, is taking root now after only about 15 years. For mass adoption of places like Facebook, I’d say it has been really only 5 years.

The media calls this “silo-ing” or building an “echo chamber” but that’s a lot of hooey. Many great apes, including Homo Sapiens, customize tools for personal use. There is no reason to believe humans will stick with a site’s default settings and offline social norms once they get familiar and comfortable with the platform, especially not if the online culture itself is encouraging such behavior. (So go out there and admit you’ve blocked friends on Facebook! Seriously. It helps progress).  

Va bene!

Next shift I see is in the language learning field. I’ve discovered some growing grass-roots theories online about how an adult can learn another language. Instead of the traditional classroom-type book learning, some movers and shakers out there have generated apps around the immersion theory of language learning. Immersion theory is what it sounds like: learning by doing. One of the lessons from this theory is that most people use, on a daily basis, 300-1000 words. Learn the most-used words and phrases in any language and you can quickly rise to a basic proficiency. Next step in immersion theory is to get out there and have basic conversations with native speakers.

This month I’m concentrating on Italian, and in the coming months I’ll be perfecting my Spanish. Two apps have helped me learn some basic Italian: Duolingo and HelloTalk. Duolingo teaches you the most common words and phrases in basic subjects like social greeting, foods, basic body functions and needs, etc. HelloTalk is an app that matches up language learners. Right now I have a few Italian language partners who seek to learn English. We help each other with pronunciation, culture, practice, etc. After only a few days, I feel comfortable with the idea of greeting and briefly chatting with an Italian speaker.

I should add that speaking in person is the challenge. With today’s translation tools, writing in a chat room is super easy if you have a general idea of what you want to say. A few weeks ago I had a long conversation in Italian with a reporter who needed help locating an American for an article he was writing. I did this using Google Translate and my knowledge of Spanish.

The Star Trek communicator is not far off. And with voice-generation software that will be out in public in the next few years, you’ll be able to answer in your own voice in another language.

Photo Credit: C Slack on Flickr

Comment sections are a liability. Really.

street sign saying "Unnecessary noise prohibited"A European court has (correctly) decided that sites carry liability for the user comments they let stand. Here’s the short story: If your site makes money via click rates/ads, then you must monitor your user comments.

Let’s follow the money, shall we? US Newspapers, for example, claim they are defending democracy by giving the people a place for open discourse. OK. Maybe. Newspapers and a free press are key to a democratic state. BUT the REAL reason news sites want to keep (but not monitor!) comments is to MONETIZE that public discourse. Newspapers make any money from ads. Crazy-ass comment sections bring in views. No money is made if the newspapers shut down comments and/or conversations move over to Facebook et al.

So, if we look at open, non-monitored comments as a money-making, click-baiting venture, then we should hold liable those entities which allow hate and threatening speech as well as libelous slander on their pages. It’s a treacherous line: take away all the gawk-worthy comments, you take away the viewers. Don’t monitor at all and get sued. 3rd-party monitoring companies will rise up to take the slack, but they will have to devise a formula to keep the comment sections entertaining enough to attract the click rates. Professional commenting will become a work-at-home position, if it isn’t already.

But the revolution for the news industry? It will be pushed through by the courts and newspapers’ own capitalism. This will finally remake (part of) the face of online news.


Photo Credit: My Internet friend Shawn Rossi on Flickr
1 comment

The problem with early adopting

Since the whole “minimally viable product” approach came out, I’m not much of an early adopter. I’m done with being a lab rat for selfish developers and greedy start-up investors who don’t mind annoying people with their shit “products.” Silicon Valley asswipes should conduct some due diligence before they ask me to test their stupid app. They can pay my consultancy fee. Until then, I’m not their fucking app maid.

Other things have been going shitty for early adopters lately. What did early Google glass wearers get, besides a $1500 bill and shade thrown at them everywhere they went? They surely didn’t share in any Google Glass profits (if there were any). But forget Google. Start-up entrepreneurs regularly turn to crowdfunding sites to get the initial cash to get up off the ground, but their early supporters get little more than a nod when the project takes off. A little while ago, the Kickstarter supporters of Oculus Rift got shafted in the company exit to Facebook for $2billion. The early supporters didn’t even get their initial investment back, let alone share in that huge payout.

The sharing economy

RocketClub-Logo-300x300Some MIT grads have made an app to solve that problem. RocketClub is a way for early adopters to support new apps by being test dummies. (See this obviously-paid-for review of RocketClub.) In turn, the new app gives over a teeny bit of equity share in the company. This sounds like a good idea, right? Early adopters try things out for free anyway, so why not hook up potential users and developers?

Money motivation doesn’t work well for fans, that’s why. Behavioral Economists argue monetary compensation actually diminishes the participation of early adopters. It’s like asking friends to help you move. They’ll do it for free (perhaps expecting pizza and beer at the end of the day), but if you present it like a $10/hour job, they’ll turn you down. Motivation changes when money is introduced. RocketClub shouldn’t court early adopters, as the tiny bits of compensation will only decrease their interest. The company should reach out to more “normals” – people who wouldn’t ordinarily test buggy software. They will be more apt to work (by testing apps) than new-tech pioneers. Also, normals will be the ultimate end-users; it’s probably better to go directly to them if you want to know where the real bugs hide.

But here’s the real problem

It seems like the MIT grads didn’t do a lot of research, OR… they are counting on young people to spam their friends. Their main membership push is a straight-outta-the-90s pyramid scheme. Get friends to sign up and your ranking moves up on some vague-promises list.

Congratulations on joining RocketClub!  You are invite #5434.  Interested in priority access? Invites #1500 and below will get exclusive early access! Refer this URL to your friends and jump to the front of the invite list: http://rocketclub.co/invite/1943   Friends signed up via your linkYour invite # 05434 12717 2	1358 3	679  Details: Early members will get a chance to double-dip, i.e. get stock in RocketClub and get stock with our launch partners. Access to our campaign will be sent to you via email based on your invite number. Our top level rewards are reserved for the first 1,500 members, the 2nd level for the next 3,000 members, and the final level for the last 6,000 members. Refer friends to sign up using the link given above to boost your invite number.  See our launch startups.   Sincerely, Your RocketClub Team

Have your friends join so you can kick their skulls while you climb to the top on their shoulders. Fun!

This email ended my participation. If I can’t find an unsub link at the bottom of their emails, I’ll just dump it in the spam folder and forget about it. Going back to the site and deleting my account is even more effort than I’m willing to give. (And no, this isn’t a disguised call for you to use that URL. I’m just too lazy to blot it out. If I wanted to participate in spamming my readers, I’d just tweet out the URL).

Who knows? Maybe they’ll get it to work for them. But most users are pretty savvy and pretty hateful toward gameable rankings (see Foursquare). they’ll get the select few young (probably white, middle class) competitive men and those idiots will in turn chase away the normals. Welcome to the Silicon Valley clusterfuck, MIT. You just pumped more air into the bubble.

Shit’s annoying.

One of these days there will be an Internet for the rest of us.




RocketClub-Logo-300x300 2 comments

Raised with Jackals

Growing up in the 70s and 80s was a precarious endeavor. Instead of 1 or 2 savants-du-torture, we GenXers collectively bullied each other on a constant basis. Sure, some kids stood out but mostly we were a rotting pile of equal-opportunity menaces. Even the shiest, weakest kids spit out epithets on the regular. Boomers had manners; Xers had mayhem.

Any tiny family nickname, minor incident, or nondescript event was fuel for the merciless machine. Any. Little. Thing. was fair game. Your mother called you “Honey” at pick-up? You were a cloying version of “Sweetums” or “HoneyCheeks” for months. Some kid landed a good punch on your brother at baseball practice? Your family’s honor would be the next day’s lunchroom fodder. Any sign of weakness was also documented and replayed.

cartoon drawing of little girl at a school desk with vomit on the floor. The vomit contains in-tact peach slices.

Moving to the country, gonna eat me lots of peaches (not)

One day in 2nd grade I felt sick, but the nuns insisted I finish my school lunch. I swallowed whole the canned peach slices in order to clear my plate. Less than an hour later, that lunch and those perfectly preserved peach pieces came out in a violent stream of vomit all over the classroom floor. The very LAST DAY of my SENIOR year, when the cafeteria served those peaches again (they were a recurring nightmare), my classmates reminded me to chew them thoroughly.

My school community was dinky, of course, but the never-live-anything-down culture stuck with me. In my (mostly analog) life, my secrets had not only been exploited by classmates but also by my mother, who would quickly circulate anything I said to my friends’ mothers, which in turn would came right back to me. (My father and brothers were absent in various ways; Confiding in the walls would’ve been more productive). Strict self-censorship was my only chance at survival and escape. What started out as a defense finished up as a way of life.

The Silent Hack

When I started blogging in 2004, mommy blogs were in full swing and I couldn’t understand the pull. How could these mothers expose their children in this way? How could they put themselves at risk of ridicule or compromise their personal safety?

Fast forward more than a decade later to today. The money’s gone out of it, so mommy bloggers are throwing out excuses for shuttering their sites: “I don’t have time;” “I want to go into consulting;” “The kids are older and need more privacy.” This last bit of reasoning irks me. Babies and toddlers don’t deserve privacy? Am I annoyed the mothers never thought of this? Or, really, am I still angry with the people and betrayals of my youth? This last bit probably holds the most truth.

But there is another reason for my ire about mommy bloggers, especially ones that quit for “privacy” reasons: I’m miffed/jealous they had the eggs (or blissful ignorance) to blog so personally and publicly, and I’m livid/disappointed they came back to feeling my same wariness of online soul-baring. Their changing ways make me question everything about being a writer or a blogger. I wonder if having Internet access is worth the effort. How does an essay writer (because this is what blog posts tend to be) live a sometimes-on life? Which parts are allowed onstage? Are we supposed to write, live, love with abandon?

How to raise a writer in 2 easy steps: dress your child in black. tell her she can't talk about it.

According to my first fiction workshop teacher James Rahn of Rittenhouse Writers Group


When friends’ Facebook pages have post after post of

  • wonderful vacation pics
  • fierce workouts
  • kids’ milestones
  • new cars & houses
  • parties (that I’m not invited to, natch)

my 1980s snark sets in. I think, it must be nice to get your money for nothing and your chicks for free.

We’ve talked about this ad infinitum: Rare is true struggle or distress portrayed on social media. These friends use Facebook as a photo album service. So then I ask,

How is it OK to show only one side of your life?

But that just brings me back to the other extreme and my original confusion:

How is it OK to raw-blog every post-partum depressive wave?

Nora Ephron and other back-in-the-day analog typists seemed to say everything about life while revealing almost nothing about themselves (which famous mommy blogger Dooce claims she did in her posts but it sounds a bit protest-too-much-y. She revealed more than she was conscious of at the time, and she probably knows that now).

No Justice, No Peace ––for a Writer’s Family


This sets my head spinning about “privacy” (<-nice quotes, if I do say so myself) and what it means to feel legitimate in this online culture. None of this is new, of course. In writer lore there are two (pre-Internet) sayings:

  1. “You own your story and you are allowed to write it,” and
  2. “If people in your life didn’t want to be written about, they should have behaved better.”

A bit mercenary, admittedly, but engaging the guerrilla-writer’s mind is necessary to build courage. Writing is art. Doing art is hard. One is never more vulnerable than when sharing creative efforts. It’s like hoping no-one will slice into your heart after you’ve offered it up on a serving dish. These mottos give us writers permission to release the relentless urge for words.


“Be free, no matter who it hurts” ??? Sometimes writers need cut-throat mottos to be able to simply start writing.

Thinking maybe it would help, I set out to make my own guerrilla motto: Be free, no matter who it hurts. Just reading that hurts all of my sensibilities. Some writers are indeed assholes. Just like some stand-up comedians are jerks, laying waste to their anything and everything for a laugh, some writers wreak havoc on those they write about. When Ayelet Waldman says, 10 years later, that her kids got through the “I love my husband more than my kids” row just fine, I have to wonder what her and Michael Chabon’s children would honestly say about it. Maybe they just know the deal: being the spawn of two writers comes with certain risk. We all have our sliced peaches to swallow, I suppose.

But it must be possible to balance the need to write honestly and the need to not hurt those who matter. The conspicuously false balance mommy bloggers thought they were striking is my nightmare, and the fear of vomiting all over my kids’ (and yeah, I guess my husband’s) feelings keeps me from writing the essays I need to write (and publish, let’s be honest. I journal 3 pages/day and it isn’t the same).

The Burning End

One of these days, I’ll stop hemming and hawing, and you will have something of worth to read.

But I can’t figure it all out tonight.

I guess I just gotta chew my peaches, one by sickeningly-syrupy one, and hope they don’t all come back up to haunt me. I have to believe I have a right to write, and even if I do brandish backsides I never meant to burn, I won’t end up, ever, as alone as I felt growing up.




Powered by ShareThis