Data + the Modern Pandemic:
Making data-driven business decisions despite *gestures around broadly*
“I hope this email finds you well in these unprecedented times”
We’ve all received countless emails with this same hook. It may have been a nice sentiment in the first couple weeks, but now it’s overused and cliche. It’s actually remarkable how fast that sentiment has become played out; in a few ̶v̶e̶r̶y̶ ̶l̶o̶n̶g̶ ̶a̶n̶d̶ ̶m̶o̶n̶o̶t̶o̶n̶o̶u̶s̶ short weeks a message that started out heartfelt now feels thoughtless. It’s a symptom of us all fumbling around in a new world, grabbing on to anything that shows promise, and often working it until all merit has been drained dry. So how do we find our footing and navigate through these new waters?
This isn’t just a problem for marketing and advertising; everyone is learning to navigate a new landscape, oftentimes feeling like they’re blindfolded. If you’re lucky enough to have a data/analytics/BI/big-brain department, then they’re often the ones that people turn to for guidance and help making decisions. Unfortunately, a lot of the value we provide comes from information extracted from prior data.
The problem with living through an unprecedented event is that it’s just that: unprecedented. Sure, there have been recessions in the recent past, but the last time there was an event of this scale the issues were along the lines of “Jebediah can’t wash his hands because there’s no indoor plumbing on the farm” rather than “Susanne from accounting can’t figure out how to change the background in her zoom meeting from when her daughter switched it to minecraft”. So when my boss messages me asking for the latest growth (lol) projections so she can determine if marketing spending should be increased, what am I to say? Here are the tips that I’ve found in order to squeeze as much value out of your data as possible during a time where no one is really certain of anything.
Step 1: Turn off your forecasting models
Six months ago you might have had a machine learning model that predicts whether a customer will buy your product or not with an F1 score of 0.95, a recommender system that increased probability of sale by 6%, and a machine learning-based tool that automatically identifies and emails the best targets for upselling your product’s deluxe tier. Well, not anymore you don’t (unless your model always predicts the worst case, then it might be accurate). Any sort of machine intelligence-based model will fall flat on its face when circumstances are changed so drastically from what it was trained on. Not only will these models start to feed you inaccurate recommendations, but machine learning systems are notoriously bad for not communicating uncertainty in their results. Turn them off; they’re probably doing more harm than good at this point. In the future, when economies stabilize, you may be able to turn them back on, but they will likely need major modifications.
Now, I’m not saying it’s impossible to build an accurate machine learning model in volatile times such as these, but that will require a lot of data, expertise, and, most importantly, love (read: days or weeks of data cleaning, augmenting, tuning, testing, and monitoring). For most small to medium businesses the value provided by an automated ML system is not worth the effort it will take to maintain them at this time. Besides, human empathy is more valuable now than it has ever been, so don’t feel guilty about ignoring your fancy marketing recommendation system and going with your gut. Your AI and ML tools can still be used as a guide, but your heart is what allows you to really connect with your customers or clients.
In a world that’s constantly changing, and where everyone is going through their own personal struggles, it doesn’t make sense to rely on computers to make decisions for us. Instead, it’s more important now than ever to use our data as a tool that allows us to connect to our customers on a personal level, human-to-human.
This brings me to one tool I want to pay special attention to: AB tests.
Step 2: Rethink your AB tests
Throw out all of your AB tests. The fundamental assumptions you made for any AB test you started pre-covid are no longer valid. Even subtle assumptions that would have never crossed your mind, like “there are customers out there willing to pay for my product,” are not valid. I’m not recommending that you don’t run AB tests. In fact, I think that performing many rapid AB tests is the best way to learn anything useful in this covid world, but be warned; anything you learn in such a volatile environment has a very short expiration date. For example, take the opening hook: “I hope this email finds you well in these unprecedented times.” It may have performed well on an AB test in week 1, but I would bet that the opposite is true this week. That being said, data has a way of bypassing all of your biases and assumptions if you let it, and if you listen to it it will give you answers. So, by constantly performing multiple AB tests and measuring the correct things, your data will give you insight into what it is your customers actually need from you.
While we’re on the subject of AB tests: MAKE SURE YOUR GROUPS ARE ACTUALLY RANDOM. There are a lot of different factors that will drastically affect the way that any human behaves. Everything from location and employment status all the way to political stance and whether they get their news from Facebook or TikTok can affect the way a person makes decisions. An incorrect AB test that groups people from the same city, or age group, or mobile device type will skew your results, so make sure your customers are separated into groups based on a random number that has nothing to do with the customer. I can talk for hours about AB tests, so expect a follow-up article on the subject. Until such time that I’ve written a second article, suffice it to say that every piece of data that can be collected about a customer contains more information about them than you would expect. This information can be used to put individuals into groups, and group psychology can be predictable. This brings me to my next point.
Step 3: Group customers
The current economic climate has not treated everyone fairly, not in the least. As such, there’s no one-size-fits-all approach to dealing with customers. If you sell software to Zoom, Netflix, and Cineplex, you’re going to want to know which one you’re dealing with in order to effectively sell (or not sell) to them.
I want to take a second to mention that not all data should be used when looking at customers. Machines have a tendency to learn biases, so data points like race, age, and gender should never be included in a machine learning model. In most cases these data points shouldn’t even be stored; they are just a recipe for perpetuating stereotypes. It’s also private, personal information that shouldn’t even be collected except in research settings.
If you’re smart about the data you have, and are able to look at relevant data points, you’ll be able to better serve your customers. That’s right: “serving”, not “selling to”. In many cases, if you can afford to, not selling may be your best approach right now. Offering coupons, discounts, free trials, and blog posts full of useful advice are going to help to build brand loyalty, which will be incredibly valuable in the long run. Many people have a lot of free time right now, so offering extended free trials might be a perfect way to get them using and loving your product. Alternatively, if you have customers like Shopify that are doing extremely well right now, this might be the time to sell them on the next product tier in order to handle the extra volume they’re experiencing. Reaching out to your customers like a friend is the difference between building a relationship and sounding like a car commercial about us all being in it together. Grouping your customers based on the industry they’re in, or if they’re currently open for business or not are 2 ways to ensure you’re addressing each customer appropriately and effectively.
Separating your customers by location is also an option. Regulations around COVID are literally all over the map. In some states and countries restaurants, gyms, and theatres are open, and in others they aren’t. Some places are opening up travel soon, and some places are still under lockdown. Use this information to your advantage. Offer coupons to customers in heavily affected areas, offer advice and tools to help customers reopen their business in locations where that’s appropriate, and then start capitalizing on your customers that have resumed some semblance of normalcy.
What if you find yourself in the unfortunate circumstance where you haven’t been collecting customer information? You can still let customers tell you information so that you can help them better. Offer coupons but let them fill out a survey telling you what they need and how they want to be helped. Set up a help email where they can contact you. This might not scale to a ton of customers, but the ones that reach out will feel like they’re important to you and you can build a relationship with them.
And in order to evaluate success…
Step 4: Fix your KPIs
Let’s circle back to AB tests (have I mentioned how important these are?). Whatever metric you’ve been using to evaluate AB tests or product success may not be relevant anymore. It’s incredibly important to measure the proper KPIs. Measuring conversion doesn’t make sense when many people aren’t paying for software right now. Measuring time spent in an app doesn’t make much sense either when half of your users are working and the other half are bored out of their minds re-watching Tiger King in their pajamas. Each test or feature should have a meaningful KPI associated with it. Allow me to make some suggestions:
- Daily or monthly active users. Despite all other circumstances, someone using your product is still someone using your product, and therefore is a potential future customer. This can also be broken down and measured per-feature or per-page for a more granular look.
- User stickiness. If someone keeps coming back to your app or feature, that means they enjoy it. Other alternatives include number of sessions per user per day or week, number of users with more than one session per week, or some other metric that captures repeat customers.
- Customer engagement. Again, there is no silver bullet for this one. Each product and feature is going to have to come up with a different metric that measures engagement. For a news app this might be “scrolled to bottom of page” or “clicked on a related article.” Users are still users, so these KPIs are good at capturing the essence of how users interact with your product. Be careful though, these KPIs make assumptions about what you think customers will find valuable, which may not be correct.
One of my teammates is fond of saying “track what you want to measure,” and right now one of the main things you might want to measure is “do I have customers now that will pay for my product once they’re able?” Without a crystal ball, the closest thing that’s reasonable to track is if you’re gaining customers that are finding value in your product. By defining the correct metrics you can ensure you’re on the right track, constantly delivering products that your customers will eventually pay for.
Closing Thoughts
When there’s so much uncertainty in the world people just want to be certain, and while certainty is a rare commodity, we can still find answers if we’re willing to look in new places. We may not find the answers we want, but we can still find answers that are useful. In the long run, these answers will help us deliver valuable products and drive business as fast as the economy lets us.
If you find yourself wanting to know more about your data and what it can do for you, check out flatland.AI or send me an email at logan@flatland.ai.
