While we talk about customer experience a lot on this show, today’s focus is going to be a little different than some of our past conversations. Today we’re going to talk about adding science to customer experience programs, and, more specifically the science of CX surveys.
To help me discuss this topic, I’d like to welcome Martha Brooke, Chief Customer Experience Analyst at Interaction Metrics.
RESOURCES
Interaction Metrics website: https://interactionmetrics.com/
Connect with Greg Kihlström on LinkedIn: https://www.linkedin.com/in/gregkihlstrom
Headed to MAICON 24 – the premier marketing and AI conference? Use our discount code AGILE150 for $150 off your registration code. Register here: http://tinyurl.com/5jpwhycv
Don’t miss a thing: get the latest episodes, sign up for our newsletter and more: https://www.theagilebrand.show
Check out The Agile Brand Guide website with articles, insights, and Martechipedia, the wiki for marketing technology: https://www.agilebrandguide.com
The Agile Brand podcast is brought to you by TEKsystems. Learn more here: https://www.teksystems.com/versionnextnow
The Agile Brand is produced by Missing Link—a Latina-owned strategy-driven, creatively fueled production co-op. From ideation to creation, they craft human connections through intelligent, engaging and informative content. https://www.missinglink.company
Transcript
Note: This was AI-generated and only lightly edited.
Greg Kihlstrom:
While we talk about customer experience a lot on this show, today’s focus is going to be a little different than some of our past conversations. Today, we’re going to talk about adding science to customer experience programs, and more specifically, the science of CX surveys. To help me discuss this topic, I’d like to welcome Martha Brooke, Chief Customer Experience Analyst at Interaction Metrics. Martha, welcome to the show.
Martha Brooke: Hey, Greg, I’m glad we could do this.
Greg Kihlstrom: Yeah, absolutely. I love this topic and looking forward to it. Before we dive in, though, why don’t you give a little background on yourself and what you’re currently doing?
Martha Brooke: Sure. So my job is founder and, like you said, chief analyst at Interaction Metrics. That’s interaction like what we’re doing here and interaction and metrics like the number. And what I do is I oversee the research and analysis phases for clients like Convergix and Yaskawa America and California State Bar. So my key, key role is to ensure we hold projects to the highest levels of science.
Greg Kihlstrom: Yeah, great, great. So, you know, certainly there’s a lot of measurement, there’s a lot of theory, there’s a lot of practices with customer experience and customer experience programs. But, you know, we’re here to talk a little bit about the science of it. And so, you know, our CX program scientific, are they not, you know, what, and if not, you know, what’s why, why not?
Martha Brooke: Well, Craig, that’s a very big question with a very big answer.
Greg Kihlstrom: I know, I know. I’ll give you the easy ones.
Martha Brooke: Okay. But if programs were scientific, we’d really expect that NPS and AXI scores would be quite a bit higher. In other words, what would be happening is we would all routinely be having good experiences. I guess another way to say it is that customer experiences would work. And the best metaphor that I know of is really sort of medical based. Like good seizure medications that are backed by evidence routinely result in patients having fewer seizures. So likewise, if customer experience measurement were really good, we’d routinely expect no matter where we were, that we would have good experiences. So, I guess there’s some other sort of ancillary backup to why I believe science is not being practiced as much as it could and should be. One is that, well, now we like the net promoter question and almost all of our clients want us to use it. That said, it wouldn’t be as prolific as it is. It would be used more gingerly, for lack of better words. It would be used as Bain intended it to be used, which is to summarize now how you feel about the company and not at every single touchpoint. For instance, I have an interaction with the Bank of America call rep Based on that interaction, I’m not likely or not likely to recommend Bank of America. And yet, I’m not here to pick on Bank of America. Routinely, companies are asking it at every touchpoint, kind of willy-nilly. And it’s just not, it’s not the way NPS was intended. And it’s really not a scientific approach. And also, And again, big topic, big, big answer. But I would say that companies tell me all the time they have very low response rates and they tend to hear from very, very particular kinds of customers. Maybe those with more time on their hands. At a conference I spoke at recently, a company was complaining it’s only older customers we hear from, but that’s not the entirety of their customer base. And so, Good science is representative response. So if you’re only hearing from a certain kind of customer, well, then you’re not really getting, you know, the full ante of who, you know, who your customers are. So that’s just a little bit about why I believe customer experience programs are not held in general. I mean, ours are, but in general are not held to the highest levels of science.
Greg Kihlstrom: Yeah, yeah. And so, given that, I mean, what are a few things that could be done, I mean, to make those, to make programs more scientific in their approach?
Martha Brooke: Well, the one thing that I talk about all the time is that companies would work very, very hard to remove leading constructs from their surveys, right? So a leading construct is one that directs the customer toward an answer you want to hear. So how satisfied were you with X, Y, Z? Well, that assumes the customer was somewhat satisfied, right? So that’s a problem. NPS, I would argue, has bias, like how likely are you to recommend it? It does assume the customer is somewhat likely to recommend. But in any event, and yet, again, we use NPS because it’s a good benchmarking question when it’s used properly. So there’d really be a team approach to scour surveys for anything that’s leading customers toward what you want to hear. Customers have priorities. In other words, it’s not all equal. And the goal of any customer experience program and surveys in particular is to really come up with an accurate measurement of customer experience. So if every question is of equal importance, then you’re not really capturing the nature of the customer experience. Right? If some things are more important than others, you have to include that weighting factor. So that weighting factor can be based on asking customers to rate, like, what is most important in this experience, and then using that as the weighting factor. Or we sometimes use correlation analysis to determine a weighting factor. But that’s a really important aspect of survey design and survey calculations. And then I would edit surveys for, you know, just the bundle of usability flaws that lead to gibberish data. So, you know, that’s all there, you know, now, now you’ve got me on a really a topic that I could go on for hours. But, you know, examples are like double-barreled questions. That’s where you ask two things at the same time. So was your server efficient and courteous? Well, what, you know, what are you asking?
Greg Kihlstrom: One if you’re only one of them. Yeah.
Martha Brooke: Right. So, and often those are at odds with each other. So you get information, but it’s gibberish information. So that, you know, the customer is just like, I don’t know, eeny, meeny, miny, moe. Or insufficient answer options. I think about this all the time because I’m a huge Amazon user. It’s just the easiest way in the world to buy stuff. But then because I’m a huge Amazon user, I’m a huge Amazon returner. And so the list of options. is never, it doesn’t include why it is. It’s like, I didn’t like it. They don’t include that. It’s like the website description was wrong. I almost always pick that. Like, okay, I guess the website description was wrong. That’s the closest thing to, I didn’t like it. But whenever there are insufficient answer options, you’re going to get gibberish information, right? Or one of my favorites is not allowing for anonymity. Because if you don’t allow for the option for anonymity, you’re going to omit a whole group of respondents. That can be as many as 40% of respondents. If you’re going to name me, which now that seems like now you’re going to hassle me if I give you a low score, I’m not going to take your survey. But their data is as important as those who name themselves. I’d say possibly more important. One that we already talked about is using NPS when it just doesn’t make sense, when the rating scales are off or there’s no zero. So we see this in reviews all the time where customers will write, well, the choice was one star, two star, three star, up to five stars. Really, if you’d given me zero stars, that’s what I would have given. You know, zero is, I mean, one assumes that you’re somewhat satisfied in a sense, right? So really the better scale is zero to 10 or is, you know, I would say internal language. We see gibberish from that all the time. Like when we review a customer’s surveys and they ask, well, what do you see? Do you think this is ready to go? And, you know, because we do free audits of surveys and we’ll say like, you know what, you’re in a good place. You don’t really need us. We’re happy to say that. Or, well, actually, this is not very scientific. Here’s some things you want to consider. So, in any event, when companies submit their surveys, we’ll often see all this kind of internal language questions about design, white space balancing, things that you can’t expect customers to know. So they’ll just kind of eeny, meeny, miny, moe. So those are some examples of ways that companies are collecting data, but not all data is good data. So it’s sort of gibberish data. And I just, you know, you just hope they’re not making business decisions based on that.
Greg Kihlstrom: I’m sure there’s lots of different causes of this, but a few things. As a consumer myself, it would seem that it’s actually very easy to send surveys. Not as easy to construct well-constructed surveys, but it seems pretty simple to put one together in the scheme of things. Well, you know, we we want some information like your your example about the white space or whatever. So some designer somewhere on a website is like, you know, I want to know this answer to this very specific question, not very scientific, not very customer friendly, let’s say to, you know, to ask such a kind of a niche question, but, you know, is, is some of this just because the tools are so easy to use and there’s no, there, there doesn’t seem to be other mechanisms or, you know, why are, why are we getting so many of these surveys? And yet, so the quality is so low, I guess.
Martha Brooke: Because anybody can buy Photoshop. Yeah. It doesn’t make everybody a designer.
Greg Kihlstrom: Right, right.
Martha Brooke: Right. I think that’s the sort of obvious, maybe it’s even a facile answer. I think the deeper issue could be sort of a lack of awareness of science and a lack of awareness of what good data is. And maybe, maybe it’s become just a task. Everybody’s like, task? I did it. Check, check, check the box. And yet really collecting, there’s, it could be nothing, really nothing for any company that’s more important than customer listening. I mean, honestly, right? What could be more important than that? And yet maybe there’s kind of a company centricity. where they don’t really want to know like that’s sort of a psychological thing like maybe they don’t really want to know in some cases they just want to check the box and I do feel like in some cases it’s like the
Greg Kihlstrom: the job of CX is to send surveys. I would say like the seasoned CX professionals out there that know what they’re doing, their job is certainly not just that, but there are literally people in companies that that’s their job is CX and it’s to send surveys. And I think to your point, if it is, it requires some more education and some kind of sense. And I know we’re kind of relegating our conversation to surveys. If we open this up to leading lagging indicators, all that kind of stuff, then it becomes a very, probably even more unwieldy conversation to talk about how those things tie into each other. But if your only tool is a survey, I guess you’re going to use it for everything, right? Whether it fits or not, right?
Martha Brooke: Right. Well, there’s that. And I guess just kind of picking up on what you’re saying, the discipline of CX has many methods at its disposal. Surveys simply happen to be the least expensive of those methods. But yeah, there are all kinds of methods, like we do customer service evaluations at statistically valid levels. There are customer interviews. Those can also be done at statistically valid levels. So there are other methods outside of surveys that also should be held to the standards of science. And so, you know, I think that maybe, Greg, it’s possible the discipline of customer experience is so new that it really hasn’t absorbed the science message yet. Like if when medicine first came on board, thousands of years ago. I can’t say that it was very scientific, wasn’t it? Like bloodletting or leeches, you know. So sometimes a discipline comes on board and it takes a while for it to really catch up to science, which is important. I mean, what we determined in the Renaissance was it really is the best way to understand the world. And so by extension, it’s the best way to understand customer experiences. Yeah, right. Like it’s it’s better than conjecture and belief. And it really is the best way we know of, to understand what is what the nature of the world is.
Greg Kihlstrom: Yeah, I like that. So moving ahead a little bit, we got to talk about AI, so we’re going to. So how does AI factor into this? We’re talking about surveys and CX. Is it Is it going to help us? Is it going to hinder us? Where do you see that?
Martha Brooke: First of all, I love ChatGPT. We actually have our own ChachiBT engine, so fully bought into AI. Actually, it really should be called large, large language models, right, right now, because intelligence is not where it’s at. So it does a lot of great things. So that’s, that’s the first thing I’d like to say, but it doesn’t write surveys. So do not use it for that. Really don’t use it for that. It’s, it’s a large language model. So it’s just combing what other surveys are doing. And most surveys are not being held to a scientific standard. Okay, so don’t use it for writing surveys. Now, another very common thing that people do with AI is they use it for analysis. And with quantitative information, it just doesn’t work. Because the kind of analysis that you want to do for quant is what we call segmentation analysis. That’s where you’re comparing different populations sort of side by side. Say you have OEMs and distributors and end users, you want to be able to compare each of those populations and how they’re responding to each survey question. And so that’s a little too complex. for any kind of analysis that AI can do right now. Now, another way that companies use AI is for their text analysis. That can be great, but Again, remember, right now, AI is not general intelligence. It’s a large language model, which means it really has to be trained to be effective. So it’s not, yeah, you can just throw, and we do, we experiment with mostly chat GPT, but some of the other large language models too. Yeah, you can put the text into one of those engines, But in general, especially for B2B, it’s just not pulling out the nuances that you need. And sometimes it’s, you know, hallucinogenic. It’s coming up with stuff that sounds really great. Yeah. You know, it’s like we did it recently and I was like, oh, wow, this is amazing. And I was like, wait a minute, let’s like really read this. And it was wrong.
Greg Kihlstrom: Yeah.
Martha Brooke: You know, it looked really good. Like these were good sentences and really And I was just like, oh, done and done. No, no, no, no, no, no. So it can be quite misleading. That said, AI, very useful if it’s trained. So you need researchers working side by side with AI. It’s easily confused, even with sentiment, which is the easiest part of text. So sentiment is like, are they happy? Are they sad? Like, how do they feel? But a sentence like, love your company in particular, but your customer service is a real hardship. Well, many AI solutions are not going to rate that for what it is. That was a mixed comment. That was not, they’re going to say, go with that first phrase. Oh, they’re a positive. Love your company. That’s not actually what they said. And that’s very simple. Sentiment is very, very easy. What’s more complex is finding the meaning the emergent themes, what customers are actually talking about. And so that’s much more difficult than sentiment. And so even sentiment has its problems. But OK, let’s put that aside. And what’s more important almost is, what are customers talking about? And how are they thinking? What are the topics? And so LLMs can be a useful kind of side-by-side with researchers, but they really do need to be trained. I hope that wasn’t too shaggy, shaggy, whatever they say, shaggy dog and answer shaggy.
Greg Kihlstrom: I mean, we’re, yeah, it’s still, we’re, I feel like we’re, I mean, AI has been around for decades, but I feel like we’re in early days of this, this wave of, of really using it in these ways. So there’s a lot to. There’s a lot to do. I mean, there’s a lot of opportunity, but there’s a lot to also kind of unpack and really understand. And I think one thing that you touched on was just how it can be really helpful to use AI, but it needs humans to make it better. Just like we can use AI to make us better. It goes both ways. I think that’s where maybe someday to the general AI point, maybe someday it won’t be that, but for the time being, it can be really powerful when either it’s a first draft or a second draft or something like that, but it’s part of the process, not just some kind of end goal.
Martha Brooke: Right. I mean, so the social science way of dealing with text is to do what we call coding the data. And so that’s where you have a team of researchers, not one researcher, because you need a team to come up with, you know, this is verifiable, replicable results. And you go through and tag comments within set protocols. And then you can sometimes compare that against large language models and use that to train large language models. But there really are techniques for unpacking what’s in a conversation or in a body of text. And these are important proven techniques.
Greg Kihlstrom: Yeah. Yeah, absolutely. Well, Martha, thanks so much for joining. One last question before we wrap up here, you’ve given a lot of great advice and insights already, but for those that are listening here, know that they need to inject a little more science into their CX programs. What’s one piece of advice to, where could they start?
Martha Brooke: I think they could take a day and just study the principles of science that so random selection controlled experiments and and then see what of that applies to their survey. You know, I think that would be a day very, very well spent. And, you know, of course, feel free to reach out to me on LinkedIn or website or however you like to chat, because I’m truly always open to that conversation about how do you get evidence-based quality data-driven information about the customer experience.
