We are recording live at Qualtrics X4 in Salt Lake City and seeing and hearing all about how to create and enable amazing customer experiences.
How long would you like to wait to get customer insights? Weeks or minutes? Today we’re going to talk about how increasing the speed to insights can be a game changer for brands, and how synthetic feedback allows rapid testing and on-demand marketing intelligence..
To help me discuss this topic, I’d like to welcome Ali Henriques, Global Director of Edge at Qualtrics.
Resources
Qualtrics: https://www.qualtrics.com
Connect with Greg on LinkedIn: https://www.linkedin.com/in/gregkihlstrom
Listen to The Agile Brand without the ads. Learn more here: https://bit.ly/3ymf7hd
Don’t miss a thing: get the latest episodes, sign up for our newsletter and more: https://www.theagilebrand.show
Check out The Agile Brand Guide website with articles, insights, and Martechipedia, the wiki for marketing technology: https://www.agilebrandguide.com
The Agile Brand podcast is brought to you by TEKsystems. Learn more here: https://www.teksystems.com/versionnextnow
The Agile Brand is produced by Missing Link—a Latina-owned strategy-driven, creatively fueled production co-op. From ideation to creation, they craft human connections through intelligent, engaging and informative content. https://www.missinglink.company
Transcript
Note: This was AI-generated and only lightly edited
Greg Kihlstrom:
We are recording live at Qualtrics X4 in Salt Lake City and seeing and hearing all about how to create and enable amazing customer and employee experiences. How long would you like to wait to get customer insights? Weeks, days, or minutes? Today we’re gonna talk about how increasing the speed to insights can be a game changer for brands and how synthetic feedback allows rapid testing and on-demand marketing intelligence. To help me discuss this topic, I’d like to welcome Ali Henriques, Global Director of Edge at Qualtrics. Ali, welcome back to the show.
Ali Henriques: It is so great to be here and actually meet in person.
Greg Kihlstrom: I know, I know. Yeah, you were on the show last fall, but always great to be able to meet in person and talk about this stuff. And as we’ll quickly get into it, things are moving quick. So this is a good topic to talk about.
Ali Henriques: Sure are. I’m happy to.
Greg Kihlstrom: Yeah. So for those that didn’t catch the show you were on before, why don’t you just give a little background on yourself and your role at Qualtrics?
Ali Henriques: Absolutely. So again, Ali Henriques, I have the pleasure of running strategy and services for Qualtrics Edge. Our kind of history, if you will, and I’ll keep it brief. has always been connecting clients to audiences that they don’t have access to. So our CX clients have access to their customer databases. EX clients have, of course, access to their employee rosters and such. Strategy and research clients don’t often have access to their competitors’ customers, their prospects, things like that. So that was kind of our history and upbringing. And so what we’re doing now as Qualtrics Edge is as you just mentioned, putting this power back into the researchers’ hands to get to answers faster with a couple of new solutions. So it’s been kind of fun. We are still doing the research projects. I call ourselves kind of an in-house market research agency, so a full-service strategy research firm running projects end-to-end for our clients, writing questionnaires and fielding them and analyzing them. We’re just talking about different ways to cut out steps of that process with AI.
Greg Kihlstrom: Yeah, yeah, love it. So yeah, let’s dive in here. And so the last time we spoke, we talked about synthetic personas and research and some of the things that you just referred to. But I feel like a lot has happened since last fall. At X4 here, where we are today, Qualtrics announced Edge Instant Insights and Edge Audiences, allowing teams to transform weeks of research into minutes, leveraging synthetic feedback for rapid testing, and gain on-demand marketing intelligence. That was a lot there, so can you talk, provide a little, maybe, let’s just take a step back, because, you know, those that aren’t as familiar with that term, synthetic research, maybe start there and then talk about the products that use it.
Ali Henriques: Absolutely, happy to. So I think our new favorite way to describe synthetic is AI modeled responses. Because that’s really a theme of everything we’re talking about here. But that’s true. That’s the nature of how we’re getting these. What they’re modeled off of is what matters. So I’ll just bear with me for a second as they take you through kind of what’s happened honestly, since last fall, we’ve started to categorize the types of synthetic offerings that are in the market. First, you have what I call wrapper, models, so it’s a really sleek interface on top of something like ChatGPT, where what it’s doing is effectively building the prompts for you to emulate certain populations. It’s great, it has great application, but it’s only referencing publicly available information. Another approach is the RAG method, which is really machine learning on steroids, where we’ve collected, say, 200 human responses, but I want to turn those into 400. And so I’m going to consider everything I know of the patterns and behaviors within these 200 responses, make it 400. The third approach is truly custom foundational model. That’s what we’re building and it will include the other two kind of elements of synthetic. But the idea and the differentiation here is that when you’re building a truly foundational model, You have to have access to robust data sources. And so we have the benefit and luxury of decades of data collection across all of these different pillars. So that goes through a rigorous aggregation and anonymization process across our experience data plus the research data that we’ve conducted. So by design anonymous, we don’t have any way to recontact these humans, but We have millions and millions and millions of responses, even just from the last couple of years, right, I’m talking like 30 million, that have taken all sorts of different studies. So those behaviors, preferences, those are things like, how do you like this new product, right? What do you think of this ad? You know, if you weren’t going to dine with us today, where else would you have dined? all of these types of, again, consumer behaviors, that’s what’s feeding our model in addition to what’s publicly available, right? So we’ve got the weather, we’ve got the news, right? We’ve got just what’s on the web. And then we’re also feeding it with training data. So keeping a pulse, daily pulse on the markets. to make sure that the data is constantly refreshed, not just from what I’ll call standardized training data, but also a continuous influx of all of the experience data and research that we’re collecting every day. So that’s kind of a little bit about synthetic. So that gives us an idea of what repository we’re considering when we say, I need 500 prospective cruisers, because that’s what I needed back in the day when I was an analyst, right? I need people who have never been on a cruise, but might be open to it. Synthetic would be a perfect application for saying, I know generally what these people look like, what age range they are, what markets they come from, what other types of travel they’re interested in. What do we want to ask them, right? So that’s synthetic in a nutshell. And I’ve also kind of briefly described edge audiences. So I’m explaining why Qualtrics is in a unique position to deliver on this synthetic demand. And where we’re at from the last time we spoke, Greg, it’s here. You can go to the XM Park and demo it today. So it’s super cool. And yeah, so I’ll pause on audiences, or shall I move on to instant insights?
Greg Kihlstrom: Well, yeah, I mean, just one thing on that. And so just kind of underscore what you’re saying. I mean, I think someone might be tempted to go to chat GPT and like type in, like, you know, do the thing like you are a blah, blah, blah, and try to emulate. And to your point, that can be somewhat effective. But I think To me, the power of this is in the combination of all of those things. And also, Qualtrics, obviously you have a software platform, but as a research organization too, I think the marriage of those two things brings a little more rigor to it, would you say?
Ali Henriques: Absolutely, yeah. And there are other agencies out there developing their own synthetic. We’re all taking slightly different approaches. And we’re starting with more of an audience angle, pun intended or not. And that meaning like we’re targeting U.S. general population consumers versus going use case, right? So we could say, look, we’re gonna become the experts in replicating responses for concept tests or ad tests or something like that. And so we’re taking a slightly different approach.
Greg Kihlstrom: Yeah, yeah, makes sense. So yeah, why don’t you talk about Instant Insights?
Ali Henriques: Yes, of course. Instant Insights is an industry-specific kind of market intelligence platform. So what it does is it combines survey research, so a syndicated study effectively, I’ll pick on restaurants because that’s the one that’s been running the longest, So we’ve got a lot of what you’d expect, brand perceptions, where did you dine last, where else would you consider, some kind of competitive benchmarking type of components in there. But what we’re doing is we’re augmenting that with about five or six other data sources. So we’re pulling in search trends that gives us a sense of rising and trailing kind of trends and topics. That way we can see promotional offers. you know, what else is happening in the market that might be influencing the consumer behavior. We also get that from news and trends. So topics, anything that might even be popular or viral on Instagram, right, will make its way here. So we’re pulling in what’s happening in the moment, in addition to transaction data. And then what really makes this cool is behavioral data. And what that means is we’ve got in-person kind of foot traffic. I heard this morning, absolutely loved the line, people vote with their feet, right? And so that’s part of what we’re tracking here. We’ve got physical in-person location tracking as well as web and digital. And so part of the story you can tell here is, If I’m not choosing Chipotle, where am I going and why, and how do I spend my free time? Who is this person outside of the context of the transactions and the experiences I have with them? That’s often really hard for brands to stitch together. They know their customers very well. They know when Greg came in the restaurant. They know what he ordered. They know when he left. No idea, really, who you are outside of that. Instant Insights is here to kind of bring that to life, but also give you more of that kind of competitive context. So we start with these concentric circles of my category, my brand, my competitive set would kind of be that next ring. And then outside of that, how do you move about this world, right? And where else are you spending your time and money?
Greg Kihlstrom: Well, I think the other thing there is things move very quickly. I mean, even my behavior could change rather quickly as well. So even taking a traditional like predictive analytics or taking survey results from two months ago, because how many companies can run surveys? constantly and stuff like that. So, you know, what I’m hearing there is, yes, there’s that legacy stuff as well, but it’s also augmented by things that are up to the minute because for better or worse, these are the times we live in that anything could change sort of on a dime, right? So it’s kind of, it’s mixing the known and the now, right?
Ali Henriques: That is exactly right. Try as we may, my past behavior does not necessarily tell you what I’m gonna do next.
Greg Kihlstrom: Right. I want to talk a little bit about a use case with synthetic data. One of your sessions at X4 is a presentation with booking.com and how they’re using synthetic data. So I’d love to hear just a little bit, those that were here, I’m sure we’ll check it out. But can you talk maybe at a high level about the partnership first?
Ali Henriques: We actually just did that this morning. It was so fun. And what I love about this, we’re in this kind of pilot phase and we’re all going into it eyes wide open. And so the way that Elina, the client, represents the work is this is not an A-B test. We did not run a human set of data and want to see how well the synthetic did against it. Instead, we wanna understand the journey and the process. And so while she did have human responses to be able to compare, we certainly showed that it wasn’t the spirit of the experiment. And so I think my favorite, and this was all her work, and it was done so beautifully. And I just kind of interjected with research on research and some of what we’re seeing from the market. Her biggest kind of takeaway was that The researcher is still very, very important, right? The way that we construct questions and the way that we think about presenting what we’re curious about to human or AI responses is really important. That matters. So for example, she took a question of Which of the following types of trips did you take in 2024, Greg?” And what she found is that there was actually recency bias in the humans. So the number one answer, because the study is conducted in January, what did you do in December? Oh, you visited friends and family, right? Well, AI said, no, I actually think most people went to the beach. Beach trips are the most popular type of travel. And so she found herself questioning, have we been lying to ourselves for 10 years thinking that friends and family are the top travel destination? So it’s just, it’s so funny how this has just been, a process of really scrutinizing our own work as humans and how we ask questions and how I take one simple question and I think each of us would answer it differently. And so, you know, progressing that same kind of thought, her next big finding was that synthetic responses are really, really good for attitudinal, psychographic type of work. That makes sense. Because a lot of who we are doesn’t change, our behaviors will, right? And there’s absolutely an element of prediction there. The other finding around, oh, this one was fun. It was such a mouthful for her to get out. We did not trust humans’ reflection of how they’ve used AI in travel planning. AI actually told us how humans use AI for travel planning better than the humans did. And so, and it’s in here we’ve got, we have evidence, we have operational data to tell us that we’re using AI to help with itinerary planning and the sightseeing and what’s not to miss in a certain place. But the humans reflected maybe a bit more, again, based on recency, restaurant recommendations and things like that. So she said, here I have external evidence to suggest that the AI responses were more accurate than the human responses. And so just fascinating how we’re really forcing ourselves as researchers to think a lot more critically about the questions that we write and the implications of who’s on the other end. At the end of the day, we’re lazy, right? We’re going to skim the question, think we get the spirit of it, and choose the couple of things that are top of mind for us. AI is not doing that. It’s thinking very, very critically and reflecting on that. And so you see that come through in a couple for other examples, like solo travel or traveling alone, right? And it’s fascinating. So it was really, really great session.
Greg Kihlstrom: Yeah, I mean, and it’s an interesting thing of, you know, certainly there’s a lot of talk about bias in AI, and that’s, you know, something, you know, companies and people should take very seriously, but there’s not as much talk about the bias in humans. You know, I find it, what you’re saying, very interesting in that you know, if AI has the opportunity, I mean, sure, there’s things, there’s ethical things that need to be taken care of, but it also can help us solve some of those biases that, you know, nefarious or not, like, there’s just, there’s bias, whether it’s recency, anchor, there’s 100 cognitive biases, right? So it’s like, How can AI actually help us get past those? Because my assumption is, at the end of the day, that actually helps the customer. I mean, if that’s really what it’s about, it’s kind of helping us think past the holiday family visit that we did and the thing that really matters the rest of the year or something.
Ali Henriques: Exactly.
Greg Kihlstrom: For those, and Booking.com, I mean, what I know about them is, you know, they’re very, they’ve been very forward as far as testing and all of that stuff as well. So definitely sounds like a great partnership and a great opportunity. For those that are, want to do something like that, like what’s the mindset or where, you know, how can an organization kind of get ready to do some of this work?
Ali Henriques: It’s such a great question and a couple of folks approached us afterwards and said, how do I guide my stakeholders, right? And I’ve heard both extremes of this. I actually had them do a quick poll before we got started. I wanted to know who the haters were in the room, right? Who do I need to convince that this is real and it will work for you? And then I had the opposite challenge. A woman approached me and said, I need to rein my people in because we’ve all just gotten enterprise access to chat GPT and they all think they’re researchers now, right? And so it’s a crazy spectrum here. But I think we get the question a lot, right? How do I get started? What types of studies should I pilot? We talk a lot about early innovation testing, so idea screening, feature optimization, promotional offers, things like this. Those are great project types that are really purpose-built for synthetic. But even back to our last topic, Greg, you think about survey design, semantic and cognitive testing. Like, that is no brainer. That is available to all of us today to just have these tools help us improve the questions that we’re intending to ask of humans and make our response options more exhaustive. So there are just really low risk, tangible things that we could try tomorrow. But in terms of more fully synthetic responses and considering a blend or full replacement, I cannot answer that for most clients because it really depends on the weight of their decision. And, you know, I said in the room, if you’re making a $5 million marketing decision based off of the output of this work, I would probably get some humans, right? So let’s be careful here because it is so new. But what I find most of, and I’ll speak for the researchers out there, they want a look under the hood and they want to compare side by side as real time as possible. How did your synthetic compare to these humans? Accepting that the humans are flawed as they are, right? And so that’s what we’re finding most often. I actually, I’m coming to you just now from a session with Google Labs and they piloted our synthetic and we ran a parallel with humans so that we had, something to compare it to, right, and to be able to assess what our comfort levels were with different question types and topics. And so I’d recommend experiments like that to just get started to convince yourself that this will work for either the use case or the audience or, you know, the project that you’re thinking about.
Greg Kihlstrom: Yeah, yeah. So let’s maybe look out a couple years. I know things move very, even a couple months, but like a couple of years out, you know, we’re at the beginning stages, as you’re saying. Things move quickly. I mean, if we would have guessed three years ago what Chat GPT would have been doing, you know, it would have been hard to predict. Where does Synthetic kind of go from here? You know, what’s kind of next on the horizon there?
Ali Henriques: You want the alley answer? or the Qualtrics, no, I’m kidding.
Greg Kihlstrom: It’s not very different.
Ali Henriques: I think what’s different is the kind of timescale, right? And it’s not our ability to develop the technology. I actually believe what will, I’ll call it, hold us back most is the client receptivity, their openness, their willingness, and then their own organization’s level of kind of on that spectrum of innovation and progression. So where I see this going, and I was just telling Google just before this, I imagine a world where, yes, and I realize I’m a researcher and I’m quite arguably putting myself out of a job, right? I imagine a world where we just ask a question and get an answer. These models have gotten so smart, so real time, that it’s taking in the weather, the traffic, every signal we can possibly feed it about what’s happening around us right now in this moment. And it’s guiding us, right, in whatever we can think to ask. So what I mean by that is today, the researcher needs 1000 rows of data to feel confident in the answer that they’re gonna give their stakeholders, right? We’re holding on to that a lot. And I understand, right, within the next few years, We’ll move away from these thousand records and we’ll accept one question answer, right? And that’s where we’ll get talk about rapid insight testing. That’s in everybody’s hands. That power has to be in the marketer’s hands, the operator’s hands, the CX leader’s hands, not just the researchers. And there will still be projects that only the researcher can execute, right? And that will always be true. And so I joke, but I do maintain that job security is still there. Because even with synthetic, think about how much of this time we’ve spent talking about question design, right?
Greg Kihlstrom: I mean, that’s what I’m thinking there is that, you know, doesn’t that then elevate the role of the person asking the question? You know, some of the busy work and all that gets taken care of, or even some of the details, but it would seem to elevate the researcher’s ability to ask questions, right?
Ali Henriques: That’s the dream, yeah, exactly right. And then we’re freed up to focus on the things that we actually really, really are genuinely passionate about and not coding the open-ended responses and tabulating the data.
Greg Kihlstrom: Yeah, yeah, totally. Well, as we wrap up here, two questions for you. First, I know we’re about halfway through the conference here, so maybe too soon to ask, but what’s been a highlight for you so far?
Ali Henriques: What a good question. I love main stage. There is just so much energy and buzz and excitement. So I loved today’s and was particularly, I found myself literally laughing out loud at Ken Hughes. Like, who’s this guy? Where did he come from? And what got me the most was, now I’m dating myself, but when you click save in a document, his child asking if that was a microwave for a fridge or something. I just, I’m dead. The kids holding up the cassette tapes. So I just, I love this kind of perspective on our world, right, that we just eat, sleep, and breathe every day. And then, you know, the breakouts are wonderful. I’ve had the pleasure of running a couple of them earlier today. the one with booking.com, of course. And so it was at the same time as the Google Lab session, so I missed that. So I’m turning my highlights into lowlights. We can’t be in multiple places at once, right?
Greg Kihlstrom: I feel like missing out a little, but yeah.
Ali Henriques: So no, it’s great. I think then to sum it up, the energy and the buzz that you just get from being around folks who are all tackling the same kind of business challenges.
Greg Kihlstrom: Yeah, love it. Well, last question. I know since you’ve been on the show, I’ve asked this before, but we’ll see how close the answers are. That’s a research project in and of itself. Your own validation. I know, I know, right. So what do you do to stay agile in your role, and how do you find a way to do it consistently?
Ali Henriques: You know I’m trying to think about what I might have said last time. You’ve stumped me, Greg. I think designed to actually be different, right? I’ve taken to podcasts a lot more. So I have a long commute. I love to first make sure I get news and just kind of really timely stuff out of the way. And then I spend most of the rest of my time just listening to very like business related type of content. in particular AI. I may not understand the importance or significance of all the chips and the storage and the who’s who and the who’s what in this category, but the more I listen, I think it’s more important to just understand a lot of where some of these visionaries see this going. because it helps me more confidently answer for you how quickly we’re gonna go from question to answer. So lots of just content consumption. But I think this one, I’m confident I didn’t tell you this last time. I have three kiddos and I absolutely love seeing the world from their perspectives and vantage points. And so I think a lot lately about just the way that they take information and they take the world in and I try to kind of harness that and bring it to my day-to-day too. And in some cases that might be just approaching things totally eyes wide open, right? Genuinely excited to learn about something even if it may seem super simple and non-consequential to my day-to-day. So yeah, I think a couple of those things.
