In-app survey experiment

Tomáš Veselý - podpořen AI
Jun 13
3 min read

We're building a comprehensive knowledge library about product development as part of our mission. The library is for anyone looking to make better decisions — primarily decisions about product development. Whether you're an inventor, a product manager, or a Chief Product Officer, using the right research methods and experiments increases your chances of building the right things for the right audience. Today we'll introduce the in-app survey validation method.

When to Use This Experiment?

An in-app survey is a good fit when there's live traffic or an existing user base and you need to understand specific behavior at the exact moment it happens. Use it when:

A running product or website with real users already exists.
The question concerns a single, narrowly defined user step (sign-up, task completion, leaving a page) rather than the overall impression.
You need the qualitative "why" behind your analytics data.
In larger companies, when product managers are optimizing one part of the product rather than the entire customer journey.

Basic Experiment Principles

The principle is to ask a single short question at the precise moment of the relevant user behavior, while the customer experience is still fresh in the user's mind.

Define the trigger. Identify the specific event that launches the survey — for example, finishing onboarding, abandoning a cart, or first use of a new feature.
Deploy the tool. Integrate a service like Hotjar, Qualaroo, or Intercom that can respond to behavior on specific URLs.
Insert the question into the existing product. Using a tool like Hotjar, place the question inside the product. The question is short, unambiguous, and tied to the user's action ("What stopped you from completing your sign-up?") and the user experience associated with it.
Collect responses. Track the survey completion rate (response rate) and the content of open-ended answers. Roughly 100 short qualitative responses is the target. A recurring theme across answers shows what to improve.
Identify risks. In-app surveys measure only a single moment, not the entire customer journey, and they easily disrupt the customer experience if they pop up too often. The returns are self-selecting (the more vocal users respond), and small samples are indicative, not robust. A poorly timed trigger yields irrelevant data.

Real-World Experiment Example

Link to research: How Udemy Learns From Their Students (Qualaroo)

The education platform Udemy faced a traffic attribution problem: a large share of users appeared to arrive directly, even though they had actually been brought in by an ad or a friend's recommendation. Most of these users simply typed Udemy.com into their browser and became part of organic traffic. To address this, the company deployed in-app surveys that asked new visitors a single question — how they had heard about Udemy.

By surveying users, Udemy watched how the answers shifted as individual advertising channels were switched on one by one. This revealed which acquisition channel worked in which country — search performed better in some places, influencer marketing in others — allowing them to allocate their ad budget more precisely.

What Can Be Tested With This Experiment?

The main strength of in-app surveys is capturing the qualitative "why" at the exact moment a user behaves in a certain way — which makes them ideal for narrowly defined questions where a fresh, specific user experience matters.

Acquisition attribution: where a user actually came from, asked right after arrival — a repeatedly named channel that analytics labels as "direct" reveals an undervalued source (the Udemy case).
Causes of drop-off: why a user abandons a specific step (sign-up, cart) — an exit-intent trigger (exit trigger point); a clustering reason in the answers shows which obstacle to remove.
Quality of a specific feature: whether a newly deployed feature serves its purpose — ask right after it's used; a low rating plus a recurring suggestion for change gives development a concrete brief.
Onboarding clarity: where a new user gets stuck — ask after onboarding is completed (or abandoned); a repeatedly mentioned step is the place to rework.
Problem validation: whether the target need actually exists, before any development — ask about frustration at the moment a user would feel it; consistently phrased pain across answers is the confirmation.