Summary: Iterative user research becomes its own pilot study | Continued growth in AI companions | The government of Queensland in Australia warns against dark design patterns | Set of templates for the practical side of running a usability study | Using ChatGPT without having to register
UX Roundup for April 5, 2024. (Midjourney)
Iterative User Research Is Its Own Pilot Study
It’s always been a recommendation for any user research to start with a pilot study before launching the real research study.
The goal of pilot testing is not to test the user interface or to improve the design. Rather, the goal is to refine and validate the design of a study before it is conducted on a larger scale. This preliminary phase involves a small group of participants and serves to identify any issues with the research methodology, tools, or procedures that could impact data quality or participant experience. By conducting a pilot test, UX researchers can ensure that their study design will elicit meaningful insights, thereby minimizing the risk of encountering significant obstacles during the full-scale research.
(That said, anything you learn about the UI during a pilot study should obviously be used to improve the design. You don’t throw away data. The point is simply that we don’t expect to learn that much about our design when we focus on improving our study methodology.)
A pilot study is like taking a magnifying glass to your test plan to inspect its details and see if they hold up to the behavior of real users. Just as nobody can design the perfect user interface on the first try, nobody can design the perfect user research study on the first try. There are always some instructions that are misinterpreted or tasks that don’t represent something users want to do. (Midjourney)
I just gave you the classic advice, which I have given countless UX newbies over the years. While good advice, it’s not the full story. Suppose you apply my recommended discount usability method and conduct many rounds of testing with around 5 users in each round. In that case, you might dispense with pilot testing or at least downsize it significantly.
The cheaper each study, the less the need for pilot studies. Why? If each study is fast and cheap, it matters less if it’s perfect or if you make some errors in the methodology. Furthermore, the entire idea behind testing with 5 users per round is that you save so much time and money that you have the budget to run many more rounds of user research. You undermine this philosophy if you spend too many resources on up-front pilot testing.
Let’s say you’re an abject newbie. Maybe this is even your first usability test. In that case, I might still recommend a small pilot session with two users. Allow for a full day between the pilot sessions and your first “real” test participants, so that you have time to make the inevitable changes in your test script and other study procedures. It would be even better if you could rope a senior colleague into watching your pilot sessions and give you constructive feedback on how to improve.
In the general case, pilot testing is less needed when you conduct many rounds of quick iterative tests. Each round of testing does double duty as the pilot test for the next round. Assuming you’re planning for many rounds of testing, it doesn’t matter if the first few rounds are imperfect and don’t give you full data. Whatever you missed, you’ll pick up in later tests. It’s much more important to get started with collecting customer insights early in the design project.
In a camel race, it’s essential to get your camel out of the starting gate quickly so that he can run ahead of the other camels. The same is true for user research: early user data is immensely more valuable than data that arrives later in the project. (Midjourney)
Suppose you’re doing an expensive study, such as a measurement study (which I tend to advise against for most projects) or a card sorting exercise where you will be collecting data from a hundred users or more. In such cases, pilot testing is indispensable. You don’t want to see how 100 users sorted a bunch of cards with labels that they misinterpreted. The entire study will be wasted because of a flaw that could have been corrected by having a few users think aloud while doing a pilot sorting.
Continued Growth in AI Companions
Artificial Intelligence is infusing computers with personalities and opening up a myriad of possibilities ranging from friendship and mentorship to coaching and even romance. I recently wrote an article about how AI companions are killer app for AI, and they are the high-end of the spectrum of anthropomorphizing AI.
Justine Moore from the leading venture capital firm Andreessen Horowitz (often referred to as A16Z) posted a thread of observations to X, noting how the AI companion space continues to grow and now spans a broad spectrum of applications. She also provided an extensive list of startup companies working on different aspects of this vision.
At one end of this spectrum are AI companions designed purely for entertainment, while at the other, solutions aimed at addressing specific needs, such as aiding children in safely navigating the Internet or alleviating loneliness among the elderly, take precedence.
AI companions are expanding into a broad range of applications, according to a survey of this product category by investment firm A16Z. (Midjourney)
The value of AI companionship is undeniable, especially in its capacity to serve as a listening ear. With over half of adults reporting feelings of loneliness, applications like Replika have demonstrated the potential to reduce suicidal ideation and highlight the significance of having a 24/7 friend or coach. Conversely, the early adoption and success of sexually oriented (and sometimes pornographic) AI bots underscore a different aspect of digital companionship, reminiscent of the substantial funds users spend on platforms like OnlyFans and the widespread popularity of erotic fan fiction.
A16Z’s vision encompasses the growth potential within both wholesome and highly suggestive segments of AI companionship. Despite the differences in product offerings, successful companion apps will likely share common strategies for engaging and retaining users over time. Their focus is on four key themes that will define the future of AI companionship:
Differentiated Value Proposition: The necessity for companion products to distinguish themselves from generalist chatbots, such as ChatGPT, through unique user experiences or uncensored content models for NSFW bots.
New Modes of Interaction: The evolution from text-based interactions to dynamic, immersive experiences with AI companions featuring voices, avatars, personalities, and animations, as seen with innovations like Moemate for the Apple Vision Pro. (Slogan: “Moemate lets you have your AI-enabled virtual assistants in your room!”)
Memory and Progression: Compelling AI relationships must build and deepen over time. Companions that learn from conversations, remember details, and allow the relationship to evolve will be the most engaging.
Human-Companion Interactions: Expanding the scope of AI companionship beyond one-on-one interactions to include group settings with human friends, facilitated by advancements in platforms like Discord bots (e.g., Shapes Inc.).
Moemate in action: your virtual assistant in your room, as seen through an Apple Vision Pro headset. (Screenshot posted by Moemate on X.)
Dark Design Patterns: An Australian Warning
The state government of Queensland in Australia has published a guide to Identifying dark patterns in your business practices, including a description of 8 dark design patterns:
Scarcity cues
Trick questions
Activity notifications
Confirm shaming
Forced continuity
Redirection or nagging
Data grabs
False hierarchy
There is not much new, compared to the article about dark patterns I published recently, based on equivalent work by the Government of India, which outlawed 12 dark patterns in India.
The Australians break “False Urgency” into scarcity cues and activity notifications, which are indeed two of the insidious ways a website can induce a false sense of urgency. The Australian “forced continuity” is the Indian “subscription trap,” which is a clearer — if less general — term.
Data grabs is new: this is the collection of more consumer data than the business needs to provide the product or service the customer wanted. While not as bad as most of the other dark patterns, data grabs can certainly open the door to privacy violations and are also likely to be against GDPR in the EU.
False hierarchy is a classic visual design dark pattern: using the (good) visual design principle to make some design elements more prominent than others to hide important information in those areas of the visual design that are made to appear as unimportant.
False hierarchy: you’ll notice the big, brightly-colored elephant before you pay attention to the smaller, subdued-colored one. This is true even if the small elephant (metaphorically) is the information you really need, such as the cancellation policy for a “free trial.” (Leonardo)
While the Australian contribution to our list of dark patterns is not as extensive as the Indian, I still applaud the Queensland Office of Fair Trading for publishing its analysis of dark patterns. The more, the better, especially when coming from important authorities in major countries.
One interesting data nugget in the new Australian publication: according to research by the Consumer Policy Research Centre, 30% of Australian users stop using a website or app when they encounter dark patterns, which they often consider manipulative or deceptive. As public awareness of dark patterns grows, we can hope that more users will boycott websites that employ these unethical practices.
Free Templates for The Practical Side Of Running a Usability Study
R. Kirán Khan (AI Design Lead for Provenir in London) posted a nice short carousel with a 14-slide overview of usability testing. Perfect for giving a colleague (or even a friend) a quick introduction. As one of my readers, you probably know most of this information already, though it never hurts to see it explained concisely.
Of broader use for my audience: she also posted a set of free templates for the practical side of user testing:
Email Templates
Informed Consent Form
Participant form
Participant profile questionnaire
Recording Permission
Example Tasks for Usability Testing
Observation Sheet
Script for Moderator
Usability Testing Task sheet
Post Interview Questions Template
Post-Test Questionnaire Scale
System Usability Score (SUS) Scale sheet
I’m not a huge SUS fan — it’s too many questions for my taste, consuming valuable study participant time that I would rather spend on observing behavior. (My comments on SUS and my preferred satisfaction questionnaire are currently scheduled for this newsletter on April 15, so stay tuned for more.)
Speed up your user testing projects with templates from R. Kirán Khan. (Ideogram)
Using ChatGPT Without Registration
OpenAI is launching the option to use ChatGPT without having to register first. While this seems nice, I caution against it, except if you’re in a pinch and have to use it from a shared computer.
First, the no-registration version of ChatGPT is more limited than the version you get after becoming a registered user. Using a stunted AI is a prescription for poor results.
Second, you’re getting GPT 3.5, just as you would when using the normal free ChatGPT. (Except you’re getting a reduced-power version of 3.5.) GPT 4 is vastly superior to 3.5 and well worth the $20/month subscription fee for any serious users.
Even if you’re only experimenting with AI to find out whether it’s useful in your job, you should not use the free version. You are less likely to discover the most useful ways of using AI in UX unless you use the most powerful version of AI you can lay your hands on.
Remember Jakob’s 1st Law of AI: Today’s AI is the worst we'll ever have. GPT-5 is coming soon, and all rumors are that it will be yet another step up in AI power. To compete in the future job market, you must build experience and skills that utilize advanced AI and not crippled AI. Subscribe to ChatGPT Plus, if only for a month or two while you make up your mind, or you’ll make the wrong decision.
Don’t use a little-brother AI when you can get the big version for $20. (Midjourney)