Home testing has a place in the (post-) pandemic world

An ESN-funded study finds an advantage of home testing compared to lab testing. [Download text as PDF]

Makes people feel more at ease - testing products at home

An analysis of 245 data sets from members of the European Sensory Network revealed that the pandemic-induced switch to testing products at home instead of in the lab did not impair the quality of the data gained from consumer product research.

It turned out that testing products at home reduced the noise in the data for certain testing methods. This improvement could be due to the study participants feeling more at ease and more committed when evaluating products from their own homes. Statistician Anne Hasted from ESN member Qi Statistics, who spearheaded the analysis, emphasizes: “This analysis indicates that In-Home testing has a place in the future post-pandemic world”.

A pandemic changes sensory methods

Flashback: March 2020 – one country after the other introduces lockdown measures to prevent the spread of the Corona virus. “This didn ́t come as a total surprise to us. Nevertheless, we didn ́t have much time to prepare and adjust our sensory testing to a situation in which we couldn ́t use our labs anymore”, remembers Tracey Sanderson, Managing Director at Sensory Dimensions, a sensory and consumer research agency in Reading, UK.

Meanwhile, Keren Corley sits in Tel Aviv at the kitchen table in her home office and faces the same problem: How to ensure that ongoing and planned sensory and consumer testing continues without labs? “Actually, I felt energized by this challenge”, reports Keren, CEO of NewSense. The rest is history. Sensory Dimension, NewSense, SAM, isi and all the other ESN members who are involved in conducting sensory research switched from lab testing to testing at home.

A smartphone, a kitchen table and orange juice samples – NewSense brought sensory testing to the homes in Israel.

A start-up mentality helps with the adjustment

Vegan burger patties, sugar-reduced lemonades, protein-enriched ready-meals for the elderly – no matter which product, and no matter which country – the research participants’ living rooms became the new sensory labs. This was all possible because sensory and consumer research agencies and are used to react quickly and creatively.

“I never would have thought before the pandemic that we could change our methods that quickly” remembers Martin Kern, head of SAM Sensory and Marketing International (Watch an interview with him here). Bigger industries, on the other hand, had a harder time adjusting to the change. The pandemic revealed that the sensory and consumer research agencies have a hidden startup mentality at the heart of their businesses.

“We changed one thing, and then another, it was like the pandemic had reactivated the courageous entrepreneur in us”, reports Keren Corley, and you could hear the pride in her voice. (Watch an interview with Keren about how NewSense mastered the Corona crisis.)

Sensory testing with a personal touch. A study participant from isi insights (Germany) sent a picture of her testing site to the study leaders.

What if the switch leads to worse data?

Yet, some cautious ESN members put out a warning: “attention: reduced visibility”. Anne Hasted, statistician and head of the UK-based data analysis agency Qi Statistics warned that testing at home might compromise the data. “We need to have a much better understanding of how the testing at home changes the evaluations of the test participants”, she said at an online ESN meeting in spring 2020. The faces online nodded in agreement. “The solution to the challenge created by the lockdown of the lab testing sites should not lead to worse data”, added Jean A McEwen, research manager at ESN.

An eye for data - Anne Hasted during a pre-pandemic ESN meeting at Windsor

Combining data from ESN members

It turned out that many members had had similar questions. When compared to lab testing, does home testing with consumers create more variation in the participants ́ responses? Does the comfy context of the home change the acceptability of a product? And one important question that every sensory researcher needs to address before the testing starts: How many participants are needed so that the data quality in home testing matches that of lab or central location consumer testing?

Many of the ESN members had already addressed these questions and had conducted studies to compare the outcome of lab and home testing with trained sensory panels. “We were testing lemonades and switched in the middle of lab testing to home testing, and thus had data sets we could easily compare”, remembers Sven Henneberg, Senior Manager at isi in Munich (Watch an interview with him on pandemic-related changes at isi here).

The next step was clear. Anne Hasted and her team would collate all the data comparisons between consumer lab / CLT and home testing that ESN members had collected and then look at the outcome. Since Anne ́s company Qi Statistics doesn ́t conduct sensory and consumer testing itself, Anne would have a somewhat detached look at the results and would be able to come to an unbiased conclusion.

All set up at the kitchen table. Cream cheese testing at home.

How noisy is home testing?

ESN members and partners play an active role in identifying research topics, some longer term and others like this one which are fast-turn-around and budget-friendly, often referred to as snapshot projects. The ESN Agile Research Process allows more immediate research questions to be addressed in a timely way without unnecessary bureaucracy slowing us down. The results of the project, entitled “Home testing for the future” were presented to ESN members by Anne Hasted in June 2021. During the online presentation you could hear sighs of relief from the attending members – as well as some hold-your-breath-silence. Here are the results.

ESN members provided 245 data sets for the analysis, spanning tests that were carried out in 19 countries. The testing sites most represented in the data set were the UK, Israel, France and Germany. Most tests were on such food products as yoghurt, cake, milk substitutes. There were also tests on personal care and household products.

Sequential monadic testing was the method most often used – a series of product variants are tested one after the other (e.g. instead of the testing of just one sample). At the center of the analysis was the noise variation in the testing. A high noise variation means that there is an unwanted variability in the evaluations of the test participants. The noise makes the data less reliable.

Not as cosy as at home - a typical setup in the sensory lab

And the winner is ...

The results demonstrated that there was no higher noise variation in the home testing compared to the lab testing (included in the analysis were traditional Central Location Tests, traditional Home Use Tests and In-Home Central Location Tests). In general, the noise became louder as the number of products tested increased – be it in the lab or at home. But, for the ESN members who were listening attentively to Anne’s presentation, the data had a surprise to offer. She reported that, “If you compare the lab testing with the traditional test protocol – the so-called Central Location Test – to In-Home testing with the same test protocol, In-Home testing gets you a reduction in average noise variation of nine percent”.

Could the pandemic have propelled a new method forward that makes sensory testing even more reliable? The answer seems to be yes. After discussing the reason behind this unexpected finding, Martin Kern from SAM sums up the thoughts of the ESN members. “The study participants probably were more committed when they were conducting the test at home compared to the unfamiliar environment in the lab, with strangers around them.

The number of samples matters

This was good news for the shift of testing during the Covid pandemic, and also good news for the future of sensory testing at home. But statistician Anne Hasted also had a warning at the end of her presentation. The aggregate of data from all the sets hinted at two aspects that need to be addressed to ensure reliable test results. The number of participants needs to go up when the number of products samples goes up. For example, to acquire highly significant results in a test with 16 product samples, one has to recruit over 400 participants (alpha < 0.05; 90% power). “The statistical power will be higher for In-Home testing because the noise variation is lower,” Anne Hasted points out. In the future, more participants will do the testing from their home.

Home Testing could be an option for people who are not so easy to recruit for lab testing, such as kids and the elderly.

Home testing has a future

Furthermore, the aggregated data revealed unwanted effects having to do with the order of the presented samples. Usually, the participants gave the first sample a higher liking score. There are different ways to avert this effect. Each sample should be presented in each position an equal number of times, or a dummy sample needs to be presented at the start of each session. The analysis of the data set has led to ESN Consumer Testing Guidelines that are helpful not only for testing in the Covid times but also in a post-pandemic world.

The pandemic challenged ESN members to take a closer look at the data acquisition in sensory and consumer research. Sensory researchers will be glad when the testing can fully and safely resume in the research facilities, but there are good reasons to integrate more Home Testing in sensory and consumer research in the future. This text just captures a part of the analysis from the ESN research programme “Home Testing for the Future”.

Link to the original publication: Anne Hasted, Jean A. McEwan, MaryJenner: Controlled product testing in the home - implications for sample size. Food Quality and Preference Volume 103, January 2023, 104688

For further insights into the statistics (especially regarding order effects, effects of number of products tested) and the ESN Consumer Testing Guidelines, please contact Anne Hasted or Jean A McEwan.

[Text Written by Fabienne Hübener]