110 JOURNAL OF THE SOCIETY OF COSMETIC CHEMISTS as the secretarial staff of a large office building, there may be a tendency toward a bias in that group. The data in the above tables may apply only to the preferences of similar groups of secretaries with similar environ- ment, background, economic status, and tastes. The results of the test may have little relationship to the likes and dislikes of the total consumer group for whom the product is designed. We have no statistical method of measuring the relationship of pref- erence judgments from-such a group as compared to a total population of a city, state, or country. Such a correlation can be established only by testing each group to measure the diversity of likes and dislikes. Even when this information has been obtained it cannot be used in most cases to apply to any other set of samples. Therefore, it behooves the researcher to select his sample as representatively as possible from the type of people whose opinions he wishes to measure. The data in the above tables which show the chances of failing to find real preferences with a panel of a given size can also be presented in graph- ical form as in Fig. 1. This chart consists of so-called power curves (2). The use of these curves permits reading directly, for a given size panel, the chances of "Missing the Boat." Another way of expressing the reliability of consumer test results is by showing the precision limits or the range within which the true preference 1.00 o.oo 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.0 1.00 50 80 55 60 65 70 75 True Preference •/o) • = 0.05 Power curves for panel performance of different size N o.o 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 o.oo

CONSUMER TESTING AS A GUIDE FOR TECHNICAL RESEARCH 111 can be expected to fall. Again, in referring to the true preference we mean that preference result which would be obtained if a very large sam- ple of the population were tested. Table 4 shows a typical range chart which can be calculated for convenience in evaluating consumer test results. As Table 4 indicates, the ranges given indicate where one may expect to find the true preference. For example: A consumer test has been run with a panel of 100 people. The result shows a 60 per cent preference for one of the samples. If this test were to be repeated again and again with new groups of 100 consumers each time, the final average preference of all the tests would most probably fall somewhere between 50.4 per cent and 69.6 per cent. Because this range was calculated for the 5 per cent significance level it follows that there is a 5 per cent chance, that after all this repetitive testing the true result might be found to fall above or below, the given range. Obviously, similar ranges could be calculated for a 1 per cent significance level or any other degree of security that the re- searcher may wish to stipulate. This chart is presented here because it emphasizes the relatively poor precision of consumer test da, ta. It is apparent that this precision increases as the size of the panel increases, but even so, the limits of confidence are large compared to other data with which most technicians work. Most analytical data, for instance, are much more precise than any but the larg- est of consumer panel results. TABLE 4--RANGE CHART (PREFERENCE FOR SAMPLE A OVER SAMPLE B) Range About Observed Result Within Which True ------Observed Results- ' Values Should Lie (5% Significance Level)- As Preferred, As % Preferred, 100 Judgments, 300Judgments, 500Judgments, Ratio A/B for A Possible Range Possible Range Possible Range 1.0/1.0 50.0 •-9.8 •-5.7 •-4.4 1.2/1.0 54.6 q-9.8 q-5.6 q-4.4 1.4/1.0 58.4 4-9.7 4-5.6 •-4.3 1.5/1.0 60.0 4-9.6 4-5.5 4-4.3 2.0/1.0 66.7 -½9.0 4-5.3 -½4.1 Training and experience to create a group of "experts" is not a remedy for the poor precision of panels. The precision discussed here is partly made of reliability and partly representativeness. The reliability can be improved by training and experience. That is, experts in tasting and smell- ing can become very reproducible in their ability to recognize specific characteristics of a product, to describe these characteristics, and to estimate their strength or their contribution to the over-all flavor or scent. This sensitivity, however, necessarily means that they are less representative and less typical of the consumer public. In most respects they become less useful as a guide to consumer acceptance.