CONSUMER TESTING AS A GUIDE FOR TECHNICAL RESEARCH 111 can be expected to fall. Again, in referring to the true preference we mean that preference result which would be obtained if a very large sam- ple of the population were tested. Table 4 shows a typical range chart which can be calculated for convenience in evaluating consumer test results. As Table 4 indicates, the ranges given indicate where one may expect to find the true preference. For example: A consumer test has been run with a panel of 100 people. The result shows a 60 per cent preference for one of the samples. If this test were to be repeated again and again with new groups of 100 consumers each time, the final average preference of all the tests would most probably fall somewhere between 50.4 per cent and 69.6 per cent. Because this range was calculated for the 5 per cent significance level it follows that there is a 5 per cent chance, that after all this repetitive testing the true result might be found to fall above or below, the given range. Obviously, similar ranges could be calculated for a 1 per cent significance level or any other degree of security that the re- searcher may wish to stipulate. This chart is presented here because it emphasizes the relatively poor precision of consumer test da, ta. It is apparent that this precision increases as the size of the panel increases, but even so, the limits of confidence are large compared to other data with which most technicians work. Most analytical data, for instance, are much more precise than any but the larg- est of consumer panel results. TABLE 4--RANGE CHART (PREFERENCE FOR SAMPLE A OVER SAMPLE B) Range About Observed Result Within Which True ------Observed Results- ' Values Should Lie (5% Significance Level)- As Preferred, As % Preferred, 100 Judgments, 300Judgments, 500Judgments, Ratio A/B for A Possible Range Possible Range Possible Range 1.0/1.0 50.0 •-9.8 •-5.7 •-4.4 1.2/1.0 54.6 q-9.8 q-5.6 q-4.4 1.4/1.0 58.4 4-9.7 4-5.6 •-4.3 1.5/1.0 60.0 4-9.6 4-5.5 4-4.3 2.0/1.0 66.7 -½9.0 4-5.3 -½4.1 Training and experience to create a group of "experts" is not a remedy for the poor precision of panels. The precision discussed here is partly made of reliability and partly representativeness. The reliability can be improved by training and experience. That is, experts in tasting and smell- ing can become very reproducible in their ability to recognize specific characteristics of a product, to describe these characteristics, and to estimate their strength or their contribution to the over-all flavor or scent. This sensitivity, however, necessarily means that they are less representative and less typical of the consumer public. In most respects they become less useful as a guide to consumer acceptance.

112 JOURNAL OF THE SOCIETY OF COSMETIC CHEMISTS The expert fulfills a very useful function with his ability to describe and classify sensory properties. He may be able to describe in detail how Chanel #5 differs from a cheap imitation. However, the expert cannot tell whether the mass market would prefer it for its characteristics alone if it were available under another name at dime store prices. Only a con- sumer panel can truly speak for the likes and dislikes of consumers. The data on precision, as presented in this paper, should not be dis- couraging to the use of consumer panels, but only to the use of small panels. Consumer data, as the foregoing tables show, offers reliability and repre- sentativehess to the extent that we are willling to pay for it. There are no bargains in the sense of obtaining just as good results from small numbers of people as from large. The precision of the result can be predicted in advance from very simple and straightforward statistical concepts. Where the ultimate research objective is consumer satisfaction, there is no substitute for guidance direct from the consumer himself. As we all know, the consumer will eventually have his own way. BIBLIOGRAPHY (1) Brownlee, K. A., "Industrial Experimentation," Chemical Publishing Co. (1952). (2) Dixon, W. J., and Massey, F. J., Jr., "Introduction to Statistical Analysis," McGraw-Hill (1951). (3) Fisher, R. A., "Statistical Methods for Research Workers," Oliver & Boyd l.td. (1948). (4) Guilford, J.P., "Psychometric Methods," McGraw-Hill (1936). DISCUSSION MR. LOgOYELLOW: I have a question in reference to Table 2 and Table 3, whether only a choice was offered to the panelists, or whether they were permitted to express a "no prefer- ence." MR. ISHLER: Yes, they were. What we have done with all of these data--you could get into a long discussion on that, too--is to divide the "no preference" votes equally between the two samples there, by your two percentages, the preference for one sample, always adding up to 100. I should not claim that this is the proper way to do it, but it is one way, and it is useful in many respects.