108 JOURNAL OF THE SOCIETY OF COSMETIC CHEMISTS In the food field it is common practice to use small panels to judge flavor and texture quality. This table indicates, for example, that unless a panel of 25 finds a preference of at least 69.6 per cent one cannot place much con- fidence in the result. That is, a preference of less than 69.6 per cent can occur so frequently by chance alone that the research man cannot be sure it represents any real preference. It can be noted, however, that with a panel of 300 this situation has been improved considerably, in that a pref- erence as small as 55.7 per cent for one sample is significant at the $ per cent level. That is, a panel of this size will find a preference of 55.7 per cent only $ times in a hundred when the samples under comparison are either identical or equal in acceptability. Of course larger consumer panels permit measurement of even smaller preferences or will lend an even greater degree of significance to a given magnitude of preference. The least significant preference for any given panel size is easily calculated from common statistical procedures (1, 3, 4). TABLE 2--CAPABILITY OF PANELS FOR CONSUMER TESTING (PREFERENCE FOR EITHER OF Two S•.MPLES ) Panel Size "False Alarm"-- Chances of Reporting 60% Preference Where None Exists "Missing the Boat"-- Chances of Failing to Find a Significant Preference When True Preference Is 60% 15 67 in 100 88 in 100 25 40 in 100 83 in 100 50 20 in 100 70 in 100 75 14 in 100 59 in 100 150 2 in 100 31 in 100 300 1 in 100 7 in 100 The data in Table :2 show, in the "False Alarm" column, a different as- pect of the data given in Table 1. Here the assumption is made that we have decided we are interested in a preference of 60 per cent for one sample over the other (which would, of course, have a preference of 40 per cent). The "False Alarm" column shows that even a panel of 25 might report a preference of 60 per cent as often as 40 times in 100 tests. A panel of 50, however, is a little better in that this size panel would report a 60 per cent preference only 20 times in 100 when no real preference existed. To help make this clear, it should be emphasized that these figures actually mean that if panels of the sizes shown in the table are asked to express a prefer- ence between two identical samples, they would report 60 per cent pref- erences as frequently as indicated in the table. Going to the other extreme of the table, it is easy to see that these "False Alarm" results occur much less frequently, or only one time in a hundred, with a panel of 300. Table 2 also shows, in the "Missing the Boat" column, some information as to what will happen with panels of this size when they are comparing

CONSUMER TESTING AS A GUIDE FOR TECHNICAL RESEARCH 109 two samples, one of which is actually preferred to the other by 60-40 per cent margin. That is, these are hypothetical samples, about which we know, presumably from exhaustive testing, that one is preferred to the other by 60 per cent of the total population. The "Missing the Boat" column shows that a panel of 25 would fail to report a significant preference 83 per cent of the time. That is, if we ran 100 tests, each with a different panel of 25 people, we could expect to "Miss the Boat" with 83 of these tests. Similarly a panel of 50 persons would have failed to report a significant preference or will "Miss the Boat" 60 times in 100. Again we can note that the "Missing the Boat" odds are much more favorable with the larger panel sizes. We will "Miss the Boat" only 7 per cent of the time with a panel of 300. TABLE 3--CAPABILITY OF PANELS FOR CONSUMER TESTING (PREFERENCE FOR EITHER OF Two SAMPLES) Panel Size "False Alarm"-- Chances of Reporting 70% Preference where None Exists "Missing the Boat"-- Chances of Failing to Find a Significant Preference When True Preference is 70% 15 20 in 100 66 in 100 25 6 in 100 48 in 100 50 1 in 100 19 in 100 75 1 in 100 7 in 100 150 1 in 100 1 in 100 300 1 in 100 1 in 100 The weaknesses of small panels are even more dramatically illustrated in Table 3. All panels from 25 in number up to 300 perform reasonably well as regards the likelihood of reporting preferences as large as 70 per cent when, in fact, the samples being compared may be identical. How- ever, when we consider the danger of missing a true preference as great as 70 per cent it is apparent that small panels still leave us in serious difficulty. A panel of 25, as shown in the "Missing the Boat" column, may fail to report a significant preference 48 times in 100 when in fact one sample is preferred to the other by 70 per cent of the consumer public (again assum- ing we have two samples whose preference is well established). Only when we use a panel of at least 75 people do we approach a satisfactory assurance that our panel will not fail to find this important 70 per cent preference. Obviously the larger panels are even better. What do the above figures really mean? They are simply statistical calculations which represent the best performance one can expect fi'om panels randomly selected from the universe or population to be sampled. If conditions are less than ideal our panel performance will be even poorer than this. For example, if a panel is drawn from any limited group such