Consensus based assessment (CBA)

Consensus Based Assessment (CBA)

Peter Legree and Joseph Psotka proposed that psychometric g could be measured unobtrusively through survey-like scales requiring judgments. The measurement could be by either using a deviation score for each person from the group or expert mean; or by using a Pearson correlation between their judgments and the group mean. The two techniques are perfectly correlated.

They subsequently created scales that requested individuals to estimate word frequency; judge binary probabilities of good continuation; identify knowledge implications, and approximate employment distributions. The items were carefully identified to lack objective referents, and therefore the scales required respondents to provide judgments that were scored against broadly developed, consensual standards.

Performance on this judgment battery correlated approximately .80 with conventional measures of psychometric g. The response keys were consensually derived. Unlike mathematics or physics questions, the selection of items, scenarios, and options to assess psychometric g were guided roughly by a theory that emphasized complex judgment, but the explicit keys were unknown until after the assessments were made: they were determined by the means of everyone's responses, using deviation scores, correlations, or factor scores.

“Consensus Based Assessment (CBA)”, expands on the theoretical observation that expertise can be closely approximated by large numbers of novices or journeymen; and the shared knowledge that forms cultural consensus can be interpreted in much the same way as expertise. According to this view, if samples of individuals with differing competence (e.g., experts and apprentices) rate a relevant scenario, then they would provide similar mean ratings for each item. Thus, from the perspective of a CBA framework, cultural standards for scoring keys can be derived from the same population that is being assessed.

One way to understand this expectation is to consider that for many performance domains expertise largely reflects knowledge derived from experience. Because novices will tend to have fewer experiences, their opinions may err in various directions and these errors will not be very consistent. However, as experience is acquired, the opinions of experts and even journeymen will become more consistent. According to this view, errors are random and ratings data collected from large samples of respondents whose knowledge and skills cover a broad range of expertise can be used to approximate the rating means that would be collected from a substantial number of experts, were they available. Because the standard deviation of a mean will approach zero, 0, as the number of observations, N, becomes very large, estimates based on groups of varying competence will provide converging estimates of maximal performance. The means of these groups’ responses can be used to create effective scoring rubrics to evaluate performance. This approach is particularly relevant to scoring subjective areas of knowledge that are scaled using Likert response scales, and the approach has been applied to develop scoring standards for several domains that have lacked experts.

In practice, analyses have demonstrated high levels of convergence between expert and CBA standards with values quantifying those standards highly correlated (Rs-e,s-c ranging from .72 to .95), and with scores based on those standards also highly correlated (Re,c ranging from .88 to .99) provided the sample size of both groups is large (Legree, Psotka, Tremble & Bourne, 2005). For these five domains, and for an additional three domains that could only be scored using consensus based measurement, scores based on expert and CBA standards correlated with relevant criteria. This convergence between CBA and expert referenced scores and the associated validity data indicate that CBA and expert based scoring consistently evaluate options for items provided ratings data are collected using large samples of experts and examinees to construct scoring rubrics.

CBA and Factor Analysis

CBA is often computed by using Pearson R correlation of each person's Likert scale judgments across a set of items against the mean of all people's judgments on those same items. The correlation is then a measure of that person's proximity to the consensus. It is also sometimes computed as a standardized deviation score from the consensus means of the groups. These two procedures are mathematically isomorphic. If culture is considered to be shared knowledge; and the mean of the group’s ratings on a focused domain of knowledge are considered a measure of the cultural consensus in that domain; then both procedures assess CBA as a measure of an individual person’s cultural understanding.

However, it may be that the consensus is not evenly distributed over all subordinate items about a topic. Perhaps the knowledge content of the items is distributed over domains with differing consensus. For instance, conservatives who are libertarians may feel differently about invasion of privacy than conservatives who feel strongly about law and order. In fact, standard factor analysis brings this issue to the fore.

In either centroid or principle components analysis (PCA) factor analysis the first factor scores are created by multiplying each rating by the correlation of the factor (usually the mean of all standardized ratings for each person) against each item’s ratings. This multiplication weights each item by the correlation of the pattern of individual differences on each item (the component scores). If consensus is unevenly distributed over these items, some items may be more focused on the overall issues of the common factor. If an item correlates highly with the pattern of overall individual differences, then it is weighted more strongly in the overall factor scores. Now, this weighting implicitly also weights the CBA score, since it is those items that share a common CBA pattern of consensus that are weighted more in factor analysis.

The transposed or Q technique factor analysis ( brings this relationship out explicitly. CBA scores are statistically isomorphic to the component scores in PCA for a Q technique analysis. They are the loading of each person’s responses on the mean of all people’s responses. So, Q factor analysis may provide a superior CBA measure, if it can be used first to select the people who represent the dominant dimension, over items that best represent a subordinate attribute dimension of a domain (such as liberalism in a political domain) and then factor analysis can provide the CBA of individuals along that particular axis of the domain.

This suggests that when items are not easily created and arrayed to provide a highly reliable scale, that first a Q factor analysis should be carried out to create factors for each person’s Likert ratings; and then those factors can be selected whose component scores best describe the attribute dimension as the CBA scale. So, for instance, in a scale of items for political attitudes, the items may ask about attitudes to big government; law and order; economic issues; labor issues; or libertarian issues. Which of these items most strongly bear on the political attitudes of the groups polled may be difficult to determine a priori, and demand first a Q factor analysis to select the people with the greatest consensus, whose ratings correlate best with the different groups of political views. Once selected, that factor determines the CBA (component) scores.

See also

• Collective intelligence
• Consensus
• Consensus democracy
• Consensus theory of truth
• Emotional Intelligence
• Intelligence
• Participation (decision making)
• Polder model
• Social representations
• Unanimity
• Libertarian Socialism

References

Legree, P. J., Psotka J., Tremble, T. R. & Bourne, D. (2005). Using Consensus Based Measurement to Assess Emotional Intelligence. In R. Schulze & R. Roberts (Eds.), International Handbook of Emotional Intelligence. (pp 99-123). Berlin, Germany: Hogrefe & Huber.

External links

• Meta Collab - a free collaborative encyclopedia on collaboration.
• Information and Collaboration Technologies (see Chapter 5): Managing Collective Intelligence, Toward a New Corporate Governance
• Smart Mobs
• The Wisdom of Crowds