Cognitive Psychometrics – A Book in Progress

From Course to Book

When I lived in Russia and worked at the Institute of Education at the Higher School of Economics, I did a bit of teaching. More specifically, I taught two courses in our Master's program, which was initially called "Measurement in Psychology and Education" and later became "Science of Learning and Assessment." One course was about data analysis methods in the social sciences—we covered topics like multilevel regression and structural equation modeling.

But there was another, smaller course that I taught for about five years, and it evolved significantly over that time. Initially, it was something like an Introduction to Cognitive Psychology. The idea was simply that people working in education should know something about cognition. In its early form (which I didn’t teach the first year), students watched lectures from an open online psychology course and then took a test.

When I took over the course, I had the idea to connect cognitive psychology with psychometrics. Gradually, I transformed the course into something more focused on how cognitive psychology attempts to measure individual differences in cognitive processes. For example, in the topic on Perception, we touched briefly on basic concepts and classic experiments, but then shifted our focus to discussing the kinds of tasks and tests used to assess individual differences in perception. We read papers on topics like variability in the perception of visual illusions. In the Memory topic, we explored tools used to investigate memory features and considered challenges in measuring individual differences—why it’s not as straightforward as it seems (well, was it ever straightforward in psychology?). For instance, even a basic word or digit span task turns out to be not so simple.

Each year, I became more immersed in this theme—the relationship (or rather, the complex relationship) between cognitive psychology and psychometrics. Eventually, we launched a project-based learning group called Cognitive Psychometrics. For two years, we ran a Telegram channel and organized seminars. The group no longer exists, but my former colleague and good friend Yulia Tyumeneva and I continue to run the channel, now focusing more on general issues of measurement in psychology and various ongoing "crises" in the field: theoretical, replication, statistical, etc.(https://t.me/cogn_psychometric)

Out of this teaching and project experience came an article titled “Psychometrics and Cognitive Research: Contradictions and Possibility for Cooperation” (available in Russian: https://vo.hse.ru/article/view/16875). While finishing the article (by that point I was already living in Belgrade), I had the idea to write a book about the topic—something more in-depth, exploring the history and current challenges of measuring individual differences in cognitive research, and the uneasy relationship between cognitive psychology and psychometrics.

But I’ve been postponing it, still collecting material. Right now, I’m stuck in the historical part—it seems endless—and sometimes I have to dive into real rabbit holes. Lately, I’ve gotten very interested in the broader history of statistical methods in psychology. I’m thinking this could become a separate chapter. The book plan keeps growing. 😊

But I thought I could use this blog to share small excerpts from my draft materials. I’m writing in Russian for now, but the idea is to publish the book in English eventually—though I’m still unsure. What do you think—would such a book be of interest?

For now, here’s a short piece from the first chapter on history (about Cattell—this is a shortened version; I still haven’t learned to write concisely 😊):

Excerpt: Cattell and the Beginnings of Psychometrics in USA

One of the prominent American psychologists inspired by Galton’s ideas on measuring individual differences was James McKeen Cattell, who had studied under Wilhelm Wundt. As early as 1886, in his work on measuring mental reaction times, he described two possible approaches to studying perception: psychophysics (including the work of Fechner and Weber) and psychometry. He saw psychometry as an underdeveloped but promising area:

“Psychometry seems to be of as great psychological interest as Psychophysics, but it has not been nearly so fully and carefully worked over” (Cattell, 1886, p. 63).

Later, he emphasized that psychology must develop as both an experimental science and one grounded in testing and measurement:

“Psychology cannot attain the certainty and exactness of the physical sciences, unless it rests on a foundation of experiment and measurement. A step in this direction could be made by applying a series of mental tests and measurements to a large number of individuals...”[Cattell & Galton, 1890, p. 373].

Historian of science M. Sokal (2010) speculated that Cattell’s turn toward simplified experimental methods and mental testing may have stemmed from a lack of aptitude for introspection (then the dominant method in psychology). Whatever the reason, Cattell was instrumental in changing how experimental psychology was practiced in the U.S.

Even before meeting Galton, Cattell had attempted, during his time at Johns Hopkins University, to redesign experimental protocols to minimize reliance on introspection. This approach later influenced many psychologists in the U.S. and Europe. Despite his success, Cattell had to leave Johns Hopkins—possibly due to conflict with fellow psychologist G. Stanley Hall, who tried to take credit for Cattell’s work (according to Sokal).

In 1883, Cattell joined Wundt’s lab in Leipzig. Contrary to popular myth, he was not the first American there. [...] In 1886, he received his doctorate and moved to the U.K., where he met Galton and visited his anthropometric lab. Inspired, he set out to create something similar in American universities.

[...]

Cattell designed a large-scale ability testing program for college students, which included tasks to measure reaction time, motion speed, sensory thresholds, and more. He stressed that each participant should complete a significant number of trials per task so that averages, variances, and ranges could be computed—meant to describe each student’s abilities and predict academic success.

[...]

From the beginning, some psychologists criticized his approach, preferring Binet’s view that individual differences should be studied via investigating complex abilities, not simple sensorimotor reactions. Still, Cattell considered his sensory measures to be psychologically meaningful—perhaps following Galton’s lead, or perhaps simply because these measures were easier to obtain and quantify.

By the late 1890s, Cattell had gathered data from around 1,000 Columbia University students. Though he was aware of correlation methods developed by Galton and Pearson, he found the math too difficult:

“The mathematics are too much for me” (quoted in Sokal, 2006).

As a result, his students had to learn to compute correlations themselves. One of them, Wissler, analyzed the relationships among Cattell’s tests and between test scores and academic grades. The results were disappointing: no significant correlations were found [Wissler, 1901].

[...]

The number of psychologists in U.S. universities grew rapidly in the early 20th century. Most were eager for psychology to be seen as a "real" science, leading to more experimental labs and increasing use of "objective" language. Cattell played a major role in shaping this language. For example, he popularized the term "subject" to describe participants in experiments—emphasizing the analogy to physical sciences.

Because of his "objective" approach, some historians have called Cattell the "grandfather" of behaviorism—a school that saw psychology as the study of observable behavior (Burnham, 1968, p. 149). For example, in 1904, at a psychology congress in St. Louis organized by Hugo Münsterberg, Cattell gave a keynote titled “The Conceptions and Methods of Psychology.” He asserted that his 20 years of experimental work were no more reliant on introspection than the work of a zoologist or physicist:

“It is usually no more necessary for the subject to be a psychologist than it is for the vivisected frog to be a physiologist” (Cattell, 1904).

He added that introspection belonged more to the realm of art than science. According to historian of psychology Kurt Danziger, this speech marked a turning point for many psychologists, who began abandoning the Leipzig model of experimentation. Interestingly, among the audience members was James R. Angell of the University of Chicago—whose student was none other than John B. Watson, the founder of behaviorism".

What do you think—does it make sense to keep writing on this topic and share some fragments from the book-in-progress? Maybe it will motivate me to work on it faster and more consistently.