Great Careers in Testing

John Rust, Ph.D.

Dr. John Rust is Director of The Psychometrics Centre at the University of Cambridge. His exceptional academic and intellectual reputation in the field of testing and assessment stem from his work ranging from the investigation of advanced statistical and computational techniques for use in test development, to the UK standardizations of widely used psychometric tests.

Professor Rust has authored several tests, including:

• GRIMS and GRISS, assessments of personal relationships
• RISC, an assessment of clinical state
• Orpheus, a work based personality test
• Giotto, an integrity test
• RANRA, a test of numerical reasoning ability

Professor Rust's publications span a broad range of testing topics, only several of which we will list here: behavioral and cognitive development; sexual function; testing across languages and cultures; educational and achievement testing; numerical reasoning and test development using artificial intelligence. John is also a Senior Member of Darwin College.

Your Career

How did you first get interested in psychometrics? How did it happen that you chose it as the focus of your career?
My first degree was in statistics and computer science as well as psychology, and so it was a natural progression for me when I began my PhD at the Institute of Psychiatry in the University of London. I became particularly interested when I realized the damage that the race and IQ debate was doing to what was probably the most influential field of applied psychology. My subsequent MA in Philosophy focused my mind on reconciling the science with the ethics of assessment.

What have been the most enjoyable areas of your career in testing? What do you most enjoy thinking about?
It has varied -- at the moment it is testing in the online environment, particularly within social networks. Suddenly, within only a few years, everyone can test themselves online in almost any aspect of their personality or ability and get immediate results from high quality instruments. Our myPersonality Facebook App now has over 6 million users worldwide.

What has been the biggest challenge of your work and how did you overcome it?
It was maintaining the field throughout the 1980s and 1990s. During this time psychometrics was dropped as a subject from almost every psychology course in the US and UK, partly because it was too mathematical at a time of expanding access, but more potently because it was too political. I believed then, as I do now, that the only way to challenge racism in psychometrics is to tackle the arguments head on. Fortunately Jim Flynn and his 'Flynn Effect' has made this very much easier these days, and we are delighted to have him as a Distinguished Associate of the Psychometrics Centre.

Please tell us about your work in adaptive testing?
It has been a concern for some time now that only the large test publishers have access to adaptive testing technology. The absence of expertise within the universities has meant that no one was being trained in how to develop and use one of the most important trends in the field. Hence we decided to develop Concerto, an open source adaptive test development platform, now in its 3rd Version. It uses purely open source elements -- the MySQL database, the R programming language and CK editor for editing the HTML code. And our designed interfaces mean it is not even necessary to know how to program in these languages in order to present your test online, give feedback to participants, and introduce adaptivity.

Which new testing technologies have you most excited lately?
Probably the recent breakthrough in Multivariate Item Response Theory (MIRT). This has implications not just for introducing adaptivity to multifaceted personality tests but also to ipsative tests.

What are the key points of your work on test development achieved via artificial intelligence? Machine intelligence algorithms are very similar to those used in modern psychometrics, particularly within item response theory and test equating. For example, Bayesian a-priory probabilities, when combined with actual responses, can give much better initial estimates of a person's likely score, and thereby reduce quite considerably the testing time and the number of items required.

Please tell us about your GRIMS and GRISS assessments of personal relationships.
The Golombok Rust Inventory of Sexual Satisfaction and the Golombok Rust Inventory of Marital State were first developed in the 1980s to assess improvement and outcome during sex therapy and marriage guidance carried out at the Maudsley Relationship Clinic in London. They subsequently became an international standard in these fields. I enjoyed developing them because they were very challenging -- people are rather nervous about revealing these sides of themselves (quite understandably) and hence the choice, design and piloting of the questions is crucial.

What are the most important elements to consider when designing and developing secure high stakes tests and their supporting administration, scoring and other systems?
Security and reliability, but also fairness. Tests are designed to assess ability, aptitudes and characteristics as much as possible independently from very specific test-taking skills. But clearly, as with interviews, practice helps, and it is very difficult to disentangle practice effects from estimates of the true score on the underlying trait. With the increasing popularity of psychometrics, and the booming demand for training in test taking skills, the problem (and the unfairness resulting therefrom) can only get worse -- but we can't turn back the clock. In many ways it is the same problem as is found in education where teachers are increasingly tempted to 'teach to the test', rather than instill the underlying creative and critical skills in their students.

Reflecting on the lessons learned during your career, what do you now believe is the primary value of testing and credentialing in our society? How has your viewpoint evolved though the years?
That's a very difficult question to answer as it is impossible to imagine a society functioning without assessment. Assessment is everywhere -- choosing someone or being chosen for a job; beginning a suitable career; being diagnosed or diagnosing an illness; even choosing a partner. We hope all of these assessments will follow the same four fundamental psychometric principles: reliability (freedom from error), validity (they come up with the right result), standardization (the right comparisons are made), and freedom from bias. From the psychometricians' point of view it's not just about testing, it's about applying and interpreting the psychometric principles in assessment more generally.

Psychometric Education and Careers

For people considering a career in psychometrics, what are important qualities, skills or talents they should possess to be successful?
Being numerate, at the same time as being able to see things from both an arts and a science point of view. The majority but not all psychometricians have a first degree in psychology, but others come from a background in teaching, mathematics, business, statistics and computing.

To help people considering the field of testing and measurement, what are the specific fields of testing and testing industry sectors?
Education (particularly within the examination boards), recruitment consultancy and human resource management, clinical diagnosis and epidemiology, social surveys and the national census and, increasingly with the advent of social networks, advertising, marketing and security.

What are the most rare and valued talents and skills in testing?
Having a combination of skills. Some people can design tests, others can put them online. Some can do the statistics; others can provide good feedback to both candidate and client alike. The person who can do all of these is very rare -- but to have an overarching understanding of the whole process pays dividends.

What are the types of psychometric degrees, programs and certifications available in the UK and how many people earn them each year?
Very few. In terms of university degrees, psychometrics is only offered at graduate level in Cambridge and Edinburgh, and shortly in Kent and York. The ideal candidate will be numerate and already have a first degree in psychology. For Human Resource professionals there are professional training courses that enable them to administer tests in recruitment.

What have you discovered to be the best ways to attract, compensate and work with extraordinary testing educators and professionals?
Making the topic interesting. I think we achieve this quite well on our website.

What qualities do top test creators have in common? How important is expertise in making tests compared with expertise of the subject matter in question?
Today, it's a combination of imagination and internet know-how. You can no longer make a good test without understanding the principles of item response theory, so an ability to obtain the necessary skills in mathematical or statistical modeling is now essential. The subject matter is, of course, important. But generally the client will provide this -- the psychometrician's job is to turn this, whether it be a job or person description or a curriculum, into a reliable and valid assessment against predefined standards that is free from any important sources of bias.

Testing Now and in the Future

This may not be possible but I am going to ask… could you please provide us with a summarized history of testing in a few paragraphs? Is anyone credited with inventing the first test?
It's lost in the mists of time, I'm afraid. By the time of the Xi Zhou Dynasty in China (1046BC to 771BC) candidates were already being assessed for the Chinese Civil Service on the six skills (arithmetic, archery, horsemanship, music, writing and performing ceremonies) the six conducts (filial piety, friendship, harmony, love, responsibility and compassion) and the six virtues (insight, kindness, judgment, courage, loyalty and concord). The Chinese even know about reliability and validity, for example having more than one assessor and assessing blind. In modern history, the key moment was the bridge between the Leipzig school of German psychology under Wundt and British Social Darwinism under Francis Galton, which came about in Cambridge in 1878 where James McKeen Cattell created the first scientific laboratory devoted purely to psychometrics. This was to be short lived as Cattell then returned to his US homeland in 1889 to take up the first ever US Chair in Psychology at the University of Pennsylvania in 1890. His seminal text, 'Mental Tests and Measurement', published in the journal 'Mind' in 1891, probably marks the beginning of psychometrics as it was known throughout the following century. Paper was then key, and millions were tested, selected, and sorted in processes of military recruitment, university admission and educational diagnosis for decades to come. However, psychometrics was put to bad as well as good use, particularly by the followers of the eugenics movement and those with an enthusiasm for exploring and exploiting racial differences. It was only the universal recognition of the need for a scientific approach if assessment was to be fair and of benefit to society that rescued the discipline from the hole into which it had dug itself.

Modern psychometrics began in the 1960s with the development of mathematical and computational tools for the analysis of data at item rather than test level. However, Item Response Theory, or IRT as it became known, only came into its own once computers and the internet were sufficiently developed to deliver adaptive tests online in real time. The introduction of social networks has seen a massive explosion in their use during the past few years, which is set to continue into the uncertain future.

What do you estimate is the current size of the testing industry in the UK? What % growth rate do you estimate over the next 10 years?
In the UK alone, every school, most hospitals and around 70% of companies with more than 50 employees use psychometric tests, so the market runs into millions of GBP, and worldwide into billions of USD. The question about growth rate, however, raises many interesting questions. If growth is assessed in terms of number of testing sessions, it is exponential and phenomenal. But in terms of cash it's much more difficult to say as the costs involved are falling through the floor. All publishers today, whether print, music, media or testing are facing unprecedented challenges, so it would be somewhat foolhardy to guess even five years in to the future, let alone ten.

How many tests does the average organization require of employee candidates prior to hiring in the UK? What are the most popular of these tests? How many and which tests would you recommend?
I don't know, but not many. Perhaps one or more than one of these: numerical reasoning, verbal reasoning and personality.

What percentage of tests is currently taken online in the UK and what do you estimate that percentage will be 5 years and 20 years?
I don't have these figures either. But I do know that 10 years ago our clients always demanded paper and pencil even when we offered computer based or online versions. I don't even remember being asked for a paper version of a test for two years at least. Today the main issue is whether a test should be administered directly over the internet or downloaded first to avoid problems with connectivity. On current trends I would expect all tests to be administered directly, either over the internet or some other form of wireless technology, within the next two years.

What new psychometric education programs do you foresee in the future?
Many more PhD and Masters level programs are required. There is an urgent need to train the next generation of trainers, and there is a serious shortage of university academic staff with the necessary knowhow -- this is a problem generally with quantitative methods in the social sciences, but it is particularly serious for psychometrics.

If you had the power to change the world of assessment, what would be the first 3 things you would do?
Recruit many more psychometricians from ethnic minorities in the USA and Europe. Ensure that open-source software for modern psychometrics methods was available worldwide, and ask everyone to insist on feedback on the results of any assessments they may have.

What advice would you give to anyone interested in starting a new certification exam based on a new field of expertise?
Plan as best you can, but be prepared for the unexpected and be sensitive to feedback when folk say you may be getting it wrong.

What are 5 of the most influential testing organizations worldwide?
Internet companies, Governments, Examination Boards, Health Services, and test publishers, probably in that order.

Who do you consider to be 5 of the most influential test authors in history?
James McKeen Cattell, Alfred Binet, David Wechsler, Eleanor Semel and Isabel Myers

How do you envision the future of testing?
With difficulty.