Validating Jane AI Competency System In Critical Thinking Assessments

This blog post is the first of two that excerpt a HealthStream article, “The Validity of the JaneTM Competency System AI Critical Thinking Assessments,” by Randy L. Carden, Statistical Consultant, HealthStream.

The development of critical thinking/judgment skills by nurses is of paramount importance in the healthcare industry today. Factors, such as the following, have all converged to make the development of advanced critical judgment skills a top priority:

  • A growing senior population requiring nursing care
  • A high percentage of seasoned nurses taking retirement
  • Nursing shortages in many areas of the country
  • Increased patient acuity in many settings
  • The need to bring new nurses up-to-speed as quickly as possible

It has long been thought that these types of skills could only be developed through years of on-the-job training and experience. Now, however, we are finding that artificial intelligence (AI) can play a major role in providing efficient, comprehensive tools for enhancing critical judgment.

The paper excerpted by this blog post details psychometric studies conducted by HealthStream to evaluate whether computers can perform as well as human evaluators in assessing critical thinking skills in nurses.


Once all assessment completions were obtained, all data was sent to Perception Health, an independent analytics firm, where an analysis was conducted to determine the relationship between the Jane™ scores and human scoring via a trained RN using “model answers” established by PBDS.

The Pearson Correlation Coefficient was used to evaluate the relationship. The results of the statistical analysis found a strong, positive correlation between the Jane™ scores and the human RN rating scores, r = 0.827. The Pearson Correlation Coefficient ranges from negative one to positive one. A correlation coefficient near zero indicates no relationship between the variables. A correlation coefficient of 0.20 - 0.30 (either positive or negative) is considered a weak correlation. A correlation coefficient between 0.40 – 0.60 (either positive or negative) is indicative of a moderate relationship. A correlation coefficient of 0.80 or higher (either positive or negative) demonstrates that the two variables are strongly related. Thus, the results of the study indicated that there was a significant, strong positive relationship between Jane’s™ evaluative scores and the human (RN) ratings.


The purpose of the study was to investigate the validity of Jane™ as an evaluative tool to assess the critical thinking/ judgment of nurses. Twenty-eight complete evaluation sets were used in the study. Jane™ scores were compared to the scores of a trained RN rater. It was found that there was a strong relationship between Jane™ ratings and human ratings. Thus, the construct validity of Jane™ has been established by this study.

The article from which this blog post was taken includes the following additional information and findings about our validation of  janeTM:

  • Background
  • Purpose of the Study
  • How the Study Was Conducted
  • Participants
  • Validity and Reliability of PBDS
  • Content Validity
  • Construct Validity
  • Predictive Validity
  • Reliability

HealthStream Focuses on Clinical Development

At HealthStream we spend a lot of time focused on developing the clinical workforce. HealthStream’s jane™ is The World’s First Digital Mentor for Nurses. Jane harnesses the power of artificial intelligence (AI) to create a system that personalizes competency development at scale, quickly identifies risk and opportunity, and improves quality outcomes by focusing on critical thinking. Leveraging decades of research and with over 4 million assessments completed, Jane was designed to power lifelong, professional growth of clinical professionals. JaneTM is an important component of HealthStream’s suite of clinical development solutions.

Download the full article, “The Validity of the janeTM Competency System AI Critical Thinking Assessments,” in which we investigate the assessments on which janeTM is built.

HealthStream Brands