Developing a Diverse Interests Scale for STEM Learners: Based on the ROSES Survey in Japan Daiki Nakamura University of Miyazaki, Japan E-mail: [email protected] ISTEM-ED 2024 Conference Session 2B

Introduction 2  Researchers need to investigate ways to increase students' interest in STEM through empirical studies using interest scales.  Demand for STEM Occupation • The demand for professionals in Science, Technology, Engineering, and Mathematics (STEM) fields is increasing due to global competition. • Employment projections indicate higher growth rates for STEM occupations compared to overall occupations (U.S. Bureau of Labor Statistics: Zilberman & Ice, 2021). 2019-29 : All other occupations +3.7%, STEM occupations +8.0% • In response, numerous countries have launched extensive campaigns to enhance STEM education  Challenges in STEM Education • Despite extensive campaigns to enhance STEM education, a significant gap remains between the supply and demand for STEM talent (Cedefop, 2018). • Large-scale surveys like TIMSS 2019 show a low proportion of students who highly value STEM fields, highlighting the need to foster interest in STEM (IEA, 2020).

Issues of Traditional Scales 3 1. STEM not integrated • Unfried et al. (2015) : Science, mathematics, and engineering/technology. • Luo et al. (2019) : Science, technology, engineering, and mathematics. • Staus et al. (2020) : Earth/space sciences, life sciences, technology/engineering, and mathematics. • Previous scales cannot measure interest in integrated STEM because they are a collection of items that measure interest in individual disciplines. • Given the nature of integrated STEM education, measuring interest requires the use of topics that cross multiple disciplines. Item 1 Item 2 Item 3 Science Item 7 Item 8 Item 9 Engineering Item 4 Item 5 Item 6 Technology Item 10 Item 11 Item 12 Mathematics Item 1 Item 2 Item 3 Item 1 Item 1 Item 1 Item 1 Item 12 STEM integrate Topics that cross multiple disciplines. e.g., How sound is heard S / T / E / M

Issues of Traditional Scales 4 2. Lack of specificity of items • Abstract items evoke different images among students and prevent comparable and consistent measurement. • For example, when asked about their interest in science, one student might think of biology, while another might think of physics. • In order to provide a comparable, it is necessary to develop an interest scale that includes a variety of topic-specific items. “Are you interested in science?“ concretize “Are you interested in nuclear power plants?“

Purpose and Research Questions 5  The purpose of this study is to develop a scale that measures diverse interests in STEM (DI-STEM) at the topic level. RQ1. Which topics are of interest to students, and which are not? RQ2. What is the factor structure of interest measured in questionnaire? RQ3. What statistical properties does each item exhibit? RQ4. Which items should be included in the DI-STEM scale?

Methods 6  About ROSES Survey • ROSES (The Relevance Of Science Education – Second) is a network of researchers in more than 50 countries which collect data from a common questionnaire about students’ interest, thrust and engagement in relation to STEM. • ROSES measures interest in a variety of topics through a questionnaire.  We used data from the ROSES survey in Japan for our analysis. https://www.miun.se/en/Research/researchgroups/roses/

Sampling Method 7 • The ROSES survey in Japan was conducted with 9th grade (ages 14-15) students. • There are approximately 10,000 junior high schools in Japan, where about 1.07 million 9th graders are enrolled. • We used a stratified cluster sampling method to select 40 schools from 10,247 middle schools across the country. • We obtained a sample that was representative of the population. Stratification by city size

Participants 8 • The survey was conducted between August 2023 and January 2024. • 4,174 ninth graders (ages 14-15) participated in the survey. • After excluding 172 individuals who did not consent to participate and data from respondents who did not follow the instructions, 3,417 responses were analyzed.  IMC (Instructional manipulation check) item: “Please make sure to select “2" for this item for confirmation.”  All the survey procedures were approved by the Research Ethics Review Board of Hiroshima University (HR-ES-000835).

Measurement Items 9 • Participants were asked to rate their interest in the 78 topics on a four-point Likert scale. • The options ranged from 1 (not interested) to 4 (very interested). • Examples of topics: “What I want to learn about. How interested are you in…”  “How different musical instruments produce different sounds” 1・2・3・4  “How a nuclear power plants functions” 1・2・3・4  “How to protect endangered species” 1・2・3・4  These topics can be approached by integrating knowledge from each of the STEM disciplines. • This questionnaire does not measure interest in each of the S-T-E-M domains separately, but rather attempts to illustrate overall interest by showing the diversity of topics that can be addressed in STEM. • For a list of survey items, refer to Jidesjö et al. (2020). https://www.miun.se/en/Research/researchgroups/roses/material/

Data analysis 10 For RQ1. Which topics are of interest to students, and which are not? • Calculate the Mean and SD for each item. • Items with high mean values indicate that the topic is likely to be of interest to students. To increase validity, a wide range of mean topics should be included in the scale.  For RQ2. What is the factor structure of interest measured in ROSES questionnaire? • Conduct explanatory and confirmatory factor analysis. • We hypothesize that STEM-related topics will not only be divided into several factors but will also be grouped into one factor (this is called the higher-order factor model). Overall interest

Data analysis 11  For RQ3. What statistical properties does each item exhibit? • We adopted polytomous IRT model (GRM: Samejeima, 1969), and estimated item parameters using marginal maximum likelihood estimation. • By estimating the item parameters, we identified items with high discrimination for student interests. • Additionally, we checked for differential item functioning (DIF) based on item parameters to ensure that there was no bias between genders.  For RQ4. Which items should be included in the DI-STEM scale? • Finally, considering the results of these three analyses, we constructed a DI-STEM scale and explored its applicability.

Result 1: Differences in Interest by Topic 12  Student interest varied widely by topic. Items Mean SD C4 Why we dream while we are sleeping, and what the dreams may mean 3.450 0.861 C3 Life and death and the human soul 3.351 0.917 A24 How it feels to be weightless in space 3.232 0.973 A13 Black holes, supernovas, and other spectacular objects in outer space 3.191 1.014 C2 The possibility of life outside earth 3.148 1.031 E32 Phenomena that scientists still cannot explain 3.052 1.074 C5 Ghosts and witches, and whether they may exist 3.049 1.096 A5 The origin of life on Earth 3.004 1.006 Table 1. Top 8 Topics by Average Interest (N = 3,417).  Students tend to have a highly interest in topics that appear frequently in science fiction (SF).

Result 1: Differences in Interest by Topic 13  Student interest varied widely by topic. Items Mean SD A8 Birth control 2.308 0.960 E2 The greenhouse effect and how it may be changed by humans 2.311 1.026 A9 Atoms and molecules 2.316 1.033 E20 Plants in my area 2.336 1.056 A35 How a nuclear power plant functions 2.348 1.049 E4 How technology helps us to handle waste, garbage and sewage 2.354 1.038 E21 Detergents and soaps, and how they work 2.368 1.023 E14 Organic and ecological farming without use of pesticides and artificial fertilizers 2.411 1.042 Table 2. Bottom 8 Topics by Average Interest (N = 3,417).  Topics that students have previously learned in school tend to be of low interest.

Result 2: Factor of Interests 14 • We extracted six factors by exploratory factor analysis. • All factors demonstrated high reliability, with Cronbach’s alpha coefficients exceeding 0.8. Factor Number of items Mean SD 𝛼𝛼 Factor correlations � 𝜌𝜌 F2 F3 F4 F5 F6 F1: Environment 19 2.522 0.768 .952 .712 .651 .562 .703 .677 F2: Chemicals 9 2.557 0.799 .913 – .717 .465 .554 .704 F3: Bioprotection 4 2.789 0.815 .819 – – .541 .554 .711 F4: Health 6 2.723 0.809 .872 – – – .589 .416 F5: Sexual 5 2.516 0.817 .853 – – – – .490 F6: Science Fiction 13 3.068 0.701 .902 – – – – – • The higher-order factor model showed good fit and also supported the interpretation of the whole as one general factor. (CFI = .977, TLI = .976, RMSEA =.093, SRMR =.071)

Result 3: Statistical Properties of Items 15  Parameter estimation based on GRM revealed item discrimination and difficulty. Items discrimination diff1 diff2 diff3 E4 How technology helps us to handle waste, garbage, and sewage 2.770 -0.682 0.212 1.016 E16 Renewable sources of energy from the sun and the wind 2.616 -0.866 -0.032 0.802 E2 The greenhouse effect and how it may be changed by humans 2.614 -0.680 0.262 1.086 E25 Benefits and possible hazards of gene modification (GMO) in farming 2.459 -0.797 0.109 0.966 E14 Organic and ecological farming without use of pesticides and artificial fertilizers 2.442 -0.782 0.132 1.022 • Items related to environmental issues tended to be highly discrimination. • These items with high discrimination should be included in the scale.

Result 3: Differential Item Functioning (DIF) 16  Some items demonstrated DIF due to gender differences . • These items show different levels of difficulty depending on gender. • To achieve gender equality measurement, these items were excluded from the scale. A21. Explosive chemicals A28. The ability of lotions and creams to keep the skin young

Result 4: Item selection and scale composition 17 • We selected 50 items, considering the diversity of means, factor structure, item discrimination, and DIF. • The DI-STEM scale has high reliability and validity for students with a wide range of interest levels. • The figure below shows how the reliability (rxx ) of the measurement varies for each level of the individual characteristic value of interest (θ). • This implies that accurate measurements can be obtained for students of any interest level. reliability

Discussion 18  Development of DI-STEM Scale • To address the need for a more specific and comprehensive measure of student interest in STEM, we developed the DI-STEM scale. • The validity of the scale was confirmed from multiple angles based on the analysis of the means, factor structure, reliability, item discrimination, and DIF.  STEM education based on interest in the topic • Our findings highlight the challenge of differences in difficulty between topics in engaging students in STEM, a concern echoed in the TIMSS 2019 survey. • The DI-STEM scale is useful as a valuable tool for educators to identify areas of high and low interest, helping to provide STEM activities that are responsive to students' interests.

How can the DI-STEM scale be used? 19 1. To test the effectiveness of STEM interventions. • The DI-STEM scale can accurately assess the effectiveness of interventions. • It is possible to compare the use of different items in pretests and posttests because item characteristic values are estimated based on item response theory. • Teachers can identify overall classroom trends by calculating scores at the factor level. 2. STEM activities personalized to learner interests • The diversity of interests captured by the DI-STEM scale highlights the potential for personalized learning paths in STEM education. • Providing STEM activities that use topics of high interest to students will increase their enthusiasm for the activities • Also important to increase interest in topics through STEM activities using topics of low student interest

Conclusion and Challenges for future research 20  Conclusion of this study • In conclusion, the development of the DI-STEM scale marks a significant step forward in understanding and measuring students’ interest in STEM. • This scale can contribute significantly to the design of more relevant and compelling STEM education programs by providing a tool for assessing a wide range of topics that engage students.  Challenges for future research • It is essential to recognize the limitations of this study, such as the dependence of the DI- STEM scale on specific demographic and cultural contexts. • Future research should explore the applicability and reliability of the DI-STEM across different cultures and educational settings. • Furthermore, DI-STEM indicators will be integrated with adaptive learning techniques to create interest-driven learning.

Reference 21 • Bybee R. W. (2013). The case for STEM education: Challenges and opportunities. Arlington VA: NSTA Press. • Cedefop (2018). Insights into skill shortages and skill mismatch: learning from Cedefop’s European skills and jobs survey. Luxembourg: Publications Office. Cedefop reference series; No 106. http://data.europa.eu/doi/10.2801/645011 • Comer, M., Sneider, C., & Vasquez, J. A. (2013). STEM lesson essentials, grades 3-8: integrating science, technology, engineering, and mathematics. Portsmouth, NH: Heinemann. • DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171-181. https://doi.org/10.1002/job.1962 • International Association for the Evaluation of Educational Achievement [IEA] (2020). TIMSS 2019 International Results in Mathematics and Science. Chestnut Hill, MA: Boston College. • Jidesjö, A., Oskarsson, M., & Westman, A-K. (2020). ROSES Handbook: Introduction, guidelines and underlying ideas. Utbildningsvetenskapliga. https://www.miun.se/en/Research/researchgroups/roses/material/ • Luo, T., Wang, J., Liu, X., & Zhou, J. (2019). Development and application of a scale to measure students’ STEM continuing motivation. International Journal of Science Education, 41(14), 1885-1904. https://doi.org/10.1080/09500693.2019.1647472 • Matsuura, T., & Nakamura, D. (2021). Trends in STEM/STEAM education and students’ perceptions in Japan. Asia-Pacific Science Education, 7(1), 7-33. https://doi.org/10.1163/23641177-bja10022 • Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34(4, Pt. 2), 100. • Schreiner, C., & Sjøberg, S. (2004). Sowing the seeds of ROSE. Background, rationale, questionnaire development, and data collection for ROSE (the Relevance of Science Education) – A comparative study of students’ views of science and science education. Acta Didactica, 4/2004. Oslo: Department of Teacher Education and School Development, University of Oslo. • Schreiner, C., & Sjøberg, S. (2019). ROSE (The Relevance of Science Education) Final Report part 2. Western youth and science. • Staus, N. L., Lesseig, K., Lamb, R., Falk, J., & Dierking, L. (2020). Validation of a measure of STEM interest for adolescents. International Journal of Science and Mathematics Education, 18, 279-293. https://doi.org/10.1007/s10763-019-09970-7. • Unfried, A., Faber, M., Stanhope, D. S., & Wiebe, E. (2015). The development and validation of a measure of student attitudes toward science, technology, engineering, and math (S-STEM). Journal of Psychoeducational Assessment, 33(7), 622-639. https://doi.org/10.1177/0734282915571160 • Zilberman, A., & Ice, L. (2021). Why computer occupations are behind strong STEM employment growth in the 2019–29 decade. Computer, 4(5,164.6), 11-5.

STEM activities personalized to learner interests 22  Using ChatGPT to suggest STEM activities tailored to learners' interests https://chatgpt.com/share/13acbb6e-4c4e-41a1-b44d-d9819eeb5c3c You are a teacher. Your goal is to suggest STEM activities that meet the interests of your students. Attached is a record of one student's questionnaire and 4-point Likert scale responses. Based on this student's interest, please think of a STEM activity and suggest a specific one.

Background Information on Japanese Learners 23 • Japanese students show high achievement in international surveys, but their interest in STEM is lower than the international average.  PISA 2022

24

25