Measurement Working Group

Overview

 

Resources

 

This page is all about Working Group 1: Measurement! Keep reading for resources, leadership, projects, and community highlights.

Focus: Establish metrics for measuring equity and inclusion in foundational STEM courses, conduct the measurements, and identify actionable data to promote change.

Leadership

 

 

Co-chair

Stefano Fiorini

sfiorini@indiana.edu

 

Co-chair

Becky Matz

rlmatz@umich.edu

Key Projects

 

 

 

CLICK TO VIEW PROJECT POSTERS

Project Leads

Project Description

This project will conduct parallel data analyses across the 10 SEISMIC institutions in different introductory STEM courses. It seeks to explore the components of student identity - those we can capture, those we cannot, and how they interact with the learning environments typical to intro STEM. Participants will replicate the LARA analyses, exploring equity in lecture versus lab classes and comparing STEM to non-STEM courses at institutions that were not part of the initial LARA project. This project will then extend past the gender analysis done in LARA to also explore how ethnicity, first-generation status, and the interactions of these identities affect student outcomes. To complete this project, the team will need to regularize data across the institutions, determine which courses or course sequences to include in the analyses, clarify the motivations of the project and what they hope to find, and develop and apply analytical techniques designed to elucidate the complex interplay between identity and the learning environment.

Project Leads

Project Description

This project seeks to understand how people fall out of STEM and if that is connected to failing foundational STEM courses. To explore this, this team will work to answer two main questions using first quantitative data analyses and then by studying student motivation. The first question is, what happens to STEM students who fail or earn a D in an intro class? This question gets at where the students go after failing, and what causes them to make those decisions. For example, some students may repeat the class, others may switch majors or even leave the university. The second question asks, what happens to students who repeat the class? To answer these questions, this project will also determine if the paths students take after failing a foundational STEM course differ by demographic groups. After determining the paths that are present for students, this team will then examine why these paths arise in terms of motivation - for example, imposter syndrome may influence minority-group more than majority-group students. To complete the first part of this project, the team will need to establish choose a course and major to start with, determine course equivalencies between universities, regularize their data, receive IRB and FERPA approval, and conduct and interpret the initial results on pathways. Once those paths are understood, the group will work with motivation experts to develop instruments to understand what drives students into the different paths.

Project Leads

Project Description

This team will use a mixed methods approach to analyze the impact of classroom composition with respect to identity groups, and determine how it relates to student outcomes such as course grades and persistence in the major. The primary research question is, does demographic composition of classes at various hierarchical levels affect student success and/or student attitudes? To complete this project, they will need to regularize their institutional data and gather additional data regarding student motivation, self-efficacy, sense of belonging, and other affective factors through surveys or focus groups. This additional data will help the team to understand how these affective elements interact with student outcomes. This team hopes to inform the higher education community on strategies for promoting a sense of inclusion in the classroom and to raise awareness on this topic of classroom composition and its impacts on students. Ultimately this project will inform course-level, college-level, and university-level policies.

Project Leads

Project Description

This project aims to inform university and department policies around accepting AP credit. University policies on AP credit have real consequences for students, as the rejection of this credit could lead to students having to retake courses and lose motivation, particularly if they do poorly in the challenging learning environment of a large enrollment course. Applied across multiple AP courses, students will then take longer to graduate and thus have to pay more for school. The negative consequences may be especially large for students from lower income households and coming from high schools that did less AP exam preparation work. Considering that credit-granting policies vary widely across departments and across institutions, this team seeks to help institutions make evidence-based decisions on when to accept AP credit, if at all. To accomplish this goal, the team will look across the SEISMIC institutions and evaluate the consequences of accepting AP credit, both positive and negative, for students by different demographics.

Project Leads

Project Description

Active Learning has been linked to improved student performance in STEM (Freeman et al., 2014), and has become popular as a shorthand for instruction, activities and classroom environments that move away from lecture-based methods and towards a student-centered approach. However, there is still wide variation in what we mean when we say our introductory STEM courses provide Active Learning experiences for our students. Further, while most research focuses on course performance as an outcome, Active Learning contains many components that could also impact students’ persistence independently of their effects on grades (Ballen, et al., 2017). For example, placing an emphasis on communal values (Jackson, et al., 2016) and improving a sense of belonging (Walton, et al., 2015) have been shown to improve persistence decisions for minoritzed racial groups and women in STEM. Currently, few Active Learning observation protocols include measures of such components. This project aims to 1) understand the Active Learning landscape across SEISMIC institutions 2) identify gaps in current measures of Active Learning, and 3) develop measures and observation protocols that focus on components of Active Learning classrooms that are related to persistence, particularly for minoritized groups in STEM.

Project Leads

Project Description

Despite decades of efforts to increase the diversity of the STEM workforce, women and professionals of diverse ethnicities and backgrounds remain woefully underrepresented. A variety of interventions and instructional changes are currently being implemented in large lower division science courses with the goal of reducing equity performance gaps and thus promote retention and graduation of students traditionally underrepresented in STEM (URMs). In contrast to the abundant literature on disparities in STEM students in introductory courses, much less is known about the disparities that are present in upper division science courses and their causes. While the majority of students who drop out of STEM do so during the first two years of college, STEM attrition occurs throughout the whole major. Furthermore, the existing disparities result in URM students graduating with lower GPAs and taking longer to graduate, factors which reduce their competitiveness in the job and academic market, perpetuating inequities in the workforce. Furthermore, while the DBER field is starting to recognize the importance of intersectional approaches to studying equity gaps, this approach remains largely underutilized. This workgroup will complement the work currently done by WG1P1 and WG1P3 by characterizing equity gaps in large upper division courses in STEM using a variety of quantitative methods.

Project Leads

Project Description

Almost every student experiences at least one large lecture course during their undergraduate experience. This project is focused on the impacts of those large lecture courses – specifically whether they choose to go on in later courses in that department, how well they do in those later courses, and whether they stay at the university. Work this summer will be geared towards analyzing the data sets available and exploring possible predictors and identifying relevant student groups.

Publications

 

  • Whitcomb, Kyle, Sonja Cwik, and Chandralekha Singh. 2021. “Not All Disadvantages Are Equal: Racial/Ethnic Minority Students Have Largest Disadvantage Among Demographic Groups in Both STEM and Non-STEM GPA.” AERA Open 7 (November):233285842110598. https://doi.org/10.1177/23328584211059823.
  • Castle, Sarah. 2021. “Equity in the STEM Landscape: A Multi-Institutional Approach to Mapping Systemic Advantages Within STEM Courses.” In Proceedings of the 2021 AERA Annual Meeting. AERA. https://doi.org/10.3102/1689325.
  • Fischer, Christian, Eben Witherspoon, Ha Nguyen, Yanan Feng, Stefano Fiorini, Paulette Vincent-Ruz, Chris Mead, William Nicholas Bork Rodriguez, Rebecca L. Matz, and Christian Schunn. 2022. “Advanced Placement Course Credit and Undergraduate Student Success in Gateway Science Courses.” Journal of Research in Science Teaching 60 (2): 304–29. https://doi.org/10.1002/tea.21799.
  • Pearson, Meaghan I., Sarah D. Castle, Rebecca L. Matz, Benjamin P. Koester, and W. Carson Byrd. 2022. “Integrating Critical Approaches into Quantitative STEM Equity Work.” CBE—Life Sciences Education 21 (1): es1. https://doi.org/10.1187/cbe.21-06-0158.
  • Fiorini, Stefano, Nita Tarchinski, Meaghan Pearson, Montserrat Valdivia Medinaceli, Rebecca L. Matz, Juniar Lucien, Hye Rin Lee, et al. 2023. “Major Curricula as Structures for Disciplinary Acculturation That Contribute to Student Minoritization.” Frontiers in Education 8 (August). https://doi.org/10.3389/feduc.2023.1176876.
  • Farrar, Victoria S., Bianca-Yesenia Cruz Aguayo, and Natalia Caporale. 2023. “Gendered Performance Gaps in an Upper-Division Biology Course: Academic, Demographic, Environmental, and Affective Factors.” CBE—Life Sciences Education 22 (4): ar52. https://doi.org/10.1187/cbe.23-03-0041.
  • Fiorini, Stefano, Valdivia Medinaceli, M. B., Matz, R. L., Pearson, M., Tarchinski, N., Tong, X., Castle, S., Sylvester, A., Denaro, K., & Farrar, V. (2023). Analytics revealing systemic inequities at different scales. Companion Proceedings of the 13th International Learning Analytics and Knowledge Conference (LAK23), March 13-17, Arlington, TX.
  • Castle, Sarah D., W. Carson Byrd, Benjamin P. Koester, Meaghan I. Pearson, Emily Bonem, Natalia Caporale, Sonja Cwik, et al. 2024. “Systemic Advantage Has a Meaningful Relationship with Grade Outcomes in Students’ Early STEM Courses at Six Research Universities.” International Journal of STEM Education 11 (1): 14. https://doi.org/10.1186/s40594-024-00474-7. 
  • Denaro, Kameryn, Kimberly Dennin, Michael Dennin, and Brian Sato. 2022. “Identifying Systemic Inequity in Higher Education and Opportunities for Improvement.” PLOS ONE 17 (4): e0264059. https://doi.org/10.1371/journal.pone.0264059.

Additional Activities

Community Highlights

 

  • Conference Papers
    • Castle, S. D., Byrd, W. C., Koester, B. P., Bonem, E. M., Caporale, N., Cwik, S., Denaro, K., Fiorini, S., Matz, R. L., Mead, C., Whitcomb, K. M., Singh, C., Levesque-Bristol, C., & McKay, T. A. (2021). Equity in the STEM Landscape: A Multi-institutional Approach to Mapping Systemic Advantages Within STEM Courses. American Educational Research Association (AERA) Annual Meeting, April 9-12, virtual. https://doi.org/10.3102/1689325
    • Fiorini, S., Valdivia Medinaceli, M. B., Matz, R. L., Pearson, M., Tarchinski, N., Tong, X., Castle, S., Sylvester, A., Denaro, K., & Farrar, V. (2023). Analytics Revealing Systemic Inequities at Different Scales. Companion Proceedings of the 13th International Learning Analytics and Knowledge Conference (LAK23), March 13-17, Arlington, TX. https://www.solaresearch.org/wp-content/uploads/2023/03/LAK23_CompanionProceedings.pdf
  • Presentations (external)  
    • Caporale, N., Castle, S. D., Fiorini, S., Fischer, C., & Matz, R. L. (2021, May 14). Analytics to Support Student Success in STEM: Stories from the Sloan Equity and Inclusion in STEM Introductory Courses (SEISMIC) Measurement Working Group. Indiana University’s 3rd Annual International Learning Analytics Summit, virtual. 
    • Castle, S. D., Byrd, W. C., Koester, B. P., Bonem, E. M., Caporale, N., Cwik, S., Denaro, K., Fiorini, S., Matz, R. L., Mead, C., Whitcomb, K. M., Singh, C., Levesque-Bristol, C., & McKay, T. A. (2021, April 12). Equity in the STEM Landscape: A Multi-institutional Approach to Mapping Systemic Advantages Within STEM Courses. American Educational Research Association (AERA) Annual Meeting, virtual.
    • Farrar, V. S., Valdivia Medinaceli, M. B., Young, N. T., Bonem, E. M., Fiorini, S., Koester, B. P., Matz, R. L., & Caporale, N. (2023, January 14). Equity Gaps Associated with Student Demographics Persist into Upper-division Biology Courses Across Multiple Institutions. Society for the Advancement of Biology Education Research (SABER) West National Meeting, Irvine, CA.
    • Fiorini, S., Denaro, K., Lee, H. R., Matz R. L., & Koester, B. (2022, June 8). Cross-institutional Comparison of Students’ Progression to a STEM Degree. Association for Institutional Research (AIR) Forum 2022, Phoenix, AZ.
    • Fiorini, S., & Molinaro, M. (2021, May 14). Opportunities at Origin and Access to Higher Education: Revealing New Dimensions of Exclusions. Indiana University’s 3rd Annual International Learning Analytics Summit, virtual. 
    • Sweeder, R. D. (2022, May). Exposing Inequity: A Multi-institutional Analysis of Systematic Advantages in Introductory Stem Courses. Lawrence Technical Institute Howard Hughes Medical Institute (HHMI) Inclusive Excellence Course‐based Research Experience Retreat.
    • Sweeder, R. D., Castle, S. D., Koester, B. P., Byrd, W. C., Pearson, M. I., Bonem, E. M., Caporale, N., Cwik, S., Denaro, K., Fiorini, S., Levesque-Bristol, C., Matz, R. L., Mead, C., Brownell, S., Molinaro, M., Singh, C., & McKay, T. A. (2022, March 20-24). Exposing Inequity: A Multi-Institutional Analysis of Systematic Advantages in Introductory STEM Courses. American Chemical Society (ACS) National Meeting, San Diego, CA.
    • Vincent-Ruz, P., Fischer, C., Schunn, C., & Lonn, S. (2022, February). Are the Revised AP STEM Courses Equitable and Equivalent to University Courses? American Association for the Advancement of Science (AAAS) Annual Meeting, Philadelphia, PA.
  • Presentations (internal)  
    • Castle, S., Denaro, K., Farrar, V. Fiorini, S., Matz, R. L., Pearson, M., Sylvester, A., Tarchinski, N., Tong, X., & Valdivia Medinaceli, M. B. (2022, September 27). Measurement Group Presentations. University of Minnesota Week of SEISMIC, virtual.
    • Castle, S., Denaro, K., Farrar, V. Fiorini, S., Matz, R. L., Pearson, M., Sylvester, A., Tarchinski, N., Tong, X., & Valdivia Medinaceli, M. B. (2022, October 18). Measurement Group Presentations. Indiana University Week of SEISMIC, virtual.
    • Fiorini, S., Denaro, K., Lee, H. R., Matz R. L., & Koester, B. (2022, June 13-17). Cross-institutional Comparison of Curricula Structures: A Presentation on the Construction of Sameness, Minoritization and the Missing Opportunity of the Empowerment of Difference. SEISMIC Summer Meeting, virtual. 
  • Grant proposals (under review)  
    • National Science Foundation, Directorate for STEM Education, Division of Undergraduate Education (DUE), Improving Undergraduate STEM Education (IUSE) Engaged Student Learning Level 2 grant (#2336762, 2336763 & 2336764, $749,985, 2024-2027, PIs: Natalia Caporale (UCD), Jennifer H. Doherty (MSU), Ryan D. Sweeder (MSU), Rebecca L. Matz (UM)). Collaborative Research: Characterizing and Addressing Systemic Gendered Disparities in Upper-division Biology Courses Across Six Large, Public, Research Universities.
    • National Science Foundation, Directorate for STEM Education, Division of Undergraduate Education (DUE), Improving Undergraduate STEM Education (IUSE) Institutional and Community Transformation Level 2 grant (PIs: Kameryn Denaro (UCD), Stefano Fiorini (IUB), Martha Oakley (IUB), Brian Sato (UCI), Chris Mead (ASU), Marco Molinaro (Maryland), Timothy McKay (UM), Ryan Sweeder (MSU), Katerina Thompson (Maryland), Emily Bonem (Purdue)). Collaborative Research: Bright Spots: A Multi-institution Collaboration to Identify Faculty and Institutional Characteristics for Promoting Equitable Stem Student Success.