Development and Validation of an Assessment of Engineering Ph.D. Students’ Research Experiences

Global concerns about the preparedness of engineering Ph.D. students for professional practice are not new. In the U.S., educational reform has focused on the research experiences of students to foster better preparation. Yet, little is known about which aspects of students’ research experiences are essential to prepare them for practice due to the heterogeneity of the experiences, and what opportunities they have in their research to practice being a professional. The goal of this study was to develop and initially validate an instrument that measures students’ perceptions of their research experiences utilizing an ontological theoretical framework that focuses on what it means to become a professional. This framework simplified the heterogeneity and allowed for the investigation of how the research experiences of engineering Ph.D. students are providing opportunities for students to practice being a professional. Four distinct phases of development were utilized to accumulate validity evidence for the instrument: a development phase that focused on question generation and review: an initial pilot test that centered on an Exploratory Factor Analysis on responses (n = 236) from a large Midwestern University; a second pilot test that centered on a Confirmatory Factor Analysis on responses (n = 215) from multiple universities; and a Group Analysis phase that tested statistical differences between groups. Three key results emanated from this work. First, the accumulated validity evidence justifies the intended use of the instrument as a research and program evaluation survey to assess engineering Ph.D. students’ research experiences for opportunities to practice being a professional. Second, the results suggest that, on average, students had fewer opportunities to work with professionals (i.e., take on others’ forms of practice) in their research experiences than other types of opportunities. Third, the results suggest that research experiences can be categorized into those that provide significantly more and significantly fewer opportunities for students to practice being a professional. Higher education tends to focus on the epistemological aspects of professional practice preparation, but utilizing an ontological approach can identify gaps in preparation. Implications of the opportunities identified in this study are discussed for faculty, students, other researchers, instrument users, engineering administrators, and national program administrators, with a focus on providing more opportunities to students to practice being a professional. The utilization of an ontological approach for engineering Ph.D. students’ research experiences, including tangible examples and a call for a new vision for U.S. engineering Ph.D. research experiences, are discussed.