two people are engaged in discussion during an AMIGA event at UC Davis. A third person is in the foreground with their head turned away from the camera

Holistic Review Gateway: A UC-Wide Collaboration

What is "holistic review?"

A holistic, or “whole file” review process in graduate admissions considers every aspect of an applicant’s file equally, without any one factor deciding admission. Holistic review also entails understanding an applicant’s achievements in the context of available opportunities and challenges faced. Holistic review employs a systematic, comprehensive, and contextualized approach to graduate admissions.  It can help a program identify talent, avoid bias, and ensure inclusive excellence in the way it trains the next generation of scholars. Thus, holistic review serves the mission of the University of California to advance knowledge through the contributions of the diverse population of our state.

What’s involved?

Typically, departments that use a holistic process for graduate admissions place less emphasis on standardized scores and GPAs and attend more to socio-emotional variables like persistence and creativity. They use rubrics to systematize evaluation across reviewers. They keep clear, written records and revisit their processes annually, consulting application, admit, and yield data to assess effectiveness. There is no one-size-fits all approach; each program has unique needs; thus, holistic review looks different in different fields.

How do we do it?

Below is an outline of best practices organized according to a typical calendar of admissions along with suggestions for further reading.  Ideally your admissions team could get to everything on this timeline, but the world is less than ideal and not everything here will resonate with your colleagues. If you’re just getting started, the best place to begin is with a department-wide conversation about the place of graduate admissions in your program and field as a whole, what counts as success in your field, what qualities ensure graduate student success, if your process currently looks for those things in an applicant’s file, and if not, what changes to your process you might consider. The findings from these conversations can inform the admission committee’s deeper work.

Admissions Timeline

  • Summer — Review and Revise
  • Whether you are a continuing Admissions Chair or have stepped into the role, as you plan for a new academic cycle, revisit the notes from your springtime review of the past year’s successes and frustrations. You may have decided that your current application did not provide all of the information that the admissions committee needed, and so you want to consult with the Graduate Studies/Graduate Division’s Director of Admissions to see if changes are possible. You may have liked the committee’s process, but decided the workload was too heavy, and so want to recruit more members. You may have been disappointed with the depth of the applicant pool, and so want to try more outreach this year. One simple form of outreach is to provide clear and specific information on your program website.  Prospective students may rely on your webpage to understand how to navigate the application process.  It is helpful to spell out information about the program, such as how students match with a research mentor, or what your admissions committee is looking for in a statement of purpose or writing sample.  You can enhance the page by hosting and posting an informational webinar about the program or the application process.
  • Fall — Assemble and Train a Diverse Team
  • As you assemble your admissions committee, think about ways you can bring together diverse perspectives for the work, and look for a balance of faculty ranks, backgrounds, and fields. Ideally, identify the committee prior to or at the very beginning of the fall term. Consider whether including advanced graduate students on the committee can offer a valuable student view of the program and a professionalizing opportunity. Ideally, you will be able to meet a few times before the actual reviewing begins, first to discuss program goals and the annual review of admissions and retention data and second to come to a shared understanding of equitable review. One important tool for equitable review is the rubric, a scoring template that can help guide consistent evaluation of applicants across many reviewers, but they are useful only if they are the result of faculty collaboration and discussion. You may be revising last year’s rubric or creating a new one, and after you have it ready, take some time on norming conversations in which reviewers arrive at common ground when it comes to criteria for assessment, how to find evidence of these criteria in an application, and what form of scoring to use.

    One of the topics to discuss with your team is implicit bias and how it appears in many     aspects of graduate admissions. Conversation starters could include a presentation from the campus office of Diversity, Equity, and Inclusion or asking committee members to discuss the legal context that shapes admissions decisions, including California’s Proposition 209, which prohibits universities from discriminating or “granting preferential treatment” to individuals on the basis of race, sex, color, ethnicity or national origin. I is legal to discuss race, ethnicity, and gender data, but these attributes cannot be a consideration for admission. However, characteristics such as contribution to diversity or first generation college can be considered. Scholars who study equitable and inclusive methods of graduate admissions suggest including a diversity measure in their rubric or evaluation.
  • Winter  — Admissions and Recruitment
  • Following admissions preparation, reviewers apply their graduate program’s holistic graduate admissions methods.

    Depending upon the number of applications, assign files for review to each committee member. If the number of files is very large, there are some options to consider.

    -- Best practices recommend that a first review involve a minimum of two reviewers for each file, however, workload can be an issue. Some graduate programs address large numbers of graduate applications by involving considerably more reviewers (e.g. more faculty reviewers), thereby reducing the number of files to review by each person.

    -- A second option to consider is including advanced graduate students as reviewers too.  Although there may be some concerns (e.g. confidentiality and training) they can  certainly be overcome. Including advanced graduate students in admissions can reduce the number of files assigned to each reviewer and is a professional development opportunity demonstrating academic service and inclusivity.

    Some graduate programs may have a separate meeting to discuss and decide upon financial support for each admitted student. This is a good time to dialogue about multiple year offers, summer funding and other considerations that provide a competitive financial offer for your admittees.

    Once admissions and financial decisions are made, committee members and a larger faculty group may be called upon to recruit your future graduate students. Scholarship suggests that not only are admissions and financial support is critical to their acceptance, but so are faculty outreach and graduate program climate. Please see some of these resources and recommended reading for additional information.
  • Spring — Assess Results
  • Once April 15 has passed and you’ve yielded your cohort, set aside some time for reflection.  First, make a detailed record of the admissions and recruitment processes your committee followed this cycle.  Then, review your application, admission, and SIR data noting your department’s accomplishments and growth opportunities. Are you attracting a diverse applicant pool and admitting strong, well rounded cohorts? Are you able to yield your top candidates? Share your findings with your committee and have a frank discussion about their experiences evaluating files and recruiting candidates this year. What worked, what didn’t, and what more can be done, or what can be done differently, next year to build even stronger, more inclusive cohorts? Take note of committee members’ comments and recommendations, add these to your account of this year’s process, and hand these records off to next year’s admissions chair, or – if you’re next year’s chair – retain them for further review. Your notes enable the smooth transitioning and onboarding of committee leadership and members.

  •  

Additional Resources & Readings

  • Summer - Further Resources and Readings

  • Readings
    Bersola, S. H., Stolzenberg, E. B., Love, J., & Fosnacht, K. (2014). Understanding Admitted Doctoral Students’ Institutional Choices: Student Experiences Versus Faculty and Staff Perceptions. American Journal of Education, 120(4), 515–543. https://doi.org/10.1086/676923

    Griffin, K. A., & Muniz, M. (2015). Rethinking the Structure of Student Recruitment and Efforts to Increase Racial and Ethnic Diversity in Doctoral Education. International Journal of Doctoral Studies, 10, pp. 199–216. doi: 10.28945/2275

    Michel, R. S, Belur, V., Naemi, B & Kell H. J.  (2019). Graduate Admissions Practices: A Targeted Review of the Literature.  ETS Research Report, No. II-19-33.

    Villanueva, C. & Moreno, J. M (July 1, 2022). Equitable and Inclusive Graduate Program Websites. Available from mjmoreno@ucdavis.edu.

  • Fall - Further Resources and Readings

  • Resources  
    Test your implicit biases at Project Implicit: https://implicit.harvard.edu/implicit/  

    Office of General Counsel (July 2015). Guidelines For Addressing Race and Gender Equity in Academic Programs in Compliance with Proposition 209. University of California. https://www.ucop.edu/uc-legal/_files/guidelines-equity.pdf.

    Readings
    Holmes, C., & Oakleaf, M. (2013). The Official (and Unofficial) Rules for Norming Rubrics Successfully. The Journal of Academic Librarianship, 39(6), 599-602. doi:10.1016/j.acalib.2013.09.0

    Mathur, A., Cano, A., Dickson, M. W., Matherly, L. H., Maun, C., & Neale, A. V. (2019). Portfolio Review of Graduate Admissions: Outcomes of a Pilot Program. Strategic Enrollment Management Quarterly, 7(1), 7-24. https://www.utsa.edu/strategicplan/initiatives/academic/graduate-success/documents/SEMQ-0701-Portfolio-Review.pdf

    Moneta-Koehler, L., Brown, A. M., Petrie, K. A., Evans, B. E., & Chalkley, R. (January 11, 2017).  The Limitations of the GRE in Predicting Success in Biomedical Graduate School. PLoS ONE 12(1):e0166742. doi:10.1371/journal.pone.0166742.

    Posselt, J. R. (2014). “Toward Inclusive Excellence in Graduate Education: Constructing Merit and Diversity in PhD Admissions.” American Journal of Education, 20,(4), pp. 481–514. JSTOR, https://www.jstor.org/stable/10.1086/676910#metadata_info_tab_contents

    Posselt, J. R. & Miller, C. W. (n.d.). Rubric Design and Use. NSF funded.

    Schoepp, K., Danaher, M., & Kranov, A. A. (2018). An Effective Rubric Norming Process. Practical Assessment, Research & Evaluation, 23(11).

  • Winter - Further Resources and Readings

  • Readings
    Bersola, S. H., Stolzenberg, E. B., Love, J., & Fosnacht, K. (2014). Understanding Admitted Doctoral Students’ Institutional Choices: Student Experiences Versus Faculty and Staff Perceptions. American Journal of Education, 120(4), 515–543. https://doi.org/10.1086/676923

    Toor, Rachel (February 11, 2022). Lessons from a Career Spent Recruiting Graduate Students of Color.  The Chronicle of Higher Education. https://www.chronicle.com/article/lessons-from-a-career-spent-recruiting-graduate-students-of-color

  • Spring - Further Resources and Readings

  • Resources
    UC Grad Studies Stats Tool, https://grad.ucdavis.edu/diversity-data-and-research

  • More Resources and Readings

  • Affirmative Action and California's Proposition 209
    Bleemer, Z. (2022). Affirmative Action, Mismatch, and Economic Mobility After California’s Proposition 209. Quarterly Journal of Economics 137(1), pp.115–160.

    Diversity in Higher Education
    Barajas, H., & Musil, C.M. (2021). Diversity: An Investment in Democracy and Academic Excellence in Re-envisioning the Public Research University: Navigating Competing Demands in an Era of Rapid Change. Eds. Furco, A, Bruininks, R.H., Jones, R.J., and Kent, K. Routledge: New York & London.

    Bleemer, Zachary (July 1, 2019).  Diversity in University Admissions: Affirmative Action, Percent Plans and Holistic Review.  UC Berkeley Research and Occasional Papers Series. CSHE.6.2019.

    Hurtado, S. (Winter 2007). Ashe Presidential Address. Linking Diversity With The Educational and Civic Missions of Higher Education. The Review of Higher Education, 30(2), pp. 185-196.

    Holtzman, S., Minott, T., Devasia, N., Dessi, K., Kirova, D., & Klieger, D. (December 2021).  Exploring Diversity in Graduate and Professional School Applications. ETS Research Report -21-17.

    Grade Point Average
    Pattison, E., Grodsky, E., & Muller, C.  (2013). Is the Sky Falling? Grade Inflation and The Signaling Power of Grades.  Educational Researcher, 42(5), pp. 259-265. DOI: 10.3102/0013189X13481382.

    Holistic Admissions
    Glazer, G., & Bankston, K. (September, 2014). Holistic Admissions in The Health Professions: Findings From a National Survey. Urban Universities for HEALTH, pp. 1-24.

    Kent, J. D. & McCarthy, M. T. (2016). Holistic Review in Graduate Admissions: A Report from the Council of Graduate Schools. Washington, DC: Council of Graduate Schools.

    Kira Talent. (2018). Demystifying Holistic Admissions: The Comprehensive Guide to Holistic Review.  www.kiratalent.com.

    Liu, L. O. (2022). Holistic Admissions in Higher Education: Challenges and Promises. Journal of Postsecondary Student Success, 1(4), pp. 1-19.

    Michel, R. S., Vinetha B., Bobby N. & Harrison J. K. (December 2019). Education Testing Services Research Report 19-33. Princeton, NJ: Educational Testing Services. https://doi.org/10.1002/ets2.12271.

    Implicit Bias
    Fine, E. & Handelsman, J. (2012). Reviewing Applicants: Research on Bias and Assumptions.  Women in Science & Engineering Leadership Institute, University of Wisconsin-Madison. http://wiseli.engr.wisc.edu

    MacLachlan, A. J. (February 2017). Preservation of Educational Inequality in Doctoral Education: Tacit Knowledge, Implicit Bias and University Faculty. Research and Occasional Paper Series: Center for Studies in Higher Education, CSHE.1.17. http://CSHE.berkeley.edu/

    Ogunyemi, D. (August, 2021). A Practical Approach to Implicit Bias Training. Journal of Graduate Medical Education. http://dx.doi.org/10.4300/JGME-D-21-00598.1

    Tversky, A. & Kahneman, D. (Sep. 27, 1974). Judgement Under Uncertainty: Heuristics and Biases. Science, 185(4157), pp. 1124-1131.

    Letters of Recommendation
    Akos, P. & Kretchmar, J. (2016). Gender and Ethnic Bias in Letters of Recommendation: Considerations For School Counselors. Professional School Counseling, 20(1), pp. 102-113.

    Equity in Graduate Education (2020). Better Letters: Equitable Practices for Writing, Reading, and Soliciting Letters of Recommendation.  https://equitygraded.org/rsrc/equitable-practices-for-writing-reading-and-soliciting-letters-of-recommendation/.

    Madera, J. M., Hebl, M. R., Dial, H., & Valian, V. (2019). Raising Doubt in Letters of Recommendation for Academia: Gender Differences and Their Impact. Journal of Business and Psychology, 34(3), pp. 287-303.

    Trix, F. & Psenka, C. (2003). Exploring The Color of Glass: Letters of Recommendation for Female and Male Medical Faculty. Discourse & Society, 14(2), pp. 191-220.

    Meritocracy
    Bleemer, Z. (October 26, 2021). On the Meritocratic Allocation of Higher Education. W.E. Upjohn Institute For Employment Research.  Dissertation Award Summaries.

    The Harvard Law Review (June, 2008). "Trading Action for Access": The Myth of Meritocracy and The Failure to Remedy Structural Discrimination.  Harvard Law Review, 121(8), pp. 2156-2177.

    Nonacademic Factors (also called Non-Cognitive Variables)
    Conard, M. (2005). Aptitude is Not Enough: How Personality and Behavior Predict Academic Performance. Journal of Research in Personality, 40, pp. 339-346.

    Hossler, D., Chung, E., Kwon, J., Lucido, J., Bowman, N. & Bastedo, M. (2019). A Study of the Use of Nonacademic Factors in Holistic Undergraduate Admissions Reviews. The Journal of Higher Education, 90(6), pp. 833-859. DOI: 10.1080/00221574694.  https://doi.org/10.1080/002215462019.1574694.

    Humphries, J. E. & Fabian K. (2017). On The Interpretation of Non-Cognitive Skills- What is Being Measured and Why It Matters. Journal of Economic Behavior & Organization. 136, pp. 174-185.  http://dx.doi.org/10.1016/j.jebo.2017.02.001

    Kalsbeek, D., Sandlin, M., & Sedlacek, W. (July 1, 2013).  Employing Noncognitive Variables to Improve Admissions and Increase Student Diversity Retention. Strategic Enrollment Management Quarterly, V1(2), pp.132-150.

    Sedlacek, W. (2004). Why We Should Use Noncognitive Variables With Graduate and Professional Students. The Advisor: The Journal of The National Association of Advisors for the Health Professions. 24(2), pp. 32-39.

    Rubrics
    Posselt, J. R. & Miller, C. W. (n.d.). Rubric Design and Use. NSF funded grants material.

    Schoepp, K., Danaher, M., & Kranov, A. A. (2018). An Effective Rubric Norming Process. Practical Assessment, Research & Evaluation, 23(11).

    Standardized Tests
    Feldon, D. F., Litson, K., Cahoon, B., Feng, Z., Walker, A., & Tofel-Grehl, C. (2023).  The Predictive Validity of The GRE Across Graduate Outcomes: A Meta-Analysis of Trends Over Time. The Journal of Higher Education, DOI:10.1080/00221546.2023.2187177.

    Miller, C., & Stassun, K. (2014). A Test That Fails. Nature.org, 510(7504), pp. 303-304. doi:10.1038./nj7504-303a.