USE OF PORTFOLIOS IN EVALUATION (2024)

WHAT IS PORTFOLIO ASSESSMENT?

In program evaluation as in other areas, a picture can be worth a thousand words. As an evaluation tool for community-based programs, we can think of a portfolio as a kind of scrapbook or photo album that records the progress and activities of the program and its participants, and showcases them to interested parties both within and outside of the program. While portfolio assessment has been predominantly used in educational settings to document the progress and achievements of individual children and adolescents, it has the potential to be a valuable tool for program assessment as well.

Many programs do keep such albums, or scrapbooks, and use them informally as a means of conveying their pride in the program, but most do not consider using them in a systematic way as part of their formal program evaluation. However, the concepts and philosophy behind portfolios can apply to community evaluation, where portfolios can provide windows into community practices, procedures, and outcomes, perhaps better than more traditional measures.

ortfolio assessment has become widely used in educational settings as a way to examine and measure progress, by documenting the process of learning or change as it occurs. Portfolios extend beyond test scores to include substantive descriptions or examples of what the student is doing and experiencing. Fundamental to "authentic assessment" or "performance assessment" in educational theory is the principle that children and adolescents should demonstrate, rather than tell about, what they know and can do (Cole, Ryan, & Kick, 1995). Documenting progress toward higher order goals such as application of skills and synthesis of experience requires obtaining information beyond what can be provided by standardized or norm-based tests. In "authentic assessment", information or data is collected from various sources, through multiple methods, and over multiple points in time (Shaklee, Barbour, Ambrose, & Hansford, 1997). Contents of portfolios (sometimes called "artifacts" or "evidence") can include drawings, photos, video or audio tapes, writing or other work samples, computer disks, and copies of standardized or program-specific tests. Data sources can include parents, staff, and other community members who know the participants or program, as well as the self-reflections of participants themselves. Portfolio assessment provides a practical strategy for systematically collecting and organizing such data.

PORTFOLIO ASSESSMENT IS MOST USEFUL FOR:

*Evaluating programs that have flexible or individualized goals or outcomes. For example, within a program with the general purpose of enhancing children's social skills, some individual children may need to become less aggressive while other shy children may need to become more assertive.

Each child's portfolio asseessment would be geared to his or her individual needs andgoals.

*Allowing individuals and programs in the community (those being evaluated) to beinvolved in their own change and decisions to change.

*Providing information that gives meaningful insight into behavior and related change.Because portfolio assessment emphasizes the process of change or growth, atmultiple points in time, it may be easier to see patterns.

*Providing a tool that can ensure communication and accountability to a range ofaudiences. Participants, their families, funders, and members of the community atlarge who may not have much sophistication in interpreting statistical data can oftenappreciate more visual or experiential "evidence" of success.

*Allowing for the possibility of assessing some of the more complex and importantaspects of many constructs (rather than just the ones that are easiest to measure).

PORTFOLIO ASSESSMENT IS NOT AS USEFUL FOR:

*Evaluating programs that have very concrete, uniform goals or purposes. For example, it would be unneccessary to compile a portfolio of individualized "evidence" in a program whose sole purpose is full immunization of all children in a community by the age of five years. The required immunizations are the same, and the evidence is generally clear and straightforward.

*Allowing you to rank participants or programs in a quantitative or standardized way(although evaluators or program staff may be able to make subjective judgements ofrelative merit).

*Comparing participants or programs to standardized norms. While portfolios can (andoften do) include some standardized test scores along with other kinds of "evidence",this is not the main purpose of the portfolio.

USING PORTFOLIO ASSESSMENT WITH THE STATE STRENGTHENING EVALUATION GUIDE

Tier 1 - Program Definition

Using portfolios can help you to document the needs and assets of the community ofinterest. Portfolios can also help you to clarify the identity of your program and allowyou to document the "thinking" behind the development of and throughout theprogram. Ideally, the process of deciding on criteria for the portfolio will flow directlyfrom the program objectives that have been established in designing the program.However, in a new or existing program where the original objectives are not as clearlydefined as they need to be, program developers and staff may be able to clarify theirown thinking by visualizing what successful outcomes would look like, and what theywould accept as "evidence". Thus, thinking about portfolio criteria may contribute toclearer thinking and better definition of program objectives.

Tier 2 - Accountability

Critical to any form of assessment is accountability. In the educational arena forexample, teachers are accountable to themselves, their students, and the families, theschools and society. The portfolio is an assessment practice that can inform all ofthese constituents. The process of selecting "evidence" for inclusion in portfoliosinvolves ongoing dialogue and feedback between participants and service providers.

Tier 3 - Understanding and Refining

2) Assessment Criteria

Once the purpose or goal of the portfolio is clear, decisions are made about what will be considered sucess (criteria or standards), and what strategies are necessary to meet the goals. Items are then selected to include in the portfolio because they provide evidence of meeting criteria, or making progress toward goals.

3) Evidence

In collecting data, many things need to be considered. What sources of evidenceshould be used? How much evidence do we need to make good decisions anddeterminations? How often should we collect evidence? How congruent should thesources of evidence be? How can we make sense of the evidence that is collected?How should evidence be used to modify program and evaluation? According to Bartonand Collins (1997), evidence can include artifacts (items produced in the normalcourse of classroom or program activities), reproductions (documentation ofinterviews or projects done outside of the classroom or program), attestations(statements and observations by staff or others about the participant), andproductions (items prepared especially for the portfolio, such as participantreflections on their learning or choices) . Each item is selected because it adds somenew information related to attainment of the goals.

Steps of Portfolio Assessment

Although many variations of portfolio assessment are in use, most fall into two basictypes: process portfolios and product portfolios (Cole, Ryan, & Kick, 1995). These arenot the only kinds of portfolios in use, nor are they pure types clearly distinct fromeach other. It may be more helpful to think of these as two steps in the portfolioassessment process, as the participant(s) and staff reflectively select items from theirprocess portfolios for inclusion in the product portfolio.

Step 1: The first step is to develop a process portfolio, which documents growth overtime toward a goal. Documentation includes statements of the end goals, criteria, andplans for the future. This should include baseline information, or items describing theparticipant's performance or mastery level at the beginning of the program. Otheritems are "works in progress", selected at many interim points to demonstrate stepstoward mastery. At this stage, the portfolio is a formative evaluation tool, probablymost useful for the internal information of the participant(s) and staff as they plan forthe future.

Step 2: The next step is to develop a product portfolio (also known as a "best piecesportfolio"), which includes examples of the best efforts of a participant, community, orprogram. These also include "final evidence", or items which demonstrate attainmentof the end goals. Product or "best pieces" portfolios encourage reflection aboutchange or learning. The program participants, either individually or in groups, areinvolved in selecting the content, the criteria for selection, and the criteria for judgingmerits, and "evidence" that the criteria have been met (Winograd & Jones, 1992). Forindividuals and communities alike, this provides opportunities for a sense ofownership and strength. It helps to show-case or communicate the accomplishmentsof the person or program. At this stage, the portfolio is an example of summativeevaluation, and may be particularly useful as a public relations tool.

Distinguishing Characteristics

Certain characteristics are essential to the development of any type of portfolio usedfor assessment. According to Barton and Collins (1997), portfolios should be:

1) Multisourced (allowing for the opportunity to evaluate a variety of specificevidence)

Multiple data sources include both people (statements and observations ofparticipants, teachers or program staff, parents, and community members), andartifacts (anything from test scores to photos, drawings, journals, & audio orvideotapes of performances).

2) Authentic (context and evidence are directly linked)

The items selected or produced for evidence should be related to program activities,as well as the goals and criteria. If the portfolio is assessing the effect of a program onparticipants or communities, then the "evidence" should reflect the activities of theprogram rather than skills that were gained elsewhere. For example, if a child'smusical performance skills were gained through private piano lessons, not through4-H activities, an audio tape would be irrelevant in his 4-H portfolio. If a 4-H activityinvolved the same child in teaching other children to play, a tape might be relevant.

3) Dynamic (capturing growth and change)

An important feature of portfolio assessment is that data or evidence is added atmany points in time, not just as "before and after" measures. Rather than includingonly the best work, the portfolio should include examples of different stages ofmastery. At least some of the items are self-selected. This allows a much richerunderstanding of the process of change.

4) Explicit (purpose and goals are clearly defined)

The students or program participants should know in advance what is expected ofthem, so that they can take responsibility for developing their evidence.

5) Integrated (evidence should establish a correspondence between programactivities and life experiences)

Participants should be asked to demonstrate how they can apply their skills orknowledge to real-life situations.

6) Based on ownership (the participant helps determine evidence to include andgoals to be met)

The portfolio assessment process should require that the participants engage in somereflection and self-evaluation as they select the evidence to include and set or modifytheir goals. They are not simply being evaluated or graded by others.

7) Multipurposed (allowing assessment of the effectiveness of the program whileassessing performance of the participant).

A well-designed portfolio assessment process evaluates the effectiveness of yourintervention at the same time that it evaluates the growth of individuals orcommunities. It also serves as a communication tool when shared with family, otherstaff, or community members. In school settings, it can be passed on to otherteachers or staff as a child moves from one grade level to another.

Analyzing and Reporting Data

As with any qualitative assessment method, analysis of portfolio data can pose challenges. Methods of analysis will vary depending on the purpose of the portfolio, and the types of data collected (Patton, 1990). However, if goals and criteria have been clearly defined, the "evidence" in the portfolio makes it relatively easy to demonstrate that the individual or population has moved from a baseline level of performance to achievement of particular goals.

It should also be possible to report some aggregated or comparative results, even ifparticipants have individualized goals within a program. For example, in a teen peertutoring program, you might report that "X% of participants met or exceeded two ormore of their personal goals within this time frame", even if one teen's primary goalwas to gain public speaking skills and another's main goal was to raise his grade pointaverage by mastering study skills. Comparing across programs, you might be able tosay that the participants in Town X on average mastered 4 new skills in the course ofsix months, while those in Town Y only mastered 2, and speculate that lowerattendance rates in Town Y could account for the difference.

Subjectivity of judgements is often cited as a concern in this type of assessment(Bateson, 1994). However, in educational settings, teachers or staff using portfolioassessment often choose to periodically compare notes by independently rating thesame portfolio to see if they are in agreement on scoring (Barton & Collins, 1997).This provides a simple check on reliability, and can be very simply reported. Forexample, a local programmer could say "To ensure some consistency in assessmentstandards, every 5th portfolio (or 20%) was assessed by more than one staff member.Agreement between raters, or inter-rater reliability, was 88%".

There are many books and articles that address the problems of analyzing and reporting on qualitative data in more depth than can be covered here. The basic issues of reliability, validity and generalizability are relevant even when using qualitative methods, and various strategies have been developed to address them. Those who are considering using portfolio assessment in evaluation are encouraged to refer to some of the sources listed below for more in-depth information.

ANNOTATED BIBLIOGRAPHY

Barton, J., & Collins, A. (Eds.) (1997). Portfolio assessment: A handbook foreducators. Menlo Park, CA: Addison-Wesley Publishing Co.

A book about portfolio assessment written by and for teachers. The main goal is to give practical suggestions for creating portfolios so as to meet the unique needs and purposes of any classroom. The book includes information about designing portfolios, essential steps to make portfolios work, actual cases of portfolios in action, a compendium of portfolio implementation tips that save time and trouble, how to use portfolios to assess both teacher and student performance, and a summary of practical issues of portfolio development and implementation. This book is very clear, easy to follow, and can easily serve as a bridge between the use of portfolios in the classroom and the application of portfolios in community evaluations.

Bateson, D. (1994). Psychometric and philosophic problems in "authentic" assessment: Performance tasks and portfolios. Alberta Journal of Educational Research, 40 (2), p. 233-245.

Considers issues of reliability and validity in assessment which are as important in"authentic assessment" methods as in more traditional methods. Care needs to beexercised so that these increasingly popular new methods are not perceived as unfairor invalid.

Cole, D. J., Ryan, C. W., & Kick, F. (1995). Portfolios across the curriculum andbeyond. Thousand Oaks, CA: Corwin Press.

Authors discuss the development of authentic assessment and how it has led toportfolio usage. Guidelines are given for planning portfolios, how to use them,selection of portfolio contents, reporting strategies, and use of portfolios in theclassroom. In addition, a chapter focuses on the development of a professionalportfolio.

Courts, P. L., & McInerny, K. H. (1993). Assessment in higher education: Politics,pedagogy, and portfolios. London: Praeger.

The authors describe a project using portfolios to train teachers to assess exceptionalpotential in underserved populations. The portfolio includes observations of thechildren's behavior in the school, home, and community. The underlying assumptionof the project is that teachers learn to recognize exceptional potential if they areprovided with authentic examples of such behavior. Results indicated thatparticipating teachers experienced a sense of empowerment as a consequence of theproject and became both involved in and committed to the project.

Glasgow, N. A. (1997). New curriculum for new times: A guide to student-centered,problem-based learning. Thousand Oaks, CA: Corwin Press.

This book is an attempt to identify and define current practices and presentalternatives that can better meet the needs of a wider range of students in facilitatingliteracy and readiness for life outside the classroom. Discussion centers on currentcurriculum and the need for instruction that meets the changing educational context.Included is information about portfolio assessment, design and implementation, as asexamples of a new curricular style that promotes flexible and individualistic instruction.

Maurer, R. E. (1996). Designing alternative assessments for interdisciplinarycurriculum in middle and secondary schools. Boston: Allyn and Bacon.

This book explains how to design an assessment system that can authentically evaluate students' progress in an interdisciplinary curriculum. It offers step-by-step procedures, checklists, tables, charts, graphs, guides, worksheets, and examples of successful assessment methods. Specific to portfolio assessment, this book shows how portfolios can be used to measure learning. Provides some information on types and development of portfolios.

Patton, M. Q. (1990). Qualitative evaluation and research methods, 2nd ed. NewburyPark, CA: Sage.

A good general reference on issues of qualitative methods, and strategies foranalysis and interpretation of qualitative data.

Shaklee, B. D., Barbour, N. E., Ambrose, R., & Hansford, S. J. (1997). Designing andusing portfolios. Boston: Allyn and Bacon.

Discusses the history of portfolio assessment, decisions that need to be made before beginning the portfolio assessment process (eg., what it will look like, who should be involved, what should be assessed, how the assessment will be accomplished), designing a portfolio system (eg., criteria and standards), using portfolio results in planning, and issues related to assessment practices (eg., accountability).

Shaklee, B. D., & Viechnicki, K. J. (1995). A qualitative approach to portfolios: TheEarly Assessment for Exceptional Potential Model. Journal for the Education of theGifted, 18 (2), 156-170.

The creation of a portfolio assessment model based on qualitative research principlesis examined by the authors. Portfolio framework assumptions for classrooms are:designing authentic learning opportunities, interaction of assessment, curriculum andinstructions, multiple criteria derived from multiple sources, and systematic teacherpreparations. Additionally, the authors examine the qualitative research proceduresembedded in the development of the Early Assessment for Exceptional Potentialmodel. Provided are preliminary results for credibility, transferability, dependability,and confirmability of the design.

Winograd, P., & Jones, D. L. (1992). The use of portfolios in performanceassessment. New Directions for Educational Reform, 1 (2), 37-50.

Authors examine the use of portfolios in performance assessment. Suggestions are offered to educators interested in using portfolios in aiding students to become better readers and writers. Addresses concerns related to portfolios' usefulness. Educators need support in learning how to use portfolios, including their design, management, and interpretation.

Back to Alternative Methods website.