Self-Assessment Guide for Small Learning
Communities and Career Academies

The original version of this guide was made possible
by funding from the Wallace-Reader's Digest Fund.

It was revised and updated with support from
The James Irvine Foundation.

Career Academy Support Network
University of California, Berkeley
Graduate School of Education
Berkeley, CA 94720-1670

Introduction and Purpose

In order to assess how completely a Small Learning Community (SLC) and/or Career Academy is implemented in any given site, and to connect the degree of implementation with the amount of improvement in student performance, two kinds of information are needed. The first is information on the quality of implementation, which can be collected by academy staff or outside observers, to assess how well the SLC/academy features have been established. The second is data on student performance, the kind normally kept by schools and districts, which can give a picture of who is enrolled in academies, whether their course-taking experience is consistent with the academy model, and how their performance is affected by the SLC or academy.

To assist in collecting implementation information, this Guide begins with a combined Self-Assessment Checklist and Scoring Guide . The three sections of thisóSmall Learning Communities, Curriculum and Instruction, and Partnerships with Employers, Community, and Higher Educationóderive from the definition of Career Academies agreed to by the organizations working to support them nationally, supplemented by more recent guidelines for evaluating SLCs.

The purpose of the Checklist is twofold. One is to obtain an indication of how well the SLC/ Academy stacks up against nationally defined criteria for quality implementation. To calculate a score, check the circle corresponding to the point value for each indicator and add the points to determine the score. There are 25 indicators in all; thus the top score possible is 100. If the SLC/ Academy is not yet fully implemented, some of the items will be inappropriate (e.g., post-secondary plan, mentor/ internship/ community service programs), and the scoring needs to be adjusted accordingly.

A second and perhaps more important purpose of the Checklist is to identify each SLC's/ Academy's relative strengths and weaknesses and initiate a process of reflection and improvement. Notes concerning each indicator can be included, such as what parts of the component are well implemented or need attention, and what needs to be done to bring improvement. Comparisons can be made among the three sections and the components within each to see which features are strongest and weakest. This process is best done by an SLC/ Academy team so the teachers, administrators, and partners involved can explore options and define paths toward improvement.

A second section on Analyzing Student Data includes suggestions for compiling and analyzing commonly available data for measuring SLC/ Academy effectiveness. This can answer three kinds of questions:

• DemographicsóDo SLC/ Academy students represent a cross section of the school?

• Program experienceóDoes student course taking reflect the SLC/ Academy design?

• Student outcomesóAre SLC/ Academy students showing improved attendance, retention, credits, grades, test scores, graduation rates, and college entrance rates?

As explained in the section on Analyzing Student Data , these questions can be addressed by:

• Comparing snapshots over time for an individual SLC/ Academy

• Analyzing year-to-year changes for individual students

• Relating program characteristics to student performance

SLC/ Academy Self-Assessment Checklist and Scoring Guide

For Small Learning Communities, there are a total of 20 items in two sections: ìSmall Learning Community Structuresî, and ìCurriculum and Instructionî. Checks can be entered in one of five "level of implementation" circles for each indicator on a continuum from 1= no implementation to 5 = full implementation. Scores should be determined using the criteria in the scoring guide. For example, if none of the criteria listed in the scoring guide have been met, circle 1 should be checked. If all the criteria listed have been met, circle 5 should be checked. Thus a perfect score is 100.

For Career Academies, a third category is included, ìPartnerships with Employersî, with five additional items. Thus the total number of items for Academies is 25. In this case each item should be scored on a four point scale, from 1=no implementation to 4=full implementation. Again, the perfect score is 100.

Choices should be based on the knowledge of the SLC/ Academy and the sense for how fully achieved each aspect of the program is, given what is possible at the high school and how serious any problems are. For example, if there are only 90 students in the SLC or Academy across three grade levels, but this is as large as is realistically possible in the high school, this can be considered a 4 or 5. Conversely, if there is an advisory program in place for all students but it is poorly run, this might receive a 1 or 2.

Part 1: Small Learning Community Structures

• Defined mission and goals ó The SLC/ Academy has a written definition of its mission and goals that are available to the administrators, teachers, students, parents, advisory board, and others involved. These include a focus on increasing student achievement, raising student aspirations and motivation, and developing post-secondary plans, including education, and in an Academy, possible future careers.

Notes:______________________________________________________________________

___________________________________________________________________________

• Student selection óGrade nine or nine-ten SLCs may assign students if all options are essentially equivalent and there is no defined theme. Career Academies, and SLCs at higher grade levels or with themes, inform students of their options at least the semester before they enter, and students enter voluntarily via an application process. Each SLC/ Academy reflects the high school's demographics. Attrition removes less than 50% of the original participants by graduation.

Notes:______________________________________________________________________

___________________________________________________________________________

• Cohort scheduling ó SLC or Academy students have two or more classes together each semester, limited to the students enrolled in the program (minimally 80%), and taught by a team of teachers who work together to plan the program, coordinate their instruction, and evolve common strategies for helping students having problems.

Notes:______________________________________________________________________

____________________________________________________________________________

• Grade levels/ courses óSLCs may operate at a given grade level, especially in grade nine. Themed SLCs and Academies operate at a minimum of two grade levels (11-12), preferably three (10-12) or four (9-12), with teacher looping where feasible. Academies have one Career-Technical or career themed academic class each year. A written course sequence across grade levels exists. SLC/ Academy students have the same options for non-SLC/ Academy courses as others at the high school.

Notes:______________________________________________________________________

____________________________________________________________________________

• Teacher selection, roles óTeachers choose to participate in the SLC/ Academy. One teacher (sometimes two) agrees to take the lead, serving as the SLC/ Academy Coordinator(s): e.g. interacting with administrators and board members, managing the budget, helping to coordinate teacher professional development, and helping to coordinate employer, higher education, and parental involvement. Teacher replacements are guided by the SLC/ Academy team. A majority of SLC/ Academy teachers' classes are taught in the program.

Notes:______________________________________________________________________

____________________________________________________________________________

• Administrative support ó The district Board of Education is aware of the SLC/ Academy and its mission and goals, and is on public record in support. The Superintendent publicly endorses the program and offers active support. The high school principal/ administrators are knowledgeable of the program, public advocates, and are involved in its funding, staffing and support. This support results in adequate funding, facilities, equipment and learning materials.

Notes:______________________________________________________________________

____________________________________________________________________________

• Counselor/ scheduling support óAll high school counselors can explain what cohort scheduling is, and understand its importance. Master scheduling is a cooperative effort that integrates the needs and support of SLC/ Academy teachers. The master schedule indicates which classes are in the SLC/ Academy. All SLC/ Academy classes are comprised of at least 80% program students. Counselor caseloads are assigned by SLC/ Academy.

Notes:______________________________________________________________________

____________________________________________________________________________

• Governance and leadership ó The SLC/Academy has an advisory board with members from the district and high school administration, program teaching staff, and those involved from outside the high school (e.g., supporting employers, institutions of higher education). It may also include community representatives, and academy parents and students. The board holds meetings at least quarterly, with defined agendas and outcomes, and helps to define policies. Students have input to these policies.

Notes:______________________________________________________________________

____________________________________________________________________________

• Parental involvement óParents or guardians learn of the SLC/ Academy in advance of their son or daughter joining it, and help to support them in the program when needed. Where enrollment is by choice, they endorse the student's application. They attend recognition events, and are involved in other appropriate activities (e.g., Advisory Committee, instructional support/ volunteer aides, field trips, other program activities). They meet with the SLC/ Academy team when needed to resolve problems their son or daughter is having.

Notes:______________________________________________________________________

____________________________________________________________________________

• Enrichment and personalization óThe SLC/ Academy maintains limited size, teacher teamwork, and a supportive atmosphere. There are no more than 150 students enrolled per grade level/ SLC/ Academy. Where possible, academy classrooms are near each other in the building. Teacher and/or peer tutoring is available for students. Student achievement is recognized publicly at least quarterly. At least one SLC/ Academy social event is held per semester. At least one SLC/ Academy recognition ceremony is held per school year.

Notes:______________________________________________________________________

____________________________________________________________________________

• A cycle of improvement óSLC/ Academy implementation is regularly examined. Program leaders regularly assess the SLC/ Academy's functioning, studying its strengths and weaknesses. This involves gathering feedback from key stakeholders, including students. These reviews lead to plans to address any problems. Such plans include timetables and benchmarks for improvement, and refer back to the SLC/ Academy's underlying mission and goals.

Notes:______________________________________________________________________

____________________________________________________________________________

Part 1 Score: ( SLC maxó55; Academy maxó44)

Part 2: Curriculum and Instruction

• Standards, assessment ó The academic curriculum is framed around state and/ or national standards. Curriculum and instructional materials in career related classes are based at least partially on an industry source; SCANS skills are incorporated and assessed; assessments are multiple and reflect practices in the career field.

Notes:______________________________________________________________________

____________________________________________________________________________

• Academic courses ó Coursework reaches high levels of English and math, generally four years of each, in addition to substantial coursework in science and social studies. Graduates are qualified to attend four-year colleges. Flexibility is provided for students with special needs (e.g., English language learners, special education students, AP and IB students).

Notes:______________________________________________________________________

____________________________________________________________________________

• Career-technical courses óIn an Academy there is a written CTE or career themed academic course sequence across the years of the Academy; local employers from the career field help to guide the curriculum in these courses; students can demonstrate knowledge of a vertical range of careers and related educational requirements in the career field; the sequence of CTE courses enables interested students to obtain a skill certification recognized by employers.

Notes:______________________________________________________________________

____________________________________________________________________________

• Faculty and Staff ó Teachers are credentialed in their field, volunteers in the SLC/ Academy, committed to its mission and goals, and willing to work in a team and share the extra responsibilities involved. These include organizing and attending Advisory Committee meetings, helping with student recruitment, organizing parent contacts and participating in parent meetings, and providing student counseling.

Notes:______________________________________________________________________

____________________________________________________________________________

• Curriculum integration óStudents are shown how their academic subjects relate to each other and apply in real world contexts. Students engage in projects requiring the application of skills from several courses; these include a senior and/or capstone project; such projects are assessed at least in part by adults outside the high school. In an Academy, students are shown the relationship between their academic subjects and career theme; projects are assessed at least in part by employer partners from the career field.

Notes:______________________________________________________________________

____________________________________________________________________________

• Teacher coordination of curriculum óSLC/ Academy teachers have regular meeting time, at least weekly; this occurs wherever possible during a common planning period; Lead teachers are provided at least one release period per day; compensation is provided other SLC/ Academy teachers for substantial time involvement beyond their regular teaching schedule. Teachers can identify at least one project per semester that requires students to integrate curriculum among academic subjects, and in an Academy between these and the career field. Teachers are provided staff development support for these efforts.

Notes:______________________________________________________________________

____________________________________________________________________________

• Professional development óSLC/A cademy teachers are provided with training in the SLC structures, working as a team, curricular integration, student support, cohort scheduling, and involving those from outside the high school, where necessary by experts from outside the high school. Parents are adequately prepared for their involvement (if any) as classroom aides, field trip chaperones and social event organizers.

Notes:______________________________________________________________________

____________________________________________________________________________

• Postsecondary plan/ links óSLC/Academy students are exposed to two- and four-year colleges through visits and information meetings; they also have access to a career library and post-graduate counseling in this process; they develop a written post-secondary plan by the end of their junior year; there are articulation agreements between the SLC/Academy and local postsecondary programs and institutions; students have the option of earning some college credit while in the program.

Notes:______________________________________________________________________

____________________________________________________________________________

• Student assessment óStudent data are collected to describe the student body within the SLC/Academy (e.g., grade level, gender, race/ethnicity), as well as multiple outcome measures, including a variety of accepted indicators of performance (e.g., attendance, retention, credits, grade point averages, state test scores, graduation rates, college going rates). These measures are reported accurately and fairly, and show whether, and how much, the academy improves student performance. In Academies, measures include knowledge of the field's terminology, technical concepts, and ability to apply English, math, and other academic skills to authentic real world projects. Where appropriate, industry certification is incorporated.

Notes:______________________________________________________________________

____________________________________________________________________________

Part 2 Score: ( SLC max--45; Academy max--36)

Total SLC Score: (max--100)

Academy raters go on to next section.

Part 3: Partnerships with Employers

• Career field selection óThe Academy career field is selected with input from local employers; a number of such employers support the Academy (e.g., as Advisory Committee members, speakers, field trip hosts, mentors, and internship supervisors); the industry is growing and offers well paying career options with upward mobility.

Notes:______________________________________________________________________

____________________________________________________________________________

• Sophomore business speakers and field trips óAt least two speakers per semester from the employer partners describe their employer's products, services and range of jobs; at least one student field trip per semester takes sophomores to places of employment in the academy career field; these experiences are linked to the curriculum and classroom instruction.

Notes:______________________________________________________________________

____________________________________________________________________________

• Junior mentor program óJuniors have a volunteer employee who serves as a career related mentor; there is a process that pairs students with well matched mentors; a staff member coordinates this program; mentors receive an orientation for this role; a series of mentor/ student experiences is planned throughout the year; a mechanism is in place to deal with problems; students and mentors complete written evaluations of the experience at the end of the year.

Notes:______________________________________________________________________

____________________________________________________________________________

• Junior/ senior internship program óThe summer following the junior year, and part-time the senior year, students have the option of a job with a cooperating employer; there is a process to match students with appropriate positions; there is a written plan to guide this experience; students are exposed to a variety of positions and learn of their related training; a staff member coordinates this program, checks on students during the internship, and resolves problems; students and supervisors complete a written evaluation at the end of the internship.

Notes:______________________________________________________________________

____________________________________________________________________________

• Community service, school-based enterprise óIf an internship is not available for all students, there is an option of an unpaid community service position, or a role in an academy operated business; there is a process to match students with appropriate positions/ roles; a staff member coordinates these programs, checks on students involved, and resolves problems; these experiences are linked to the curriculum; students and supervising adults complete a written evaluation at the end of the experience.

Notes:______________________________________________________________________

____________________________________________________________________________

Part 3 Score: (max--20)

Total Academy Score: (max--100)

Analyzing Student Data

This is a companion to the Checklist and Scoring Guide . The procedures described here are designed to guide the analysis of student data. The two approaches are most effective when used together.

Three kinds of information about students are especially useful for SLCs/ Academies to consider: 1) demographics, 2) measures of program experience, and 3) student outcomes.

1. Demographics. These provide a picture of the students enrolled in the program. Suggested categories here: age, grade level, gender, race/ethnicity, grade-point average prior to SLC Academy entry, and standardized test scores. In order to ensure that each program enrolls a reasonably representative cross-section of the school's students, the characteristics of SLC/ Academy students should be compared with characteristics of non-program students at each grade level.

2. SLC/ Academy experience measures. These provide information on the extent to which students are participating in the planned curriculum. Although the SLC Academy lists a set of courses to be taken by students at each grade level, conflicts in scheduling may prevent some program students from taking some of the planned courses. The proportion of intended courses that a student takes at each grade level, or over the duration of the whole program, is an indicator of the extent to which the student has actually participated in the intended curriculum. This proportion can be averaged across students to give a measure of curricular integrity.

Another such indicator is the proportion of students in SLC/ Academy classes who are actually program members. In theory, such classes consist entirely of program students. However, in practice the complexities of scheduling sometimes result in non-program students being included. This may dilute the effectiveness of the program. It is possible to measure the ìpurityî of each SLC/ Academy class as the proportion of students who are members of that program. That proportion can be averaged over program classes, giving a measure of the average purity of classes in the program.

3. Student outcomes. These provide information on SLC/ Academy student performance. Outcomes to be measured each year can include: attendance (percentage of days attended); dropouts (for students who leave their high school, to determine whether they transfer to another high school, formally drop out, or simply disappear); credits earned toward graduation; grade for each course taken (to determine student's GPA); scores on state language arts and math tests; for twelfth graders, whether the student graduates on time; and for twelfth graders, whether the student qualifies for admission to a four-year college.

All or most of this information is usually available from computerized data bases maintained by school districts. The only special requirements are that ìflagsî be attached to each SLC/ Academy student, indicating the SLC/ Academy to which he or she belongs, and to each program course. This information can be used in various ways. Here are three examples.

1. Comparing snapshots over time, for an individual SLC/ Academy. All three kinds of information ññ demographics of students compared to the host high school, program experience measures, and student outcomes ññ can be compiled each year, for each grade level. Comparing these measures from one year to the next ññ for instance, information on this year's twelfth graders compared to last year's ññ gives a direct indication of whether the program is improving over time. However, changes in student outcomes may be due to differences between this year's and last year's students that are not accounted for by the available demographic information, so results must be cautiously interpreted.

2. Comparing year-to-year changes for individual students. The performance of this year's twelfth graders can be compared to their own performance in eleventh grade or across grades 9-12. This is a more direct measure of improvement in student performance, but it requires being able to link information for the same student from year to year. If this can be done, the average change for students in each cohort, compared to the cohort's own previous performance, can be computed each year as an indicator of program effectiveness. This measure of students' year-to-year progress can also be compared over time for each SLC/ Academy, as an indicator of whether the program is becoming more effective. However, if the data show changes in effectiveness, the specific program features that cause these (if they can be separated) still remain to be explained.

3. Relating SLC/ Academy program characteristics to student performance. The most powerful use of this information is to support a systematic process of continuous improvement. For this purpose, it is helpful to compare similar information from a number of different SLCs/ Academies. Correlations can be computed between students' year-to-year progress and various program characteristics, including the program experience measures described above. Statistical adjustments can be made to take account of any changes in student demographics. This kind of analysis can provide practical guidance by identifying programmatic variables that may cause student performance in some SLCs/ Academies to improve faster than in others.

Additional Options. All of these analyses can be done with information that is usually available in district databases. However, if additional data collection is possible, more extensive evaluations can add data on measures such as: student disciplinary actions (e.g., detentions, suspensions, expulsions); awards; SAT/ACT scores; college applications/ acceptances; and post-graduate plans. The purpose is to see whether SLC/ Academy involvement improves these. Comparisons with non-program students in the same high school, or changes over time, can provide a basis for analysis. Student (and teacher) questionnaires can also be used to gather information on education and career related activities and attitudes. Students can be followed up at one or more points after graduation to determine whether they go on to some form of college or work, and how they do in these pursuits. In short, this is just a brief synopsis of the possibilities that exist.