pagetop graphic

Self Assessment Guide for Small Learning Communities and Career Academies

The original version of this guide was made possible by funding from the Wallace-Reader's Digest Fund It was revised and updated with support from The James Irvine Foundation.

Charles Dayton
David Stern

2010

College and Career Academy Support Network
University of California, Berkeley
Graduate School of Education
Berkeley, CA 94720-1670
ask_casn@berkeley.edu
http://casn.berkeley.edu
http://collegetools.berkeley.edu

 

Introduction and Purpose

In order to assess how completely a Small Learning Community (SLC) and/or Career Academy is implemented in any given site, and to connect the degree of implementation with the amount of improvement in student performance, two kinds of information are needed. The first is information on the quality of implementation, which can be collected by academy staff or outside observers, to assess how well the SLC/academy features have been established. The second is data on student performance, the kind normally kept by schools and districts, which can give a picture of who is enrolled in academies, whether their course-taking experience is consistent with the academy model, and how their performance is affected as a result.

To assist in collecting implementation information, this Guide begins with a Self-Assessment Checklist and Scoring Guide. The three sections of this-Small Learning Communities, Curriculum and Instruction, and Partnerships with Employers, Community, and Higher Education-derive from the definition of Career Academies agreed to by the organizations working to support them nationally, supplemented by more recent guidelines for evaluating SLCs. The requirements of California's Partnership Academies are also considered.

The purpose of the Checklist is twofold. One is to obtain an indication of how well the SLC/ Academy stacks up against national and state criteria for quality implementation. To calculate a score, check the circle corresponding to the point value for each indicator and add the points to determine the score. The top score possible is 100. If the SLC/ Academy is not yet fully implemented, some of the items will be inappropriate (e.g. post-secondary plan, mentor/ internship/ community service programs), and the scoring can be adjusted accordingly.

A second and perhaps more important purpose of the Checklist is to identify each SLC's/ Academy's relative strengths and weaknesses and initiate a process of reflection and improvement. Notes concerning each indicator can be included, such as what parts of the component are well implemented or need attention, and what needs to be done to bring improvement. Comparisons can be made among the three sections and the components within each to see which features are strongest and weakest. This process is best done by a team, so teachers, administrators, and partners involved can explore options and define paths toward improvement.

There is a second brief section on Analyzing Student Data that includes suggestions for compiling and analyzing commonly available data for measuring SLC/ Academy effectiveness. This can answer three kinds of questions:

  • Demographics-Do SLC/ Academy students represent a cross section of the school?
  • Program experience-Does student course taking reflect the SLC/ Academy design?
  • Student outcomes-Are SLC/ Academy students showing improved attendance, retention, credits, grades, test scores, graduation rates, and college entrance rates?

As explained in this section, these questions can be addressed by:

  • Comparing snapshots over time for an individual SLC/ Academy
  • Analyzing year-to-year changes for individual students
  • Relating program characteristics to student performance

 

SLC/ Academy Self-Assessment Checklist and Scoring Guide

For Small Learning Communities, there are a total of 20 items in two sections: "Small Learning Community Structures", and "Curriculum and Instruction". Checks can be entered in one of five "level of implementation" circles for each indicator on a continuum from 1= no implementation to 5 = full implementation. Scores should be determined using the criteria in the scoring guide. For example, if none of the criteria listed in the scoring guide have been met, circle 1 should be checked. If all the criteria listed have been met, circle 5 should be checked. Thus a perfect score is 100.

For Career Academies, a third category is included, "Partnerships with Employers and Higher Education", with five additional items. Thus the total number of items for Academies is 25. In this case each item should be scored on a four point scale, from 1 = no implementation to 4 = full implementation. Again, the perfect score is 100.

Choices should be based on the knowledge of the SLC/ Academy and the sense for how fully achieved each aspect of the program is, given what is possible at the high school and how serious any problems are. For example, if there are only 90 students in the SLC or Academy across three grade levels, but this is as large as is realistically possible in the high school, this can be considered a 4 or 5. Conversely, if there is an advisory program in place for 300 students but it is poorly run, this might receive a 1 or 2.

 

Part 1: Small Learning Community Structures

Defined mission and goals-The SLC/ Academy has a written definition of its mission and goals that are available to the administrators, teachers, students, parents, advisory board, and others involved. These include a focus on increasing student achievement, raising student aspirations and motivation, and developing post-secondary plans, including education, and in an Academy, possible future careers.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Student selection-Grade nine or nine-ten SLCs may assign students if all options are essentially equivalent and there is no defined theme. Career Academies, and SLCs at higher grade levels or with themes, inform students of their options at least the semester before they enter, and students enter voluntarily via an application process. Each SLC/ Academy reflects the high school's demographics. Attrition removes less than 50% of the original participants by graduation.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Cohort scheduling- SLC or Academy students have two or more classes together each semester, limited to the students enrolled in the program (minimally 80%), and taught by a team of teachers who work together to plan the program, coordinate their instruction, and evolve common strategies for helping students having problems.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Grade levels/ courses-SLCs may operate at a given grade level, especially in grade nine. Themed SLCs and Academies operate at a minimum of two grade levels (11-12), preferably three (10-12) or four (9-12), with teacher looping where feasible. Academies have one Career-Technical or career themed academic class each year. A written course sequence across grade levels exists. SLC/ Academy students have the same options for non-SLC/ Academy courses as others at the high school.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Teacher selection, roles-Teachers choose to participate in the SLC/ Academy. One teacher (sometimes two) agrees to take the lead, serving as the SLC/ Academy Coordinator(s): e.g. interacting with administrators and board members, managing the budget, helping to coordinate teacher professional development, and helping to coordinate employer, higher education, and parental involvement. Teacher replacements are guided by the SLC/ Academy team. A majority of SLC/ Academy teachers' classes are taught in the program.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Administrative support- The district Board of Education is aware of the SLC/ Academy and its mission and goals, and is on public record in support. The Superintendent publicly endorses the program and offers active support. The high school principal/ administrators are knowledgeable of the program, public advocates, and are involved in its funding, staffing and support. This support results in adequate funding, facilities, equipment and learning materials.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Counselor/ scheduling support-All high school counselors can explain what cohort scheduling is, and understand its importance. Master scheduling is a cooperative effort that integrates the needs and support of SLC/ Academy teachers. The master schedule indicates which classes are in the SLC/ Academy. All SLC/ Academy classes are comprised of at least 80% program students. Counselor caseloads are assigned by SLC/ Academy.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Governance and leadership- The SLC/Academy has an advisory board with members from the district and high school administration, program teaching staff, and those involved from outside the high school (e.g., supporting employers, institutions of higher education). It may also include community representatives, and academy parents and students. The board holds meetings at least quarterly, with defined agendas and outcomes, and helps to define policies. Students have input to these policies.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Parental involvement-Parents or guardians learn of the SLC/ Academy in advance of their son or daughter joining it, endorse the student's application, and provide support to them when needed. They are involved in appropriate activities (e.g., Advisory Board, instructional support/ volunteer aides, field trips, recognition events, and so on). They meet with the SLC/ Academy team when needed to resolve problems their son or daughter is having.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Enrichment and personalization-The SLC/ Academy maintains limited size, teacher teamwork, and a supportive atmosphere. There are no more than 150 students enrolled per grade level. Where possible, academy classrooms are near each other in the building. Teacher and/or peer tutoring is available for students. Student achievement is recognized publicly at least quarterly. At least one SLC/ Academy social event is held per semester. At least one SLC/ Academy recognition ceremony is held per school year.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

A cycle of improvement-SLC/ Academy implementation is regularly examined. Program leaders regularly assess the SLC/ Academy's functioning, studying its strengths and weaknesses. This involves gathering feedback from key stakeholders, including students. These reviews lead to plans to address any problems. Such plans include timetables and benchmarks for improvement, and refer back to the SLC/ Academy's underlying mission and goals.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Part 1 Score: ( SLC max-55; Academy max-44)

 

Part 2: Curriculum and Instruction

Standards, assessment- The academic curriculum is framed around state and/ or national standards. Curriculum and instructional materials in career related classes are based at least partially on an industry source; SCANS skills are incorporated and assessed; assessments are multiple and reflect practices in the career field.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Academic courses-Coursework reaches high levels of English and math, generally four years of each, in addition to substantial coursework in science and social studies. Graduates are qualified to attend four-year colleges. Flexibility is provided for students with special needs (e.g., English language learners, special education students, AP and IB students).

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Career-technical courses-In an Academy there is a written CTE or career themed academic course sequence across the years of the Academy; local employers from the career field help to guide the curriculum in these courses; students can demonstrate knowledge of a vertical range of careers and related educational requirements in the career field; if appropriate, the sequence of CTE courses enables interested students to obtain a skill certification recognized by employers.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Faculty and Staff-Teachers are credentialed in their field, volunteers in the SLC/ Academy, committed to its mission and goals, and willing to work in a team and share the extra responsibilities involved. These include organizing and attending Advisory Committee meetings, helping with student recruitment, organizing parent contacts and participating in parent meetings, and providing student counseling.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Curriculum integration-Students are shown how their academic subjects relate to each other and apply in real world contexts. Students engage in projects requiring the application of skills from several courses; these include a senior and/or capstone project; such projects are assessed at least in part by adults outside the high school. In an Academy, students are shown the relationship between their academic subjects and career theme; projects are assessed at least in part by employer partners from the career field.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Teacher coordination of curriculum-SLC/ Academy teachers have regular meeting time, at least weekly; this occurs wherever possible during a common planning period; lead teachers are provided at least one release period per day; compensation is provided other SLC/ Academy teachers for substantial time involvement beyond their regular teaching schedule. Teachers can identify at least one project per semester that requires students to integrate curriculum among academic subjects, and in an Academy between these and the career field. Teachers are provided staff development support for these efforts.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Professional development-SLC/Academy teachers are provided with training in: the SLC structures, working as a team, curricular integration, student support, cohort scheduling, and where necessary, involving experts from outside the high school. Parents are adequately prepared for their involvement (if any) as classroom aides, field trip chaperones and social event organizers.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Postsecondary plan/ links-SLC/Academy students are exposed to two- and four-year colleges through speakers, visits and information meetings; they have access to a career library and post-graduate counseling; they develop a written post-secondary plan by the end of their junior year; there are articulation agreements between the SLC/Academy and local postsecondary institutions; students have the option of earning some college credit while in the program.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Student assessment-Student data are collected to describe the student body within the SLC/Academy (e.g., grade level, gender, race/ethnicity), as well as multiple outcome measures, including a variety of accepted indicators of performance (e.g., attendance, retention, credits, grade point averages, state test scores, graduation rates, college going rates). These measures are reported accurately and fairly, and show whether, and how much, the academy improves student performance. In Academies, measures include knowledge of the field's terminology, technical concepts, and ability to apply English, math, and other academic skills to authentic real world projects. Where appropriate, industry certification is incorporated.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Part 2 Score: ( SLC max--45; Academy max--36)

Total SLC Score: (max--100)

Academy raters go on to next section.

 

Part 3: Partnerships with Employers and Higher Education

Career field selection-The Academy career field is selected with input from local employers; a number of such employers support the Academy (e.g., as Advisory Committee members, speakers, field trip hosts, mentors, and internship supervisors); the industry is growing and offers well paying career options with upward mobility; there are programs in local colleges students can advance to after graduation.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Sophomore business speakers and field trips-At least two speakers per semester from the employer and high education partners describe their products, services, range of jobs, and related training programs; at least one student field trip per semester takes sophomores to places of employment in the academy career field or local college campuses; these experiences are linked to the curriculum and classroom instruction.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Junior mentor program-Juniors have a volunteer employee or college representative who serves as a career related mentor; there is a process that pairs students with well matched mentors; a staff member coordinates this program; mentors receive an orientation for this role; a series of mentor/ student experiences is planned throughout the year; a mechanism is in place to deal with problems; students and mentors complete written evaluations of the experience at the end of the year.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Junior/ senior workplace learning program-The summer following the junior year, and/or part-time the senior year, students have the option of an internship with a cooperating employer or a community service position; there is a process to match students with appropriate positions; there is a written plan to guide this experience; if an internship, students are exposed to a variety of positions in the company and learn of the related training; a staff member coordinates this program, checks on students during the experience, and resolves problems; students and supervisors complete a written evaluation at the end of the program.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Dual Enrollment Options-Links are established between the academy and local colleges, especially community colleges, that allow juniors and/or seniors to take courses that build on their Academy course sequence and provide credit at both the high school and college. College representatives provide input for the academy career related course sequence so that it will dovetail with their college program. The college courses may be offered at either the high school or college, and by teachers from either level, depending on local conditions. The cost of these is borne at least in part by the college in exchange for the increased enrollment in its courses. High school and college teachers work together in administering this program, with support from their respective administrations. Counselors at both levels are informed of these options and academy students are given ample opportunity to enroll in them.

Level of Implementation 1 2 3 4 5
  O O O O O

Notes:

 

Part 3 Score: (max--20)

Total Academy Score: (max--100)

 

Analyzing Student Data

This is a companion to the Checklist and Scoring Guide. The procedures described here are designed to guide the analysis of student data. The two approaches are most effective when used together.

Three kinds of information about students are especially useful for SLCs/ Academies to consider: 1) demographics; 2) measures of program experience; and 3) student outcomes.

  1. Demographics. These provide a picture of the students enrolled in the program. Suggested categories here: age, grade level, gender, race/ethnicity, grade-point average prior to SLC or Academy entry, and standardized test scores. In order to ensure that each program enrolls a reasonably representative cross-section of the school's students, the characteristics of SLC/ Academy students should be compared with characteristics of non-program students at each grade level.
  2. SLC or Academy experience measures. These provide information on the extent to which students are participating in the planned curriculum. Although the SLC or Academy lists a set of courses to be taken by students at each grade level, conflicts in scheduling may prevent some students from taking some of the planned courses. The proportion of intended courses that a student takes at each grade level, or over the duration of the program, is an indicator of the extent to which the student has actually participated in the intended curriculum. This proportion can be averaged across students to give a measure of curricular integrity. Another such indicator is the proportion of students in SLC or Academy classes who are actually program members. In theory, such classes should consist entirely of program students. However, in practice the complexities of scheduling sometimes result in non-program students being included. This may dilute the effectiveness of the program. It is possible to measure the "purity" of each SLC/ Academy class as the proportion of students who are members of the program. That proportion can be averaged over program classes, giving a measure of the average purity of classes in the program.
  3. Student outcomes. These provide information on SLC or Academy student performance. Outcomes to be measured each year can include: attendance (percentage of days attended); both program and school dropouts (students who leave their high school may transfer to another high school, formally drop out, or simply disappear); credits earned toward graduation; grade for each course taken (and annual GPA); scores on state tests; and for twelfth graders, a) whether the student graduates on time and b) qualifies for admission to a four-year college.

Most of this information is usually available from computerized databases maintained by school districts. The only special requirements are that "flags" be attached to each SLC/ Academy student, indicating the SLC/ Academy to which he or she belongs, and to each program course. This information can be used in various ways. Here are three examples.

  1. Comparing snapshots over time, for an individual SLC/ Academy. All three kinds of information -- demographics of students, program experience measures, and student outcomes -- can be compiled each year, for each grade level. Comparing these measures from one year to the next -- for instance, information on this year's twelfth graders compared to last year's -- gives an indication of whether the program is improving over time. However, changes in student outcomes may be due to differences between this year's and last year's classes and must be cautiously interpreted.
  2. Comparing year-to-year changes for individual students. The performance of this year's twelfth graders can be compared to their own performance in eleventh grade or across grades 9-12. This is a more valid measure of improvement in student performance, but it requires being able to link information for a student from year to year. If this can be done, the average change for students across each cohort, compared to the cohort's previous performance, can be computed each year as an indicator of program effectiveness. This measure of students' year-to-year progress can also be compared over time for each SLC/ Academy as an indicator of whether the program is becoming more effective. However, even if such data show progression over time, as hoped, the specific program features that cause these improvements may still be unclear.
  3. Relating SLC/ Academy program characteristics to student performance. The most powerful use of this information is to support a systematic process of continuous improvement. For this purpose, it is helpful to compare similar information from a number of different SLCs/ Academies. Correlations can be computed between students' year-to-year progress and various program characteristics, including the program experience measures described above. Statistical adjustments can be made to take account of any changes in student demographics. This kind of analysis can provide practical guidance by identifying programmatic variables that may cause student performance in some SLCs/ Academies to improve faster than in others.

Additional Options. All of these analyses can be done with information that is usually available in district databases. Sometimes additional measures can be added, such as: student disciplinary actions (e.g., detentions, suspensions, expulsions); awards; SAT/ACT scores; college applications/ acceptances; and post-graduate plans. The purpose is to see whether SLC/ Academy involvement improves these. Comparisons with non-program students in the same high school, or changes over time, or better yet a combination of the two, can provide a basis for gauging program impact. Student (and teacher) questionnaires can also be used to gather information on education and career related activities and attitudes.

Follow-Up Surveys of Graduates . Students can be followed up at one or more points after graduation to determine whether they go on to some form of college or work, and how they do in these pursuits. Such follow-up surveys of graduates are one of the most powerful types of information by which to judge the effects of an SLC or Academy as they give a picture of the lasting effects of these approaches.