Learning By Doing Career Acdemies

David Stern, Christopher Wu, Charles Dayton, and Andrew Maul
Career Academy Support Network
University of California, Berkeley
November 19, 2005

Prepared for a conference on school-to-work research and practice,
Russell Sage Foundation, New York City, May 17-18, 2004.

We are grateful for suggestions from James Kemple, David Neumark, Terry Orr, James Rosenbaum, other conference participants, and anonymous reviewers. We also want to express our continuing admiration for the teachers, administrators, community partners, and students who seek, and sometimes find, ways to make high school more worthwhile. We hope our work will support theirs. As always, we retain the authors' responsibility for any shortcomings in this paper.

I. Introduction

II. Career academies: definition, history, and research results

III. Describing student composition and performance in a single academy

IV. Do academy students perform better, controlling for prior performance and other factors?

V. Are results related to degrees of implementation? Introducing ìcourse metricsî

VI. Results of hierarchical modeling across sites

VII. Conclusions and cautions

References

Tables

I. Introduction

This paper's title¹ summarizes our two main aims. For a number of years, we have been helping high schools and districts that are attempting to create or improve career academies. Our assistance includes developing the schools' own capacity to keep track of results for students, relying mainly on information that is ordinarily available from student transcripts. We believe it is essential for schools themselves to continuously gauge results of career academies (or other educational programs), because even if career academies (or other programs) have been found to be effective somewhere, they are unlikely to be effective everywhere. Therefore, one purpose of this paper is to describe how schools can self-monitor, and what they can learn in the process of operating career academies. Second, we wish to add to general knowledge about career academies, by testing some ideas about why they may be more effective in some places and times than in others. Specifically, we conduct a cross-site analysis to explore whether sites that implement certain features of the academy model to a greater degree also obtain bigger gains in performance of academy students. Both parts of the analysis in this paper focus on student performance during high school, not post-high school outcomes such as employment or college attendance.

Longevity, and solid evidence of effectiveness, distinguish career academies from other programs, practices, and policies that were bundled under the ìschool-to-workî or ìschool-to-careerî label in the U.S. during the 1990s. The first known career academies were started in 1969 ññ well before more recent initiatives like tech-prep or job shadowing, though much later than co-op or vocational education (Stern et al. 1995). Relatively persuasive evidence of career academies' effectiveness comes from several quasi-experimental studies, and especially from a random-assignment evaluation conducted by MDRC. Section II of this paper spells out what we mean by a career academy, and summarizes some of the history and existing research. The paper by Orr and colleagues in this volume provides additional information about the purposes and pedagogical practices of career academies, especially those associated with the National Academy Foundation.

Despite evidence that career academies can produce positive results for students, success is never automatic. Local circumstances may enhance or undermine the effectiveness of a particular career academy, within a particular school at a particular time. The career academy model has several elements, all of which require planning and effort to put in place and keep in place. In 1997 we established a Career Academy Support Network (CASN) to help high schools and districts that want to develop and improve career academies. CASN provides professional development, on-site coaching, and other kinds of implementation assistance.

As part of its service, CASN uses available data from student transcripts to monitor implementation, and to determine whether academy students are improving more or less rapidly than non-academy students in the same high schools. Because of non-random selection of students into academies, and non-random attrition, this analysis is not intended to determine whether academy participation causes greater improvement in students' performance.² But the available data can nevertheless inform academy stakeholders about the kinds of students who are participating and how their performance is changing over time. Sections III and IV illustrate this kind of analysis.

The second question we address in this paper is whether academy students' growth, relative to non-academy students in the same school, is associated with implementation of certain key features of the model. We focus on measures having to do with scheduling and course-taking, because these are considered fundamental to the academy design, and they can be computed from student transcripts. This analysis, using a hierarchical model, is reported in sections V and VI. Although the differences between academy and non-academy students may be attributable in part to non-random selection or attrition, this exercise demonstrates a new procedure for estimating how much implementation matters.

II. Career academies: definition, history, and research results³

Teachers and community groups in Philadelphia first put the essential elements of contemporary career academies into practice in 1969. For a decade or more this structure and related practices were known as the Philadelphia academy model. Before elaborating on the definition and history, we note that teachers had an indispensable role in creating the first such ventures. The subsequent spread and development of this approach also have depended on teachers' initiative. But teachers have seldom been involved in analyzing data on academies. Part of CASN's purpose is to equip the teachers who are responsible for academies to participate in monitoring outcomes for students.

The term ìcareer academyî was coined by Stern, Raby, and Dayton (1992) to describe the kind of high school configuration that originated in Philadelphia, then spread to California, New York City, and eventually nationwide. In 1981 the academy idea was exported from Philadelphia to California, starting with two academies near Silicon Valley. A series of evaluations (ending with Reller 1987) found improved student performance, and spurred California to pass legislation in 1984 supporting 10 replications of the model. The number of state-supported academies eventually grew to about 300 in 2004. Also in the 1980s, New York City created the first Academies of Finance, sponsored by the American Express Company, which subsequently joined with many other companies to create NAF (National Academy Foundation). The foundation added the travel and tourism field in 1987, public service in 1990, and information technology in 1999. As of 2004, NAF supported about 400 academies in about 30 states. The paper by Orr and colleagues in this volume provides additional information about NAF, and reports results from several NAF academies.

In addition to the Philadelphia, California, and NAF academies, many other academies have sprung up independently, and new organizations have sprung up to support them (e.g., the National Career Academy Coalition). Common themes for career academies are health, business and finance, arts and communications, computers, engineering, law and government.

There is no authoritative, uniform definition of a career academy, and as the term has become popular the variation among programs that call themselves career academies has increased.⁴ In 1993 MDRC began the first random-assignment evaluation of career academies (Kemple and Rock 1996). MDRC abstracted three main features to define a career academy:

(1) School-within-a-school organization in which academy students at each grade level take a set of classes together, and stay with the same small group of teachers for at least two years.

(2) Curriculum that includes academic courses meeting college entrance requirements, along with technical classes, all related to the academy theme.

(3) Employer partnerships to provide internships and other experiences outside the classroom, related to the academy theme.

The Philadelphia academies began with a focus on dropout prevention and vocational preparation, but they soon evolved to include preparation for 4-year colleges and universities. The NAF academies have been college oriented since their inception. Our impression from the field is that most career academy teachers explictly aim to prepare students for both college and careers.

Until the 1990s, career academies existed only as separate, small units within larger high schools. For example, a typical career academy might serve 200 students in a school containing 2,000. Since the mid-1990s, however, some schools have converted themselves entirely into groups of career academies or of various small learning communities (SLCs), some of which are career academies. Lee, Ready, and Johnson (2001) conducted an informal national canvass to identify high schools where all students and teachers belonged to small learning environments. They identified 55 such schools, 80 percent of which were using career academies as the model. The Talent Development High School model developed by CRESPAR (Center for Research on the Education of Students Placed at Risk) is an example; every student in grades 10ñ12 belongs to a career academy. On the other hand, Chicago now uses the term career academies to denote entire high schools that have, or did have, a vocational focus -- these may not fit the definition we are using.

Research evidence on career academies . Six different researchers or research teams, using longitudinal data from different sets of academies between 1985 and 2000, compared performance of academy and non-academy students from the same high schools. These studies are described and summarized by Stern (2003). None of these studies assigned students at random to the academy or non-academy groups. Most researchers used regression techniques to control for some observed differences between academy and non-academy students. In some studies the researchers and teachers chose each comparison student individually to match one of the academy students.

All of these studies reported academy students performing significantly better while in high school. Salient findings are as follows:

• Reller (1984, 1985) studied the first two career academies in California. Academy students earned significantly more course credits than a matched comparison group from the same high schools. One-year attrition rates ranged from 2 to 6 percent in academies, 10 to 21 percent in the comparison group.

• Stern et al. (1988, 1989) studied 10 state-funded academies in California. Academy students tended to perform significantly better than matched comparison groups from the same high schools in attendance, credits earned, average grades, and likelihood of staying in school. The three-year attrition rate for the cohort entering in 1985 was 7.3 percent from academies, 14.6 percent from the comparison group.

• Hayward and Talmadge (1995) evaluated 10 different programs that used some form of vocational education to promote success in high school. Two of the sites were career academies. These showed generally better results than other programs, improving students' attendance, credits, grades, and likelihood of completing high school.

• McPartland et al. (1996, 1998) reported on the reorganization of Patterson High School in Baltimore in 1995, which included creation of career academies for all students in grades 10ñ12. Attendance in the first implementation year rose from 71 to 77 percent, contrasting with districtwide decline from 73 to 70 percent in grades 9ñ12. A survey of teachers found great improvement in reported school climate.

• Maxwell and Rubin (1997, 2000) analyzed 1991ñ95 school records for three cohorts of students in grades 10ñ12 in an urban district, including nine career academies. Controlling for other characteristics of students, academy students received significantly better grades than non-academy students, in both academy and non-academy classes. 92 percent of the academy students graduated from high school within the study period, compared to 82 percent of non-academy students.

• Elliott, Hanser, and Gilroy (2000) compared 1994ñ96 data from three Junior Reserve Officers Training Corps (JROTC) career academies in large cities with student data from other career academies or magnets in the same or similar schools, and data on JROTC students not in academies and on students not in any academy or magnet program. Propensity-weighted regression was used in an attempt to offset possible selection bias. Students in JROTC career academies and in other career academies or magnets generally received higher grades, had better attendance, completed more credits, and were less likely to drop out than statistically similar students not in academies.

Three of these studies also gathered data on post-high school outcomes. Salient findings were:

• Reller (1987) surveyed students 15 months after graduation. She found 62 percent of academy graduates were enrolled in postsecondary education, compared with 47 percent of the comparison group. Expecting to complete a bachelor's degree or more were 55 percent of academy graduates, 22 percent of the comparison group. Another survey, 27 months after graduation, found no significant differences between academy and comparison students in employment status, wages, or hours worked.

• Stern, Raby, and Dayton (1992) conducted follow-up surveys 10 and 22 months after graduation. They found no consistent differences between academy and comparison graduates in postsecondary attendance or degree aspirations. Academy graduates on average were working 3 more hours per week, but there was no consistent difference in hourly earnings.

• Maxwell and Rubin (1997, 2000) found no significant differences in wages or hours worked between former academy and non-academy students, though former academy students more often said their high school program had prepared them well for further education and work. However, participation in postsecondary education was higher among former academy students: 52 percent were attending four-year colleges, compared to 36 percent of the non-academy students. In a subsequent study of students at a public university campus, Maxwell (2001) found that academy graduates were less likely to need remedial coursework, and were more likely to complete bachelor's degrees, compared to statistically similar graduates from the same district.

In sum, different researchers studying different sets of career academies have consistently found that academy students are outperforming their non-academy schoolmates on various measures of academic success while in high school. Post-high school differences in further education and employment have been less consistent, but where significant differences have been found they have favored the academy students.

Although these studies statistically controlled for measured differences between academy and non-academy students, the possibility remains that unobserved differences account for the results. For example, the academy students may have been more motivated or better organized to begin with, and that may explain why they did better. Indeed, there might be a systematic tendency for more motivated or better-organized students to join career academies.

The classic method for eliminating bias due to selection or self-selection is for researchers to use a random procedure to assign some subjects to the ìtreatment,î and the others to the control group. This procedure ensures that the average difference between the two groups will be negligible, given a large enough sample. Despite this great advantage, there are well-known drawbacks and limitations to using random assignment in educational field studies. Denying a beneficial treatment to a control group may raise ethical questions. Some important educational variables ññ e.g., completing high school ññ cannot be experimentally manipulated. Even when random assignment is feasible and ethical, absence of a placebo means that Hawthorne effects and other biases may influence the result. Nevertheless, a well-designed random-assignment study can eliminate the problem of selection bias that plagues much educational research. (For a fuller discussion, see Stern and Wing 2004.)

In 1993 MDRC began its random-assignment evaluation of career academies, initially with ten sites, but one academy ceased operating. All nine remaining academies are in high schools with large proportions of low-income and minority students. Each was the only career academy in its high school.

At the start of the MDRC evaluation, the academies recruited more applicants than they could accommodate. Applicants knew they might not be admitted. MDRC randomly assigned about two-thirds of the applicants to the academy; the others became the control group. For more than ten years since the evaluation began, MDRC has collected student records, surveyed students during each of their high school years, and conducted follow-up surveys one year and four years after high school.

During the high school years, the career academies studied by MDRC produced several positive impacts on students' experience and achievement. Compared to the control group, academy students reported receiving more support from teachers and from other students (Kemple 1997). They were more likely to combine academic and technical courses, engage in career development activities, and work in jobs connected to school (Kemple, Poglinco, and Snipes 1999). As of spring of senior year, academies retained a larger fraction of the students whose initial characteristics made them more likely to drop out (Kemple and Snipes 2000). Among students at less risk of dropping out, academies increased participation in technical courses and career development activities without reducing academic course credits (Kemple and Snipes 2000).

The first follow-up survey, one year after scheduled graduation, found no significant impacts on students' high school completion, GED acquisition, or participation in postsecondary schooling. It also showed no significant impact on employment or earnings, though students who had been assigned to career academies were working and earning somewhat more than the control group (Kemple 2001).

MDRC's most recent follow-up, about four years after scheduled graduation from high school, found large and significant impacts on employment and earnings, and no difference in educational attainment (Kemple and Scott-Clayton 2004). In the full sample, students assigned to career academies earned higher hourly wages, worked more hours per week, had more months of employment, and earned about 10 percent more per month than the control group. All these differences occurred for both males and females, but they were not statistically significant for females. Impacts on high school completion or postsecondary education were not significantly positive or negative for the sample as a whole or for any subgroup, but Kemple and Scott-Clayton (2004) note that both the academy and control groups had high rates of high school completion and postsecondary enrollment compared to national (NELS) data on urban high school students.

III. Describing student composition and performance in a single academy

Evaluations of career academies have found that they can produce positive impacts on students, but it does not follow that every academy does produce such impacts. The career academy model is not a sure-fire recipe for success. Much may depend on the particular people involved, local circumstances, and the degree of implementation. To know whether a career academy is yielding the desired results, therefore, it is necessary to keep collecting the data. In sections III-IV we illustrate the kind of information that can be obtained from student records.

Student transcript data have the great virtue of being readily available in all schools, but they also have serious limitations. Generally, they are messy and full of anomalies. For example, it is common to find students on the rolls despite having zero attendance for the year. We eliminated such students from our analysis. We kept students who had 50 or more days of attendance, even though some of these are only marginally involved in the school. Since state funding is determined by attendance counts, attendance may be one of the more accurately recorded variables. Credits earned, grades, suspensions, and demographic data may be subject to even greater inaccuracies. We weeded out anomalies we could find, but we had to take most of the data at face value. Despite these cautions, we think it is worth trying to find out what schools might be able to learn from the data in their own files.

Are academy students a representative cross-section of the school? One of the first questions raised by academy teachers and their non-academy colleagues is whether students in the academy are different from the rest of the school. This is a contentious issue. Non-academy teachers would resent an academy that recruited only the school's ìbest and brightestî students. More generally, as many large high schools are now grouping students and teachers into smaller learning communities (SLCs), inequitable distribution of students among these smaller units can be cause for divisiveness that undermines the strategy (Muncey and McQuillan 1996). In order to know whether students are being assigned to academies or SLCs in some unfair way, schools are beginning to develop routines that enable them to monitor the data.

For example, a series of simple bar charts (not presented here for lack of space) show ethnic and gender data from a biotechnology academy and its home high school, which we call Wyles HS. Grade 10 is the first year of the three-year biotech academy sequence. In 2002, Hispanic students were over-represented in this academy, and Asian students were under-represented, relative to non-academy tenth graders. But this imbalance disappeared a year later in grade 11. At the same time, the academy enrolled a disproportionately large number of females in both years. The gender imbalance may or may not be considered problematic, but the data at least call attention to the fact. Similar analysis can be done on other student characteristics that are usually recorded in district records, including proficiency in English, eligibility for subsidized lunch, and participation in special education.

Student mobility . An important fact of life in schools is that students come and go. Keeping track of student mobility is difficult, but it may be a crucial part of the story. For instance, an academy or SLC that pushes out low-performing students would not be considered as effective as one that retains those students and improves their performance. As high schools face growing pressure to raise test scores, and as more high schools divide students and teachers into smaller groupings, the temptation to de-select low-performing students may increase.

Table 1 illustrates how we can account for student mobility semester by semester. We are especially concerned about students reported as leaving the school district for unknown reasons. Many of these may have dropped out of school entirely, although others may have transferred to schools outside the district without informing anyone in the district where they went. The number leaving the district for unknown reasons therefore can be considered an upper bound on the number of dropouts.⁵ Dividing that number by the number of students who ever enrolled gives an upper bound for the dropout rate. The number ever enrolled includes students who were there at the beginning of the period plus those who transferred in. During the two years covered in Table 1, no biotech academy students left the district for unknown reasons, but a total of 198 (= 107 + 91) non-academy students did. These 198 are 32 percent of the non-academy students ever enrolled in this cohort.

Table 2 summarizes results of similar computations for several other academies (school names are pseudonyms). In most of those cohort comparisons, academies had lower proportions of students leaving the district for unknown reasons, compared to non-academy students at the same schools and grade levels. In many cases the differences are quite large. Academies, like other SLCs, are intended to encourage stronger ties among students, and between students and teachers, based in part on their shared interest in the academy theme. The results in Tables 1 and 2 may reflect the holding power of those stronger relationships and shared interests. On the other hand, it is also possible that the academies initially selected students who were already more likely to stay in school.⁶

Year-to-year changes in attendance, credits, and grades . Whether students stay in school is an important outcome in its own right. In addition, differences in the rate of attrition between academy and non-academy students also affect the interpretation of other results. On average, students who leave the district for unknown reasons are probably not doing as well in high school as students who stay, so comparisons of performance trends between academy and non-academy students are probably biased against the academy. In other words, performance trends for the academy would look better if more low-performing students left the academy. As we compare trends over time in various student outcomes, it is important to keep this possible attrition bias in mind.

Comparing academy students' performance with their own previous performance, and making that same comparison for non-academy students at the same grade levels, is a simple but instructive procedure schools can use to gauge an academy's results over time. Attendance, credits earned, and grades are three outcomes that schools measure periodically and record in students' transcripts. For example, Table 3 shows average attendance and credits earned during tenth grade by the cohort of academy and non-academy students. Table 3 also shows attendance and credits earned by those same students during ninth grade, before the biotech academy students had entered that program. Both attendance and number of credits declined during tenth grade, for both academy and non-academy students. The biggest drop was in credits earned by the non-academy group. Academy students declined slightly more than non-academy students in attendance, but slightly less in credits earned. Again, if attrition among academy students had been the same as among non-academy students, these average performance measures for academy students at the end of grade 10 would probably have been higher. In other words, it is likely that holding on to more students lowered the academy's averages.

IV. Do academy students perform better, controlling for prior performance and other factors?

It is apparent from Table 3 that students who entered the biotech academy in grade 10 had compiled better records of attendance and credits earned in grade 9, compared to non-academy students at the school. Including students' prior performance in the analysis is necessary to gauge whether academy students are doing better than expected, compared to non-academy students. In addition, the analysis also can control for other student characteristics.

Table 4 shows results from a series of regressions predicting cumulative GPA at the end of grade 10 for these same cohorts of academy and non-academy students at Wyles HS. In model 1, the only predictor is GPA at the end of grade 9. As expected, the coefficient on grade 9 GPA is positive and highly significant, and this remains true in subsequent models. Model 2 adds demographic characteristics to the list of predictors: gender, age, and whether the student is African American, Asian, or Hispanic (whites are the comparison group). Only the coefficient on gender is significant, and it indicates that males are receiving lower grades, controlling for the other predictors in the model. The coefficients on demographic characteristics also do not change much in subsequent models. Model 3 adds a variable indicating whether a student was in the academy or not. A positive coefficient would mean that academy students on average are receiving better grades than non-academy students, holding constant the other predictors. In this case the academy coefficient is positive, but barely so, and it is far from significant statistically. Finally, model 4 adds a multiplicative interaction between the academy variable and grade 9 GPA. A positive coefficient here would mean students with high GPA at the end of grade 9 are receiving higher grades in grade 10 if they are in the academy. In Table 4 this coefficient turns out to be negative, but not statistically significant.

We conclude from Table 4 that students in their first year of the biotech academy in 2001-2002 did not show significant improvement in grades relative to non-academy students, on average. And this is true whether GPA in the prior year was high or low.

We have estimated the models in Table 4 with data from various schools and districts. Tables 5 and 6 show coefficients on the academy variable from regressions on several cohorts in this and other academies, using model 3 from Table 4 to predict each year's GPA, attendance, credits earned, and suspensions. In predicting GPA, the academy coefficient is significant in five out of those 26 cases, and negative in four of those five. In the other 21 cases, compared to non-academy students in the same schools and grade levels, academy students on average are not getting higher or lower GPAs than would be predicted based on their prior GPAs and demographic characteristics. In three out of 22 regressions predicting attendance, the coefficient on academy participation is significant and positive. Of the 21 regressions for credits earned during the year, the coefficient on academy participation was significant and positive in one case and significant and negative in three. Finally, none of the 21 regressions run for suspensions revealed a significant coefficient on the academy variable.

Our intent in sections III and IV has been to illustrate how teachers, administrators, and community partners can ìlearn by doingî career academies. Transcript data reveal whether academy students represent a cross-section of the school in terms of gender, race, ethnicity, or other characteristics. The same data also can be used to calculate attrition rates, and to monitor the performance of academy students over time, relative to non-academy students in the same school. We have seen that the academies analyzed here have relatively few students leaving the district for unknown reasons. If students who leave the district for unknown reasons tend to be lower-performing, then holding on to more of these students may reduce the apparent effectiveness of academies on other measures. In spite of that, several of the academies analyzed here do show some relative gains in student attendance, which can be seen as another indicator of academies' holding power.

Students' gains in attendance, credits earned, GPA, or (reducing) disciplinary suspensions generally do not differ significantly between these academies and the rest of their host high schools. In a few academies, the ìvalue addedî in terms of GPA or credits earned appeared to be significantly negative. Teachers in these situations would want to probe for explanations and, if the data are accurate, determine what corrective action to take.

V. Are results related to degrees of implementation? Introducing ìcourse metricsî

In addition to informing teachers and other stakeholders in individual academies, we are also interested in questions that involve comparisons across sites. In particular, we would like to know whether outcomes for academy students are related to how thoroughly the academy model is implemented. To explore that question, we have devised several new measures of implementation, which this section explains. In the following section, we present results from a hierarchical model testing whether those implementation measures are associated with student outcomes.

One of the key features of career academies, like some other SLCs, is that academy students at a given grade level are scheduled to take a set of classes together. Typically, academy sections of English, social studies, science, and a technical or lab class related to the academy theme are scheduled so that academy students can take them together. This allows teachers to design lessons, projects, and assignments that connect some of the different subjects the students are taking. In contrast to the usual high school schedule, the intent is to create more intellectual coherence among the various concepts and bodies of knowledge students are learning. Having a group of students take a set of classes together also adds to social cohesion among students, and with the teachers who share responsibility for this group.

This kind of cohort scheduling is a major departure from standard practice in American high schools.⁷ Scheduling academy students into academy classes therefore requires persistent effort by the academy teachers and school administration, especially given the varying scheduling systems and constraints in school districts. As a result, the actual implementation of this scheduling design is seldom perfect.

We have developed three ìcourse metricsî to indicate how closely students' actual course-taking comes to the academy model. We usually compute these measures for one year at a time, but they could also be calculated by semester or for more than one year.

• Course-taking : the percentage of the academy courses in which academy students actually enrolled, on average. For example, suppose the intended academy course sequence includes four classes during sophomore year, but because of scheduling conflicts the average academy student ended up enrolling in only three of these classes. Our course-taking measure would then be 3/4 = 75 percent. A low number here means academy students have actually experienced less of the intended academy curriculum.

• Course purity : the percentage of students enrolled in the average academy class who are actually academy students. For example, if academy classes on average have 30 students and 24 of these are academy students, course purity would be 24/30 = 80 percent. A low number on this index means it is more difficult for teachers to connect lessons or assignments from different classes, because the different classes do not contain the same set of students.

• Course coverage : credits earned in academy courses as a percentage of all credits earned by academy students, on average. For example, suppose the academy schedule called for academy students to take four classes during tenth grade, and that this actually happened. If the total course load all students took was six classes, then course coverage would be 4/6 = 67 percent. This measure was first used by McMullan, Sipe, and Wolf (1994) to indicate how big a part the academy or SLC curriculum represents in the total course load.

These three measures are independent in the sense that knowing two of them does not allow you to compute the third. Conceptually, low course coverage together with high course-taking would indicate that the academy is designed to offer only a relatively small number of classes, but the average academy student is successfully scheduled to enroll in a high percentage of those classes. To take another scenario, low course-taking and high purity would mean that the average academy student misses out on a substantial part of the intended academy curriculum, but few non-academy students are enrolled in the academy classes.

Although individual students do have some degree of choice about which courses they take, we view these course metrics primarily as the result of administrative structures and processes within the school, not as the outcome of individual student choices. The third metric, course coverage, is limited by how many courses are included in the academy sequence; this is a school decision, not a student decision. The second metric, purity, is determined by how well-organized the school's scheduling process is ñ in some schools it can be chaotic! ñ and by the priority given to academy membership when the school's master schedule is constructed. Similarly, the first measure, academy course-taking, depends on the degree to which schedulers try, and are able, to avoid forcing academy students to choose between taking an academy core course and enrolling in some other course they must take.

In our sample, all but one of the 22 academy cohorts have values within the [0.67, 1.0] range for course-taking, [0.55, 1.0] for purity, and [0.22, 0.57] for coverage. The standard deviations among all 22 cohorts are 0.14, 0.19, and 0.12, respectively. These measures do indeed vary, but does that matter?

VI. Results of hierarchical modeling across sites

To test whether our measures of academy implementation are associated with improvement in academy student performance, we shift to a hierarchical modeling framework (Bryk and Raudenbush 1992). Individual students are the units of observation at level one. The level-one predictors are the prior year outcome for that student, along with gender, age, and race or ethnicity. At level two the unit of observation is a cohort of students who belonged to a particular academy, or to the non-academy part of the high school, at a particular grade level in a particular year. The analysis reported here involves 22 academy groups for which we have the necessary data (from seven academies⁸), and 22 non-academy groups, each for the same grade level, calendar year, and high school as the corresponding academy cohort.

Tables 7-10 show results from estimating a sequence of five models for each outcome. The level-one regressions, not shown here, are the same as model 2 in Table 4.⁹ In each of the level-two models reported in Tables 7-10, the level-one intercept is treated as a random coefficient that varies among the 44 level-two cohorts. The first level-two model uses only one predictor: a variable indicating whether the student belonged to an academy or not.

More explicitly, the level-one regression can be written as

y ijt = b 0j + b 1 y ij,t-1 + x' ij b x ,

where y ijt is the predicted value of an outcome (e.g., attendance) for the ith student in the jth cohort during year t; b 0j is the intercept for cohort j; b 1 is the coefficient on last year's outcome for that student, y ij,t-1 ; and b x is a set of coefficients on a student's demographic characteristics x ij .

Coefficients b 1 and b x do not vary across cohorts, but b 0j does. The level-two model in which the academy variable is the only predictor would be written as

b 0j = g 0 + g 1 A j ,

where A j = 1 if cohort j is in an academy, 0 otherwise; and g 1 is the coefficient reported for model 1 in Tables 7-10.

The next three models add the course metrics, one at a time. The last model uses all three course metrics in addition to the academy variable itself. The course metrics are defined only for academies and are given a value of 0 for non-academy students. The level-two coefficients on the course metrics therefore tell us whether and to what extent student outcomes are better in academies that have higher scores on the course metrics.

Tables 7-10 show the estimated coefficients and robust standard errors, corrected for clustering of students within the 22 groups. In Table 7, the academy coefficient by itself is a significantly positive predictor of attendance. But when course metrics are added, neither the academy coefficient nor the course metrics appear significant. In Table 8, no variable is significant in predicting credits earned. In predicting GPA, Table 9 shows a negative coefficient on the course coverage metric, meaning that students tend to receive lower grades in academies where the academy curriculum represents a larger proportion of the total coursework. But when this is the only course metric in the model, this negative association is partly offset by a positive coefficient on the academy variable itself. We do not have an explanation for this finding, other than randomness in the data. Finally, Table 10 suggests that being in an academy may be associated with fewer suspensions ññ a positive result ññ but none of the course metrics is significant.

On the whole, Tables 7-10 do not reveal any consistent association between our measures of academy implementation and outcomes for students. However, it would be premature to conclude from these results that implementation makes no difference. We have measured only a few implementation variables that we could calculate from student transcript data, and these may not be the most important. In 2004 several organizations that provide technical assistance to academies published a set of standards containing ten key elements, each with three to five separate components, amounting to 38 distinct practices in all.¹⁰ Some of these other practices, or interactions among them, may be more important than the three indicators we were able to calculate.

Furthermore, even the three implementation variables we did measure might have more explanatory power if we had data on a larger number of academies. More observations would give us more precise estimates, as would including academies with a wider range of values on our course metrics. Differences in the range we observed may not matter, but bigger differences might.

Finally, the effectiveness of a given career academy may depend less on the absolute degree of implementation than on the contrast between the academy and the rest of the school (see Kemple and Snipes 2000). A perfectly implemented career academy may not help students much if the rest of the high school also offers similar benefits as the academy, but an imperfectly implemented academy may help students a lot if the quality of the surrounding high school is poor. Unfortunately, we did not have the data necessary to compare students' experience in the academy with the rest of the high school.

VII. Conclusions and cautions

This paper analyzed data collected as part of an ongoing project to develop and improve career academies in various high schools and districts. We presented two kinds of analysis: single-site and cross-site. The purpose of the single-site analysis is to inform the efforts of individual career academies. The cross-site analysis explored whether better-implemented academies showed bigger gains in the performance of academy students, relative to their non-academy counterparts.

Section II placed this analysis in the context of earlier studies, which have found fairly consistent and persuasive evidence of positive effects for career academy students at different times and places. However, the fact that academies have produced positive results in some places does not imply that such results are inevitable or automatic. Implementing the career academy model is not easy. That is why several organizations and projects, including CASN, offer help to schools that request it (see footnote 10). As part of CASN's service, we collect student transcript data and report findings back to the teachers, administrators, and others involved. Sections III and IV presented examples of the kind of analysis we give back to schools, to inform the self-guidance of individual academies. For instance, comparing the gender and racial/ethnic composition of academy students with non-academy students in the same school tells the stakeholders whether academy students represent a reasonable cross-section of the school.

The most substantive finding in sections III-IV is that most individual academies have fewer students who leave the district for unknown reasons, compared to the non-academy parts of their high schools. This suggests that academies tend to have stronger holding power, a finding that is consistent with previous research. However, unlike previous studies, our data on 26 different academy cohorts do not show academy students generally receive higher grades than their non-academy peers, when we control for prior grades and demographic variables. Nor do we find relative improvement among academy students in terms of credits earned or disciplinary suspensions during the year. As explained in section III, the stronger holding power of academies probably exerts a downward influence on measures of academy students' relative gains. In spite of that, we do find some evidence that academy students improve relative to non-academy students in terms of attendance. This seems consistent with the idea that academies have stronger holding power than the rest of the school: academy students are less likely to leave for unknown reasons, and they show bigger year-to-year gains in the number of days they actually come to school.

In addition to reporting results that may be used to inform individual academies, we also did a cross-site analysis to explore whether academies that are better implemented tend to show bigger gains in student performance. For that analysis we had to construct measures of implementation. The most innovative piece of this paper is the development of three implementation measures indicating (1) what percentage of the designated academy courses the average academy student actually takes, (2) what percentage of students in the average academy class are actually academy students, and (3) what percentage of the average academy student's total course credits are earned in academy classes. We call these course-taking, purity, and coverage. Results from a hierarchical model did not find significant associations between these measures and student performance. However, we do not view this result as conclusive, because we measured only a few aspects of implementation, had data for only a small sample of academies, observed a fairly narrow range of variation in our implementation measures, and could not assess the quality of academy students' experience relative to the rest of the school, which may be more important than the absolute degree of implementation.

We will end with two cautionary observations. First, the information in school files is not always accurate. It is not collected for research purposes, and may not be checked or audited for accuracy. For instance, we had to exclude from the analysis some students who appeared in the data files despite having zero attendance for the year.

A particularly vexing problem is how to know exactly which students are in an academy during any given term. Accurately identifying academy and non-academy students in the data base is essential for comparing their performance, but schools and districts do not have routine procedures to flag academy students in their data bases. As foundations, along with the federal and state authorities, induce more high schools to regroup students and teachers into academies or other small learning communities (SLCs), accountability pressures will require schools and districts to institute such procedures for identifying students by academy or SLC. Concretely, this means that at the the beginning and end of each term the academy or SLC leader must give the district a list of students. In addition, to construct the kind of course metrics we developed here, it is also necessary to flag every course, or every section of a course, that is designated for students from a particular academy or SLC.

Our second caution has to do with implementation standards. Many school-reform organizations, funders, government agencies, and researchers publish advice about how to improve student performance. Unfortunately, too little of that advice is based on firm evidence. Section VI illustrated one way to test whether particular aspects of implementation are associated with results for students. This kind of analysis is only a beginning. In the particular case of career academies, most previous evaluations have treated an academy as a ìblack box,î offering little or no data on the mechanisms or processes through which academies may affect student performance, or the influence of social, historical, cultural, or institutional context. The MDRC evaluation is an exception (e.g., see Kemple and Snipes 2000); the paper by Orr and colleagues in this volume also measures a range of outcomes related to different features of the academy. More studies of this kind would be highly desirable.

Obtaining desired results from career academies, other kinds of small learning communities, or generally any kind of reform requires a better understanding of how, and under what conditions, a given strategy or approach actually works. Until much more such analysis has been done, researchers and policy makers should be humble about telling schools how to improve.

¹ Readers who are unfamiliar with career academies should not construe the title to imply that career academies themselves are designed to substitute ìdoingî for ìlearning.î To the contrary, as described in the text, most coursework in career academies is in academic subjects, and the intent is to prepare students for college.

¹ This paper also does not attempt to test whether CASN's efforts are producing positive results.

¹ Some text in this section is from Stern(2003) or Stern and Wing (2004).

¹ The state of California provides grants to school districts for ìpartnership academiesî which are defined by statute, but this definition does not apply to the hundreds of academies in California that do not receive state funding. A few other states also have funded such academies. The federal School-to-Work Opportunities Act in 1994 included career academies on a list of seven ìpromising practices,î but did not not define them. Building on the MDRC definition, the Career Academy Support Network ( http://casn.berkeley.edu ) helped negotiate a common definition among several networks currently promoting career academies.

¹ Students who did transfer to other schools are also at greater risk of not completing high school. See Rumberger and Larson (1998).

¹ The possibility that academies select students with lower propensities to move is also consistent with the finding in Table 1 that the academy accepts few transfers from outside the school. On the other hand, the academy did take a substantial number of transfers from the rest of the school.

¹ Cohort scheduling for secondary school students is common practice in other countries. The individualistic, shopping-mall approach to scheduling seems to prevail only in the U.S.

¹ The four missing cohorts are the 2001-02 grades 10 and 11 from Rollo and Blizzard. They were excluded from Tables 7-10 because course metrics were not available for these cohorts.

¹ Results in Tables 7-10 are not exactly comparable with Tables 5 and 6 because the samples are somewhat different due to missing data; and coefficients on student characteristics were free to vary across academies in the equations for which academy coefficients are reported in Tables 5 and 6, but not in Tables 7-10.

¹ The organizations involved were the National Career Academy Coalition, National Academy Foundation, National Center for Education & the Economy, Southern Regional Education Board, Center for Research on the Education of Students Placed at Risk, and CASN. The standards are available at each organization's web site, e.g., http://casn.berkeley.edu/resources/national_standards.html

References

Bryk, A.S. and Raudenbush, S.W. (1992). Hierarchical Linear Models: Applications and Data Analysis Methods . Newbury Park: Sage Publications.

Elliott, M. N., Hanser, L. M., and Gilroy, C. L. (2000). Evidence of positive student outcomes in JROTC career academies. Santa Monica, CA: RAND Corporation.

Hayward, B., and Talmadge, G. (1995) Strategies for keeping kids in school. Washington, DC: U.S. Department of Education.

Kemple, J. J. (1997). Career academies: Communities of support for students and teachers. Emerging findings from a 10-site evaluation. New York: Manpower Demonstration Research Corporation.

Kemple, J. J. (2001). Career academies: Impacts on Students' Initial Transitions to Post-Secondary Education and Employment. New York: Manpower Demonstration Research Corporation.

Kemple, J. J., Poglinco, S.M., and Snipes, J. C. (1999). Career academies: Building career awareness and work-based learning activities through employer partnerships. New York: Manpower Demonstration Research Corporation.

Kemple, J. J., and Rock, J. L. (1996). Career academies: Early implementation lessons from a 10-Site evaluation. New York: Manpower Demonstration Research Corporation.

Kemple, J. J., with Scott-Clayton, J. (2004). Career academies: Impacts on Labor Market Outcomes and Educational Attainment. New York: Manpower Demonstration Research Corporation.

Kemple, J. J., and Snipes, J. C. (2000). Career academies: Impacts on students' engagement and performance in high school. New York: Manpower Demonstration Research Corporation.

Lee, V.E., Ready, D.D., and Johnson, D.J. (2001). The difficulty of identifying rare samples to study: The case of high schools divided into schools-within-schools. Educational Evaluation and Policy Analysis 23(4): 365-379.

Maxwell, N. L. (2001). Step to college: Moving from the high school career academy through the four-year university. Evaluation Review 25(6):619-654.

Maxwell, N. L., and Rubin, V. (1997). The relative impact of a career academy on post-secondary work and education skills in urban, public high schools. Hayward, CA: California State University, Hayward, School of Business and Economics, The Human Investment Research and Education Center.

Maxwell, N. L., and Rubin, V. (2000). High school career academies: A pathway to educational reform in urban schools? Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.

McMullan, B. J., Sipe, C. L., and Wolf, W.C. (1994). Charters and student achievement: Early evidence from school restructuring in Philadelphia. Philadelphia: Center for Assessment and Policy Development.

McPartland, J. M., Balfanz, R., Jordan, W., and Legters, N. (1998). Improving climate and achievement in a troubled urban high school through the Talent Development Model. Journal of Education for Students Placed at Risk, 3, 337ñ361.

McPartland, J. M., Legters, N., Jordan, W., and McDill, E. L. (1996). The Talent Development High School: Early evidence of impact on school climate, attendance, and student promotion. Baltimore: Johns Hopkins University and Howard University, Center for Research on the Education of Students Placed at Risk.

Muncey, D. E., and McQuillan, P. J. (1996). Reform and resistance in schools and classrooms: An ethnographic view of the Coalition of Essential Schools. New Haven, CT: Yale University Press.

Raudenbush, S.W. and Bryk, A.S. (2001). HLM5: Hierarchical Linear and Nonlinear Modeling . Lincolnwood, IL: Scientific Software International.

Reller, D. (1984). The Peninsula Academies: Final technical evaluation report. Palo Alto, CA: The American Institutes for Research.

Reller, D. J. (1985). The Peninsula Academies, interim evaluation report, 1984ñ85 school year. Palo Alto, CA: American Institutes for Research

Reller, D. J. (1987). A longitudinal study of the graduates of the Peninsula Academies, final report. Palo Alto, CA: American Institutes for Research in the Behavioral Sciences.

Rumberger, R.W. and Larson, K.A. (1998). Student mobility and the increased risk of high school dropout. American Journal of Education 107(1):1-35.

Stern, D. (2003). ìCareer academies and high school reform before, during, and after the school-to-work movement.î In Stull, W. J., and Sanders, N. M. (Eds.): The School-to-Work Movement, Origins and Destinations . Westport, CT: Praeger, pp. 239-262.

Stern, D., Dayton, C., Paik, I., Weisberg, A., & Evans, J. (1988). Combining academic and vocational courses in an integrated program to reduce high school dropout rates: Second-year results from replications of the California Peninsula Academies. Educational Evaluation and Policy Analysis, 10, 161ñ170.

Stern, D., Dayton, C., Paik, I., & Weisberg, A. (1989). Benefits and costs of dropout prevention in a high school program combining academic and vocational education: Third-year results from replications of the California Partnership Academies. Educational Evaluation and Policy Analysis, 11, 405ñ416.

Stern, D., Finkelstein, N., Stone, J.R. III, Latting, J., and Dornsife, C. (1995). School to Work: Research on Programs in the United States. Washington and London: Taylor and Francis, Falmer Press.

Stern, D., Raby, M., and Dayton, C. (1992). Career academies: Partnerships for reconstructing American high schools. San Francisco: Jossey-Bass

Stern, D. and Wing, J.Y. (2004). Is there solid evidence of positive effects for high school students? Prepared for a conference on ìHigh School Reform: Using Evidence to Improve Policy and Practice,î organized by MDRC, New Orleans, January 22-23. http://www.mdrc.org/publications/391/conf_report.pdf