Graduation Competency Assessments

May 14, 2008

The debate over state assessments and their use has been ongoing in Pennsylvania since the PSSA test made its debut in 1991. Contention over the purpose of state tests, its administration, cost, uses, and validity have all been voiced by education organizations, parents, students, and other stakeholders within the public education community. Not surprisingly, the current proposal on Graduation Competency Assessments has drawn opposition not only from PSBA but also from 23 separate organizations representing teachers, school district superintendents, school principals, parents, special education advocates, gifted education advocates, civil rights groups and other stakeholders in the public education community. While the list of various issues that opponents of this proposal have is a long one, this testimony will concentrate on four issues, the need for the proposal, the role of school entities in assessment and graduation, its likely effect on students, and its cost.

Are Graduation Competency Assessments necessary?

Proponents for GCAs argue for the need for change with the following statements:

PSBA believes that the assertions made by proponents of the proposal are false. A failure to attain proficiency on the PSSA test does not guarantee that a student is not prepared for what follows high school, be it the workforce or higher education. The state’s own PSSA validity study (HumRRO), which reviewed students at three Pennsylvania universities, shows that several thousand students in those universities who scored basic on the PSSAs in fact went on to college with no need for remedial classes.  Among all three universities 58.7% of students who scored basic or below on the PSSA tests took at least the standard level Math or English college course.  That is, most students who “failed” the PSSA enrolled in nonremedial college courses in the same subject area(s) in which they failed.6 

PSBA also questions the efficacy of adding a new battery of  standardized state tests as an option for students to demonstrate proficiency on state standards if, as proponents state, students are not prepared to graduate by the spring of their senior year. Specifically, assuming that there are a number of students unprepared to graduate by that time in their high school career, how does an additional testing option help remedy the situation? It seems to PSBA that the problem of students being unprepared to graduate after four years of high school would likely be caused by gaps or inadequacies in the quality of instruction leading up to the senior year, not by the current system of state or local assessments. Offering GCA tests as the solution to helping students graduate who are not prepared to do so continues the very type of system that proponents are seeking to remove, that is providing students with a false belief that they are prepared to graduate simply because they can pass a test, when they may not be prepared at all.  

Proponents’ assertions also promote an unspoken belief that local assessments aligned with state academic standards and the PSSA, which are currently allowed and being used extensively throughout the state, are inadequate. PSBA contends that proponents have no proof that such tests fail to measure student achievement accurately or that such tests allow those who are not proficient on the state academic standards to graduate.

In order to better illustrate the local assessment practices of school districts and other school entities, PSBA sent a survey to all school districts and area vocational-technical schools in January 2008 asking them to identify local assessment practices and their alignment to state standards and the PSSA.

This survey was mailed out to 522 school districts and area vocational-technical schools. PSBA received responses from 238 school entities, or about 46% of those receiving the survey.

The survey asked school entities about their current graduation requirements. The survey found that school entities were split on the question of whether or not a minimum score of proficiency on the PSSA test was required for graduation, with 46% answering that it was required and 54% answering that it was not required. Only 4% of districts (8) that answered the follow-up question on how they deal with students who do not score proficient on the PSSA responded that they would withhold a diploma from a student that did not score proficient on the PSSA test. A total of 145 school entities, or 73% of those answering the follow-up question, responded that students scoring below proficient on the PSSA could use local assessments as an alternative means of demonstrating proficiency to earn their diploma. The remaining 23%, or 46 school entities, responded that only students with IEPs who score below proficient but achieve the goals and objectives of their IEPs are permitted to graduate.

 A total of 84 respondents (35%) indicated that their school entity only considers the results of local assessments as a graduation requirement. A total of 157 respondents (66%) said that they do not consider only the results of local assessments as a graduation requirement.

A total of 416 responses were given when entities were asked to describe the nature of local assessments used as graduation requirements. The majority, 230 responses, or 55.29%, indicated that such tests are developed by individual teachers, or by departments or grade level teams. A total of 66 respondents, or 16%, responded that they used tests available through publishers of textbooks used by the school entity or other instructional resources. Thirty respondents, or 7%, indicated that they used nationally-available standardized achievement tests and 74 school entities (18%) indicated that they used other types of assessments. In this “other” category, the most common type of assessments used was “4-Sight” and Study Island, although many different assessment types were listed by respondents.

School entities that responded that their local tests were developed by individual teachers or departmental or grade level teams were asked to respond a follow-up question on the development of their tests. While the responses to this question varied, the common attributes in most of the items submitted mentioned the use of specific appropriate state standards, assessment anchors, the NAEP and PSSA questions to develop assessments that are reflective of the school’s curriculum.

Entities responded that they ensured alignment with the state’s academic standards and PSSA in a variety of ways. These include indexing questions on the local assessment to a state standard and/or anchor, alignment with state standards through a format developed during strategic planning, aligning assessments with course and curriculum mapping, item mapping, aligning local assessments with the PSSA based on the percentage of questions on the PSSA in each academic area, reviewing curriculum in each academic subject, and through the work of employees receiving  professional development in curriculum and assessment alignment.

Assessment strategies used as part of the local assessment system include locally developed tests (24%), course grades (20%), locally developed project or other performance assessments (19%) and student portfolios (14%). A discussion on the use of assessment strategies is included later in this paper.

On average, respondents reported a cost of $154/student to develop, administer and score local assessments. Respondents indicated that on average, they spend $53/student on costs related to the implementation of local assessments.

Respondents that indicated that they use nationally-available standardized tests indicated that they purchase tests from Success for All, Harcourt and CTB-McGraw-Hill most often. The most common tests fitting this description are 4-Sight and Terra Nova. A total of 81% of those using national tests indicated that the test developer ensured that the assessment purchased is aligned with Pennsylvania’s academic standards. Others (10%) indicated that while there was no reference to the state’s academic standards, the assessment was based on local curriculum, which is aligned to the state’s standards. Respondents reported an average cost to purchase these tests of $57/student, with an additional average cost of $26/student to implement those assessments.

On the issue of remediation, 192 respondents (81%) indicated that they require remediation for students who score basic or below basic on the PSSA or local assessment. As indicated, remediation efforts include Study Island courses, tutoring, use of PLATO learning software, use of an individualized data-based grade level learning plan system that includes remediation through other school programs, summer school, additional coursework and software programs, after-school and Saturday sessions, remediation built into student schedules, Math +  a half/Reading + a half programs, web-based tutorial remedial based programs and small group instruction. Respondents also indicated that study halls and other available time are used for remediation efforts.

On the issue of costs, respondents were asked to rank 10 different areas in the order in which they anticipated new costs arising from the GCA proposal. The top five results are listed are:

• Remediation, other than summer programs, in each of the 10 areas covered by the GCAs (1,411 total responses)
• Expanded summer programs for remediation (1,058 total responses)
• Staff/scheduling/remediation for students with disabilities who would not be exempted by their IEP to take these 10 tests (1,033 total responses)
• GCA test administration three times a year (1,003 total responses)
• Staff time for professional development (971 total responses)

Survey conclusions

The data from the PSBA survey make it clear that, where graduation assessments are concerned, it is not a matter of using the PSSA as a graduation requirement or not. Rather districts appear to use a continuum, part of which is the 11th grade PSSA, followed by remediation for those students not scoring proficient and then participation in the 12th grade PSSA re-test or passage of a local assessment.

PSBA conducted this survey to show that school entities throughout the state spend considerable time, effort and resources into the development and alignment of local graduation assessments. The association believes strongly that these efforts should not be summarily discounted by proponents of the GCA proposal as being ineffective or less rigorous than state assessments.

The undeniable facts are that districts use a variety of tests and means of aligning those tests with state academic standards. PSBA does not assert that the survey results prove that all school districts are following the current high school graduation regulations, but we believe it demonstrates that there are many school districts where the local assessment is developed and aligned as required under the current regulations.

Graduation, curriculum and local control

Setting graduation requirements and curriculum has historically been the province of local school districts. The Public School Code, in sections 1611 and 1613 assert that school districts have the authority to confer degrees and certificates to students who complete mandatory courses of study.   The very same Chapter 4 that is being proposed for amendment affirms local control over design and planning of curriculum. Portions of Section 4.4. (General policies) read as follows:

(a)  It is the policy of the Board that the local curriculum be designed by school entities to achieve the academic standards under § 4.12 (relating to academic standards) and additional academic standards designated in strategic plans under §  4.13 (relating to strategic plans) (emphasis added).

(b)  It is the policy of the Board that local school entities have the greatest possible flexibility in curriculum planning consistent with providing quality education and in compliance with the School Code, including requirements for courses to be taught (24 P. S. § §  15-1501 and 16-1605); subjects to be taught in the English language (24 P. S. §  15-1511); courses adapted to the age, development and needs of the pupils (24 P. S. §  15-1512); minimum school year of 180 days and minimum of 900 hours of instruction at the elementary level and 990 hours of instruction at the secondary level (24 P. S. § §  15-1501 and 15-1504); employment of sufficient numbers of qualified professional employees (24 P. S. §  11-1106) and superintendents to enforce the curriculum requirements of State law (24 P. S. §  10-1005); and this part (emphasis added).

PSBA also believes that the GCA proposal would severely limit the number of methods by which students could demonstrate proficiency for graduation. While the proposal purports to continue to allow a local assessment as a permissible demonstration of proficiency, such assessments would have to be validated using the criteria contained in the proposal. These include: 1) Assessments are internally consistent and replicable; 2) assessments adequately measure and are aligned with the academic content specified in the state academic standards assessed by the GCAs; 3) level of difficulty of assessment items is greater than or equal to those assessed on the GCAs; 4) proficiency level cut scores are greater than or equal to that of the GCAs; 5) results of local assessments correlate positively and significantly with related national and state criterion referenced assessments; 6) test administration, security and scoring regimes ensure that the integrity and validity of the local assessment is maintained and 7) policy for annually updating assessment items ensure compliance with criteria previously listed.7

The proposal is also clear that school entities would have to pay the cost of test validation. 8  A spokesperson for CTB/McGraw Hill, a leading testing company had the following to say regarding the validation requirements contained in the proposal:

“To do the [validation] study correctly for 10 GCA's with a small sample size and to conduct onsite training for content validity and alignment for 10 tests would probably be cost prohibitive for any one regular sized school district. 9

It is clear in the proposal that school districts have to develop multiple local assessments to use as graduation requirement. The proposal, in §4.24(b)(iv)(C), the language that describes the graduation requirements beginning in 2013-14, mentions “[L]ocally administered, validated criterion referenced assessments comparable to the GCAs (emphasis added).

Likewise §4,24(b)(v)(2) references validation of local assessments (emphasis added). Along with the costs of validation, the mere development of multiple local assessments would likely put severe financial and administrative burdens on all but the wealthiest and largest school districts.

Not only would the prospect of using local assessments be expensive, any district going through the extensive validation process would find that it would result in local assessments closely resembling the GCAs. Because the validation criteria include provisions that would require alignment with academic content specified in the state academic standards assessed by the GCAs, level of difficulty of assessment items greater than or equal to that of the GCAs and proficiency level cut scores greater than or equal to that of the GCAs, the resulting local assessment would greatly resemble a graduation competency assessment.

Because of these costs associated with continuing to use a local assessment for graduation purposes under the proposal, and the likely result that such assessments would likely, as a result of the mandated validation process, look identical to a GCA, PSBA asserts this option, for all intents and purposes, has been removed as a viable alternative and cannot be counted as a realistic option for school entities.

The proposal also allows the use of certain Advanced Placement and International Baccalaureate exams as alternatives to the PSSA and GCA for demonstration of student proficiency. However, the College Board reports that only 18% of Pennsylvania graduates in 2007 took at least one AP examination in their 4 years of high school.10 Anecdotal evidence from school administrators indicates that International Baccalaureate programs are even scarcer. Consequently, for most Pennsylvania seniors, the only acceptable methods of proving proficiency under the proposal will be through the PSSA or the GCAs. There will be no practical local measurement of proficiency permitted.

Effect of GCA proposal on students

As stated previously, while the GCA proposal will provide another option for students to demonstrate proficiency in the state academic standards in order to graduate, it will remove the local option, for all practical purposes, and provide two other options, the Advanced Placement and International Baccalaureate tests, that are not available to most students.

More importantly, the proposal eliminates the use of various assessment strategies that are currently required for use in local assessments by school entities. The current regulations provide that the local assessment system, of which the graduation assessment is a part, must be designed to include a variety of assessment strategies which may include:

On the other hand, the proposal requires local assessments to be “validated criterion referenced assessments comparable to the GCAs.”12 That is, a pencil and paper written test.  There are no changes to §4.51 (State assessment system) that would allow for the use of different assessment strategies. Consequently, students that are not good test takers will have more difficulty in demonstrating proficiency in order to graduate.

Proponents of the proposal argue, however, that because students can continue to take the GCAs until they pass them, the pressure will be removed from students who do not do well on standardized tests. PSBA counters that the pressure on students will not be removed but will increase because in addition to facing the prospect of not being able to graduate, students will face the prospect of staying in remediation classes, perhaps missing or being unable to schedule desired or required classes while remediation takes place. 

Finally, several responses from PSBA’s survey indicate that many school entities are concerned with the additional loss of instruction time to preparation for and administration of GCAs. When added to the instruction time already lost due to the prep time and administration of PSSA, many school entities are concerned that the emphasis placed on testing and results of testing are beginning to overtake the importance of a well-rounded education and depth of instruction. 

Proposal costs

The Regulatory Analysis Form accompanying the proposal declares that “districts will incur savings of approximately $8.0 million annually resulting from the adoption and use of the voluntary model state curriculum. These savings result from “reduced staff time, consultant fees, materials, research and development costs, etc. In addition, the Board estimates that districts will save in excess of $220,000 annually in staff time, copy costs for the development and administration of individual teacher, school or district wide final course exams.”13

Of course, in order to make these statements regarding potential savings, an assumption has to be made that a certain number of districts will have to use the voluntary model curriculum and the GCAs to replace its current examinations. The Board estimates that 250 school districts initially will use the model state curriculum. No estimate is given on how many school districts would replace their current examinations and use the GCAs as graduation requirements. It would be interesting to see how the State Board made the estimate on the use of the voluntary model curriculum. Even if their estimate is accurate, would school districts simply be able to dismiss curriculum development staff or would they retain such staff and use them in a different capacity?

The Regulatory Analysis Form also states that “[L]ocal government will face no additional costs associated with compliance.” PSBA believes that this statement is overly optimistic.

Districts will see an increase in remediation costs. The current regulations, under the requirements for local assessment systems, require school entities to provide “assistance to students not attaining academic standards at the proficient level or better…”14 The proposal adds to that requirement by mandating that school entities provide “supplemental instruction to a student who does not score proficient or above on a PSSA administered in 11th grade or GCA administered in any grade.”15  Because the proposal requires students to be proficient in all state standards that are not assessed by a state assessment and it allows proficiency on these standards to be assessed, presumably, through a local assessment, school entities potentially face additional costs for remediation for students not scoring proficient on these newly required assessments. Additionally, the new requirements for proficiency in science on the PSSA and proficiency on at least one social studies GCA will add to remediation costs as well.  The development of additional assessments to test proficiency on standards not assessed on a state assessment will also add costs.

Also, districts will have to develop extensive recordkeeping systems to match students with the GCAs that have been successfully completed and those for which remediation will be necessary in addition to the various modules that have to be taught in remediation classes and to ensure that student needing remediation in certain modules are enrolled in the proper classes.

 Districts will also experience increases in costs for professional development on the implementation and administration of GCAs. Interestingly, the State Board correctly identifies remediation and tutoring as one of the allowable uses of Accountability Block Grant funds and promotes the availability of these funds for uses to help cover increases in costs caused by the proposal (professional development is another allowable use of funds , but is not cited), however information from the Department of Education shows that tutoring and professional development are two of the lesser utilized uses of the ABG monies, giving way to such things as full day kindergarten, pre-kindergarten and smaller class sizes.16 In fact, changes to the ABG program in 2007 provided incentives to school districts to use their ABG grants for early childhood programs. Any redirection of these funds for tutoring and/or professional development would cause school district to have to find alternative funding for programs it already supports with these dollars. Also, the amount of funding for Accountability Block Grants and the separate line item for tutoring and extra instructional time, while increased over a several-year period, have been proposed for even funding in the 2008-09 state budget.  Indeed, no one can tell how these line items will change by the time the proposal takes effect in 2013-14. Governor Rendell’s proposed six-year funding increase of $2.6 billion over the next six years will be helpful but it must first win approval from the General Assembly.

The proposed budget calls for $15 million to develop the first 3 of the required 10 GCAs, although the Algebra II assessment has been developed through an inter-state partnership. These costs, plus costs to validate the tests and costs for additional remediation and professional development are estimated to cost the state almost $160 million over the next five years.17

Conclusion

Both the proponents and opponents of the current GCA proposal are united in their desire for a system that ensures that students graduate from high school proficient in Pennsylvania’s academic standards. Proponents believe that the only way to ensure this is through state tests – either the PSSA or the GCAs. Opponents, including PSBA, believe that a system that allows local tests aligned to the state academic standards and the PSSA can meet this requirement. PSBA is not convinced that the discrepancies shown by the proponents that more students are graduating without scoring proficient on the PSSA point to a deficiency in local assessments.

The association believes that the data from its survey of school entity assessment practices shows that districts expend considerable time, effort and resources, both personnel and financial, in developing local graduation assessments. Even within the limited information that could be shared through the survey, it appears to us that school entities are making a sincere effort to develop and align test to the best of their ability.

The conferring of high school diplomas and development of curriculum are issues that have a long history of local control, as stated in the current School Code and State Board regulations. Simply wiping away such local control in favor of statewide testing and curriculum development could have lasting effects on Pennsylvania’s high school students. Time and again, it has been shown that a “one size fits all “approach does not work in education.

The proposal will also be harmful to students, especially those who do not fare well on standardized tests and on those who attend school entities that do not have the capacity to provide the quality of instruction available in the state’s wealthiest school districts.

Rather than spend the estimated $160 million over five years to implement the GCA proposal, PSBA believes those dollars would be better spent on efforts that capture the school entities’best practices in local assessment and sharing and implementing them in all districts statewide. The final result would be graduation requirements that are based on a mixture of state and local assessments that accurately measure a student’s proficiency in the commonwealth’s academic standards. Additionally, such a system would allow the continued use of various assessment strategies through the local assessment, ensuring that all students would have a variety of alternative methods in which they could demonstrate proficiency.

Finally, some of the funds earmarked for the implementation of this proposal could go to school districts that cannot provide adequate assessments because of lack of financial resources. Funds could be used to help districts hire necessary staff or contract with test providers or intermediate units, as some already do, to help create and implement rigorous local assessments and to provide districts with technical assistance.

www.pde.state.pa.us, Changing High school graduation requirements, New high school requirements: Fact vs. Fiction

IBID

IBID

IBID

Regulatory Analysis Form, State Board of Education proposed regulation of 22PA Code Chapter 4 – Academic Standards and Assessment (#006-312)

Andrea L. Sinclair and Arthur A. Thacker, (2005) Relationships Among Pennsylvania System of School Assessment (PSSA) Scores, University Proficiency Exam Scores, and College Course Grades in English and Math,   (HumRRO FR-05-55) Tables 16, 17, 18, 19, 20.)

Proposed § 4.24(b)(2)(D)(iv)(A-G)

Proposed § 4.24(b)(2)(D)(iii)

E-mail message from Dan Sidelnick, CTB/McGraw-Hill to Dr. Mary Ravita, South Fayette School District, March 11, 2008.

College Board: The 4th AP® Annual Report to the Nation: Pennsylvania Supplement

22PA Code, Chapter 4, § 4.52(e)(1-9) (Local assessment system)

Proposed § 4.24(b)(iv)(C)

  IBID, Regulatory Analysis Form

22PA Code § 4.52 (a)(1) (Local assessment system)

Proposed § 4.24(c)

PA Dept. of Education, Achieving Student Success, Accountability Block Grant 2006-07 Year end report

IBID Regulatory Analysis Form