The PART

In less than two weeks, President-elect Barack Obama will be sworn in to office.  As the Obama Administration determines its approach to assessing program performance, we hope that they learn from the current PART process.  Our recommendations include: 

Create buy in, especially from Congress

If President-elect Obama truly wants to create an effective way to assess program performance, he must have buy-in from Congress, OMB, federal agencies, stakeholders and the public. Congress and other stakeholders do not trust PART as it currently stands. It is considered a subjective process that can vary significantly from reviewer to reviewer and across agencies. Future program performance or evaluation systems must ensure fair, impartial, and evenly applied outcomes across all programs and the systems should have the support of agencies and stakeholders across the political spectrum. 

Since it's inception in 2002, the Program Assessment Rating Tool (PART) has received both criticism and praise from stakeholders, researchers, and Congress.  Let's explore the pros and cons of the PART:

Pros

• The PART increased attention paid to program results.

Since the PART was initiated in 2002, stakeholders have paid increased attention to program performance.  Department of Education budget analysts and program staff, as well as staff from OMB, have worked together to assess individual programs.  As a result, many programs have created short term and long term performance measures that allow them to continually focus on demonstrating results. 

• The PART assessment asks some useful questions that highlight program management and design issues.

Overall, the PART assessment asks some useful and important questions.  Examining 93 programs at the Department of Education and over 1,000 programs government-wide through the four PART sections (Purpose & Design, Strategic Planning, Management, and Results) has helped to reveal weaknesses and capitalize on strengths. PART assessments have also provided Congress with direction when re-authorizing legislation and help the Department of Education improve strategic planning and management. 

The CCAMPIS program provides funding to colleges for childcare services for low-income parents.  In 2004 CCAMPIS received a "results not demonstrated" rating.  In 2007 it was "re-PARTed" and received an "adequate" rating. 

Let's examine how CCAMPIS fared on each section of the PART assessment:     

Program Purpose and Design - CCAMPIS Score: 2004 - 80%, 2007 - 80%

CCAMPIS received full credit for four of the five questions on both the 2004 and 2007 assessments.  However, both PART reviews deemed the program's design flawed, limiting its effectiveness. At the time, the statute stated that only schools that received $350,000 or more in Pell Grant money were eligible for CCAMPIS funding and limited the size of the CCAMPIS award based on that amount. Additionally, the statute dictated that the program only report data every 18-36 months as opposed to annually. (Both of these limitations were changed in the 2008 reauthorization of the Higher Education Act.) 

Strategic Planning - CCAMPIS Score: 2004 - 62%, 2007 - 75%

Over the past few weeks we've been looking at the PART on a macro-level.  Now it's time to take a closer look at PART results for an individual program. The Safe and Drug Free Schools program (SDFS) is a block grant that provides funding to states and school districts for drug and violence prevention in schools.  In 2006 it was given a "results not demonstrated" rating.  Why?

Let's first examine how SDFS fared on each section of the PART assessment:      

Program Purpose and Design - SDFS Score 60%

The program received full credit for three of the five questions in this section - whether the program's purpose is clear, whether the program addresses a specific problem, and whether the program is duplicative of other programs.

It received no credit for two questions. The PART assessment found that SDFS has a design flaw that limits its effectiveness. Because SDFS is a block grant, funding is thinly distributed across all 50 states.  As a result, "two-thirds of all school districts receive allocations of less than $10,000, amounts typically too small to mount comprehensive and effective drug prevention and school safety programs."  Though the PART points out the design problem, at the end of the day, this is really a funding issue. 

While PART ratings have a limited affect on the President's budget request, it appears they have almost no affect on the actual funding levels Congress appropriates.  Programs rated "ineffective" largely continue to receive funding, while programs rated "effective" do not necessarily get increases in funding. 

Why ? 

1. Many people in Congress are unaware of the PART or have limited information about it. We called several congressional staff and asked whether they thought the PART ratings were useful and whether they were actually used to make decisions about funding levels. The majority of staffers told us that they either didn't know what the PART was or had only limited information about it.  Committee staffers were more aware of the PART, but did not always have a clear sense of what it actually was or feel that it was particularly useful.  If Congress doesn't have a good understanding (or in some cases even know) of PART, it is difficult for them to factor it into its funding decisions.            

The Program Assessment Rating Tool (PART) was created as part of the Bush Administration's Budget and Performance Integration Initiative to link program goals and performance measures to the budget process.  According to the Office of Management and Budget (OMB), PART results "will be used to make decisions regarding budgets and policy."

To what degree has this actually happened? Do PART results for Department of Education programs correspond to funding levels in the President's Budget Requests? 

As noted in last week's post, almost half the programs at the Department of Education do not present enough information to determine program effectiveness and receive a PART rating.  But limited evidence exists to suggest that a program's PART rating, even when available, is considered when making funding decisions.

While there may be some connections between PART ratings and budget requests, they are not always consistent.  For example, the President's 2009 budget did eliminate funding for three of the four Department of Education programs rated "ineffective" - Even Start, the Perkins Loan Program, and Vocational Education State Grants.  But the budget did include level funding for TRIO Upward Bound, which was also found to be "ineffective" (though some prior year requests did try to cut funding). 

During the presidential campaign, both Senator McCain and Senator Obama talked about eliminating ineffective federal programs.  We commend their emphasis on funding programs that show results.  But what evidence will President-elect Obama use to actually determine which programs are working and which aren't?

One source of information the president could turn to for guidance is the Program Assessment Rating Tool (PART).  But for many programs, particularly at the Department of Education, President-elect Obama might find only limited information to determine whether some programs are actually effective.      

Results for Department of Education Programs

Since 2002, 93 Department of Education programs have gone through the PART review process. Of those, 44 were rated "results not demonstrated." That means nearly half the programs did not have acceptable performance measures or lacked the performance data necessary to make a determination. 

Last week we introduced our blog series on the Program Assessment Rating Tool (PART), a questionnaire designed to assess and rank the effectiveness of federal programs.  Since 2002, the Office of Management and Budget has used PART to examine hundreds of programs in various agencies in the federal government.  To date, more than 1000 programs have been through the PART process, including 93 programs at the Department of Education.     

How it Works

The PART questionnaire is comprised of approximately 25 standard questions, as well as a few additional questions tailored to the type of program being evaluated.  Programs can belong to one of seven categories: direct federal; competitive grants; block or formula grants; regulatory; capital assets and service acquisition; credit; and research and development.

The questions are divided into four categories: 

We all want to know if the government is spending our tax dollars wisely: Are funds being used on education programs that are actually achieving results? Or are they being spent on ineffective programs? Which programs are working? Which aren't?  Results matter. 

Most people would agree that these are important questions.  But actually evaluating the effectiveness of federal programs is an incredibly complex task. 

In 2002, the Office of Management and Budget launched an initiative to help measure program effectiveness and provide guidance to the policymaking, budgeting and appropriations process.  The resulting Program Assessment Rating Tool (PART) is designed to "assess and improve program performance so that the Federal government can achieve better results." 

Since the inception of PART, 1017 programs across 15 Departments have been reviewed, 93 at the Department of Education alone.  Programs are assessed and assigned points in four categories: Program Purpose & Design, Strategic Planning, Program Management, and Program Results.  At the end of the process, programs are rated "effective," "moderately effective," "adequate," "ineffective," or "results not demonstrated."  The President's budget request to Congress often reflects the results of different PART assessments.