Article Information

Authors:
Lindsey Joseph1
Kate Emmett1
Joha Louw-Potgieter1

Affiliations:
1Section of Organisational Psychology, University of Cape Town, South Africa

Correspondence to:
Joha Louw-Potgieter

Postal address:
Room 4.32, Leslie Commerce Building, University of Cape Town, Rondebosch 7701, South Africa

Dates:
Received: 05 Oct. 2011
Accepted: 25 Jan. 2012
Published: 13 July 2012

How to cite this article:
Joseph, L., Emmett, K., & Louw-Potgieter, J. (2012). A formative evaluation of a pay-for-performance system. SA Journal of Human Resource Management/SA Tydskrif vir Menslikehulpbronbestuur, 10 (3), Art. #426, 12 pages. http://dx.doi.org/10.4102/
sajhrm.v10i3.426

Copyright Notice:
© 2012. The Authors. Licensee: AOSIS OpenJournals.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
A formative evaluation of a pay-for-performance system
In This Original Research...
Open Access
Abstract
Introduction
   • Key focus of the evaluation
   • Background to the evaluation
   • Trends from the research literature
      • Elements of pay-for-performance systems
      • Standard requirements for implementing pay-for-performance systems
      • Pay-for-performance systems and non-management employees
   • Evaluation objectives
   • Introduction of a pay-for-performance system at a South African university
   • Evaluation questions
   • The potential value-add of the evaluation
Research design
   • Research approach
   • Research method
      • Research participants
      • Measuring instruments
      • Research procedure
      • Statistical analysis
Results
   • Question 1: Training coverage
   • Question 2: Resources
   • Question 3: Short-term training outcomes
   • Question 4: Longer-term outcomes
Ethical considerations
   • Potential benefits and hazards
   • Recruitment procedures
   • Informed consent
   • Data protection
Trustworthiness
   • Reliability
   • Validity
Discussion
   • Question 1: Training coverage
   • Question 2: Resources
   • Question 3: Short-term outcomes
   • Question 4: Longer-term outcomes
   • I receive guidance from my line manager in setting my performance objectives
   • I receive guidance from my human resources adviser on how to set the performance objectives that are relevant to my job
   • I receive regular informal advice from my line manager about how to improve my job performance
   • There are adequate training opportunities in my department to enable me to achieve my objectives
   • Human resource’s additional role in a pay-for-performance system
   • Conclusions
   • Suggestions for future research
Acknowledgements
   • Competing interests
   • Authors’ contributions
References
Abstract

Orientation: Pay-for-performance (PFP) systems emerged during the 1980s as performance improvement tools. However, research findings have shown contradictory evidence as to whether these systems motivate employees to improve their performance.

Research purpose: The main aim of this evaluation was to assess whether a PFP system, which a South African university introduced for administrative employees, improved their performance. A secondary aim was to examine whether the university implemented the system as it intended to.

Motivation for the evaluation: The motivation for this evaluation was to add to the social science literature on the effectiveness of PFP systems. There are many contradictions in the literature and further exploration of whether these systems deliver their intended outcomes seemed overdue.

Research design, approach and method: The evaluators used a descriptive design. They administered a customised questionnaire, to which 391 university staff members responded. Of these, 129 were line managers and 262 were administrative staff.

Main findings: The administrative staff, whose working lives the PFP system affected, thought that it did not improve their performance. Both line managers and administrative staff indicated that the pay aspect of the system did not differentiate between poor and excellent performance.

Practical/managerial implications: The evaluators made practical recommendations for improving the implementation of the system.

Contribution/value-add: This evaluation contributed to the social science literature on the effectiveness of PFP systems by showing that poor implementation rather than poor design often lies at the root of a system that does not deliver its intended outcomes.

Introduction

Key focus of the evaluation
The main aim of this evaluation was to assess whether the pay-for-performance (PFP) that at a South African university system introduced for administrative employees improved their performance. A secondary aim was to examine whether the university implemented the system as it intended to.

Background to the evaluation
Pay-for-performance systems emerged during the 1980s as the way to motivate employees and improve organisational performance (Cannell & Wood, 1992; Dowling & Richardson, 1997). However, research shows contradictory evidence as to whether PFP systems are effective motivational and performance improvement tools (Armstrong, 2006; Dowling & Richardson, 1997).

It seems that organisational contexts influence the effectiveness of these systems (Armstrong, 2003; Belcher, 1996; Gomez-Mejia, Wiseman & Dykes, 2005). According to Bruns (1996), PFP systems are most effective when implemented in organisations which foster trust between management and employees, do not have strong trade unions which oppose PFP systems and have an entrepreneurial or performance culture. Research also shows that PFP systems are less effective in improving team performance because PFP measurement tends to be individualised (Armstrong, 2003; Belcher, 1996; Bruns, 1996; Reilly, 2003; Wright, 2004).

Trends from the research literature

Elements of pay-for-performance systems
According to Ulrich and Brockbank (2005), two elements are the basis of PFP systems: measurement and pay.

Measurement consists of formal assessment and the ratings of managers. The main functional requirements for the measurement element of a PFP system are to distinguish good from bad performance and to separate past from future performance.

Records of past appraisals determine past performance whereas employees’ development plans define future performance (Armstrong, 2003). Furthermore, the people the system affects need to perceive the measurement element of PFP systems as fair if it is to fulfil its motivational function (Armstrong, 2003; Edwards, Scott & Raju, 2003). Doran (2008) reinforced this idea and suggested that the employee ratings managers give should undergo independent moderation so that employees will see rating decisions as less subjective and discriminatory.

The pay element of PFP systems shows what and who organisations value (Holbeche, 2004). In addition, this element should reinforce positive employee behaviour so that employees can achieve their goals and improve their performance (Henderson, 2006).

This element of PFP systems is most effective when it links visibly to performance, when it gives employees feedback soon after they achieve outstanding performance and when it deducts pay if the recipients do not continue to perform (Eichinger, Lombardo & Ulrich, 2004; Rosenthal, Landon, Howitt, Song & Epstein, 2007).

The pay element of PFP systems should provide an adequate pay incentive (Doran, 2008; Rosenthal et al., 2007). According to Henderson, reward-related pay made in lump sums is more effective than percentage increases because lump sums have a more positive effect on employees’ motivation and, therefore, on improved performance. In addition, Armstrong (2003) states that percentage increases do have a positive effect on employees’ motivation and performance as long as the increase is between 10% and 15%.

Human resources (HR) publications rarely explore the pay element of PFP systems. One may attribute this to the fact that HR does not exclusively control the pay function of PFP systems. Quite often, finance departments are responsible for the pay element. In these cases, HR has focused on the measurement element. This could be problematic because it could separate the pay and measurement elements in PFP systems. Organisations need to align both of these elements to produce effective PFP programmes.

Standard requirements for implementing pay-for-performance systems
The essential requirements for implementing PFP systems successfully are:

• the commitment of managers to the system

• a focus on the implementation process rather than on designing the systems

• support from key stakeholders

• clear communication about the systems (Armstrong, 2003; Rosenthal et al., 2007).

Streib and Nigro (1993) emphasised the importance of process training for the effective implementation of PFP systems because it guides employees how to set clear and realistic performance objectives. It also helps employees and their managers to go about interacting and collaborating in the process. In addition, process training shows employees how systems link pay to performance. This understanding is important because linking pay to performance is the main motivating mechanism for employees.

Pay-for-performance systems and non-management employees
Organisations use PFP systems mainly for employees at management level (Armstrong, 2003; Belcher, 1996; Pennings, 1993; Reilly, 2003; Schwab & Olson, 1990; Wright, 2004). They seldom target administrative and support employees for this type of pay system because they do not produce a specific product or service. Their work is usually task-related and they often work in small work groups instead of on their own.

Research shows that trade unions could hinder the implementation of PFP systems for their members (Reilly, 2003; White & Druker, 2000; Wright, 2004). PFP systems reward employees individually, whilst trade unions prefer to increase pay equally for all employees (Doran, 2008; Reilly, 2003; White & Druker, 2000; Wright, 2004). Trade unions often resort to collective bargaining to compress the range of pay for performance. Compressed pay ranges lead to decreases in the discretionary amounts available for distribution to employees, increased perceptions of the unfairness of systems and decreased motivation (Doran, 2008; Wright, 2004).

Trade unions also influence the level of rewards, the structure of rewards, the reward systems organisations choose and all procedures concerned with managing rewards (White & Druker, 2000). Because of the wide-ranging influence trade unions have on PFP systems, these systems may not suit unionised environments because they reduce the functionality of the systems (Reilly, 2003; White & Druker, 2000; Wright, 2004).

Evaluation objectives
The main aim of this evaluation was to assess whether a PFP system, which a South African university introduced for administrative employees, improved their performance. A secondary aim was to examine whether the university implemented the system as it intended to.

Introduction of a pay-for-performance system at a South African university
In 2001, a South African university introduced a PFP system for administrative staff. Most of the staff belonged to unions. According to the Performance Development Resource Guide (University of Cape Town, 2006), which the university provided, and from interviews with key stakeholders, the new PFP system aimed to improve administrative employees’ performance and help them to clarify their roles. It also intended to increase their skills, knowledge and abilities for growth and future performance.

The system intended to achieve these goals through a two-fold process. The first process, performance management, intended to assess employees’ performance against agreed outcomes and then reward employees for excellent performance. The second process, performance development, aimed to identify development opportunities so that the university could develop employees’ skills for future performance.

According to the university’s Salary Assessment Resource Guide (University of Cape Town, 2006), the university deliberately separated the performance management and development discussions. This was to ensure that the performance management discussion, with its concerns about assessment and pay, did not cloud the development discussion. Although the university dealt with them separately, both processes used a single set of performance objectives that employees would develop. The objectives would link to the university’s strategic objectives. These objectives were the link between the performance management and development processes.

The link the objectives provided between the performance management and development processes also created a performance development cycle. This cycle contains six steps:

• agree on objectives

• assess performance against objectives

• construct a development plan

• review performance against objectives

• review development needs continuously

• review objectives formally.

The activities in this cycle involve continuous collaboration between line managers and employees. During the final step, the formal review of objectives, line managers assess and rate employees’ performance. They relate performance to pay by placing the employees in appropriate performance categories that range from ‘significantly exceed job requirements’ to ‘do not meet job requirements’.

The university introduced its PFP system in 2004. It affected all administrative employees in the bargaining unit. A half-day workshop, presented over a three-month period, gave administrative staff and their line managers the necessary knowledge to use the system. Attendance at these workshops was voluntary. Human resource advisers offered the workshops to each department and provided a Performance Development Resource Guide to all employees who attended the workshops.

Administrative employees and line managers attended different workshops. The administrative employees’ workshop focused on how the PFP system was going to affect them and how their performance levels would determine their pay. It also trained them to set objectives. The line managers’ workshop trained line managers in the performance development process, to review their employees’ performance, to provide constructive feedback and to rate employees’ performance.

Based on discussions with the programme managers, the short-term outcomes of an effective PFP system would be cooperation between line managers and administrative staff to produce performance objectives and a development plan. See Figure 1.

At this stage of the evaluation, the evaluators did not know which variables would mediate between the programme and its short-term outcomes.

Evaluation questions
According to Scriven (1991), a formative evaluation is one that guides programme improvement. The evaluators formulated the evaluation questions in Box 1 to improve the PFP system.

The potential value-add of the evaluation
The evaluation seeks to contribute to the social science literature on the effectiveness of PFP systems by examining whether these systems work. If they do not, the evaluators seek to find out why.

FIGURE 1: Diagram of an effective pay-for-performance programme.

Box 1: Questions the evaluators formulated to improve the pay-for-performance system.

Research design

Research approach
The evaluators used a descriptive design. They collected primary quantitative and qualitative data to answer the evaluation questions.

Research method

Research participants
Of the 391 university staff members who responded to the questionnaire, 129 were line managers and 262 were administrative staff. There were 1769 administrative employees at the university in 2007. The evaluators could not ascertain the number of line managers from the relevant department.

Measuring instruments
The evaluators used a questionnaire that Dowling and Richardson (1997) developed. It examined whether a specific PFP worked for the current employees. Table 1 presents it.

Question 1 offered 24 options, questions 2, 3 and 4 offered ‘yes’ or ‘no’ options, question 5 offered four options and the rest of the questions had a 5-point Likert format.

Research procedure
The evaluators distributed the questionnaire electronically to participants. The questionnaire was available on-line for a period of two months and the evaluators sent two reminders to participants during that period.

Statistical analysis
The evaluators used descriptive statistics to analyse the data for evaluation questions 1–9. For questions 10 and 11, the evaluators performed a reliability analysis, a factor analysis, a linear regression and a stepwise discriminant analysis.

Results

The evaluators reported their results in terms of the evaluation questions.

Question 1: Training coverage
a. How many staff members received training?

b. Were the trainees mainly line managers or administrators?

Seventy-two per cent of line managers and 60.7% of administrative staff indicated that they had received training since the introduction of the system.

The main reasons for non-attendance were:

• unaware that training had been scheduled (15% of line managers and 26% of administrative staff reported this)

• lack of time to attend training (7.8% of line managers and 10% of administrative staff reported this)

• perceptions that the training would not be useful (7% of line managers and 8.4% of administrative staff had these perceptions).

Question 2: Resources
a. Were there sufficient human and physical resources for implementing the new performance management system?

There was no significant difference between the answers of line managers and those of administrative staff about the helpfulness of HR advisers during the implementation process. Thirty per cent of both groups indicated that the HR resource was very helpful. The line managers also rated other resources in terms of their helpfulness. They indicated that they found their senior managers (33%), colleagues (28%), training manuals (23%) and training workshops (20%) very helpful.

Question 3: Short-term training outcomes
With regard to the short-term training outcomes, the evaluators focused mainly on how well the two groups of trainees understood various aspects of the system and how skilled they thought they were in some aspects of the measurement part of performance management.

TABLE 1: Questionnaire for collecting data from administrative employees.

TABLE 2: Understanding the aspects of the pay-for-performance system.

Understanding:

a. Did the line managers and administrative staff understand the main components of the system after the training?

Table 2 shows that the line managers understood these aspects better than the administrative staff did.

Skills:

b. Did line managers and administrative staff have the necessary participative skills to implement parts of the system after the training?

Line managers and administrative staff should cooperate when setting performance objectives for administrative staff. The evaluators compared the two groups with regard to their perceived skill for this task. Line managers rated themselves as more skilled in setting objectives than administrative staff did (line managers M = 4.09, administrative staff M = 3.87; t = 2.40, p < .017). When the evaluators compared the managers’ ratings with the self-ratings of administrative staff on setting skills objectively for administrators, there was a significant difference (line managers’ ratings of administrative staff M = 3.33, administrative staff members’ self-ratings M = 3.87; t = -5.137, p < .000).

c. Did line managers have the necessary skills to implement specific aspects of the system?

The evaluators also assessed the line managers’ perceptions of their own performance management skills for implementing specific aspects of the system. Table 3 shows that line managers regarded themselves as generally skilled in these tasks.

Plans:

d. Did line managers have the necessary skills to implement specific aspects of the system?

e. How many administrative staff had these plans?

Administrative staff did create development plans after the training. In 2005, 74% of the administrative staff had plans. This increased to 81% in 2006.

TABLE 3: Self-rating of performance management skills (line managers).

TABLE 4: Longer-term perceived outcomes of the performance management system (line managers).

TABLE 5: Longer-term perceived outcomes of the performance management system (administrative staff).

Question 4: Longer-term outcomes
a. After two implementation cycles, how did line managers and administrative staff rate the outcomes of the system?

Table 4 reports the line managers’ perceived longer-term outcomes of the performance management system.

Table 5 reflects the perceptions of the administrative staff after two implementation cycles.

The evaluators added an open-ended question to the questionnaire. It required all respondents to comment on the potential advantages and disadvantages of the PFP programme.

Two major themes emerged from the comments. These were the lack of meaningful rewards (mentioned by 45% of line managers and 15.6% of administrative staff) and non-procedural evaluation by line managers (mentioned by 36.4% of line managers and 35% of administrative staff).

b. Did the PFP system at the university work for some administrative staff and not for others?

c. Are there interventions or variables that can predict for whom the system would work?

In order to answer these two questions, the evaluators conducted three analyses. Firstly, they subjected all items that relate to the administrative staff’s perceptions of the PFP system to a reliability analysis. Secondly, they reduced these items to three factors using factor analysis. Thirdly, they conducted a linear regression and a step-wise discriminant analysis to ascertain for which administrative staff the PFP system worked. They discuss each of these steps below.

The evaluators used Cronbach’s alpha to identify the internal consistency of items 6 to 22 in the administrative employees’ questionnaire. Cronbach’s alpha = .91, which suggested a high internal consistency reliability of the 17 item scale (Pallant, 2007). They also used item total statistics to show the correlation of each item with the overall interval scale. There were no items with an item-total correlation below .3.

Therefore, items 6 to 22 all measured the same construct (employees’ perceptions of the PFP system), and the evaluators did not need to delete any of the items (Pallant, 2007).

In order to reduce the data of the interval scale items into separate independent variables, the evaluators conducted an exploratory factor analysis (EFA) on items 6 to 22 (they excluded item 12, the dependent variable) of the questionnaire for the administrative staff. The Kaiser-Meyer-Olkin value was .86. This exceeds the recommended value of .6. Bartlett’s test of sphericity was statistically significant as its p value was 0.00 (< 0.05) (Pallant, 2007). Therefore, the evaluators confirmed that it is appropriate to conduct an EFA with the interval items from the administrators’ questionnaire.

A principal component analysis revealed the presence of three factors with Eigenvalues that exceed 1. This explained 62% of the cumulative variance. In addition to this, the pattern matrix confirmed that all the items had loaded onto the three factors. However, items 10, 11 and 15 loaded onto more than one factor. The difference between these cross-loading items was smaller than .25 (Pallant, 2007) and the evaluators removed them.

Table 6 presents the three distinctive and coherent factors.

The evaluators labelled the first factor ‘guidance, support and training’. It consisted of items 13 (‘I receive guidance from my line manager in setting my performance objectives’), 14 (‘I receive guidance from my HR adviser on how to set the performance objectives relevant to my job’), 16 (‘I agree on clear deadlines with my line manager for achieving my objectives’), 17 (‘I receive regular informal advice from my line manager about how to improve my job performance’) and 18 (‘There are adequate training opportunities in my department to enable me to achieve my objectives’). These items show that there is a robust support system for administrative employees on performance management. The factor contributed 42% variance of the scale.

The evaluators labelled the second factor ‘employees’ understanding’ (of the PFP system). It consisted of items 6 (‘please indicate the extent to which you are skilled in setting performance objectives’), 7 (‘I have a clear understanding of what the primary goals of the PFP system are’), 8 (‘I understand how the PFP system works’) and 9 (‘I understand how the performance assessment links to the PFP system’). This factor contributed 12% of the total variance of the scale.

The evaluators called the last factor ‘employees’ motivation to work harder’. It included items 19 (‘I consciously work harder because of the PFP system’), 20 (‘the financial rewards I receive reflect my individual contributions’), 21 (‘the PFP system contributes significantly to my receiving the credit and recognition I deserve’) and 22 (‘the financial incentive of the PFP system increases my determination to achieve my objectives’). This factor contributed 10% of the total variance of the scale. The cumulative variance of these three factors was 64%.

To measure the relationship between the three independent variables the evaluators identified in the EFA and the dependent variable (item 12: ‘I participate with my manager in setting my performance objectives’ – that they labelled ‘cooperative objective setting’), they used a linear regression.

TABLE 6: Pattern matrix and communalities of the three-factor solution of administrative employee perception scale items.

The evaluators first used collinearity diagnostics to assess whether there was a possibility of multi-collinearity. The variance inflation factor (VIF) and tolerance posed no problem of multi-collinearity because the VIF for all the scores was smaller than 10 and the tolerance scores were larger than .10 (Pallant, 2007). Table 7 shows this.

The evaluators entered all three of the independent variables (guidance, support and training; employees’ understanding; and employees’ motivation to work harder) into the regression model. Upon evaluating the regression model, they found that the model explained 49.3% of the variance of cooperative objective setting. They also found that this model reached statistical significance because the analysis of variance (ANOVA) p value was 0.00 (p < .005). See Pallant (2007).

Guidance, support and training made the strongest unique contribution to explaining cooperative objective setting when the evaluators controlled for all the other variables in the model. This variable had the highest beta value. It also made a statistically significant contribution to predicting cooperative objective setting. When the evaluators scanned the part correlation coefficients, they found that guidance, support and training explained 32.8% of the variance in the dependent variable. Employees’ understanding and employees’ motivation had p values larger than 0.05. Therefore, they did not make a statistically significant contribution to explaining cooperative objective setting (see Table 7).

The evaluators conducted a step-wise discriminant analysis to investigate whether the three independent variables (guidance, support and training; employees’ understanding; and employees’ motivation to work harder) could be predictors of the dependent variable: the existing development plan (item 3: ‘I had a performance development plan for 2006’). Of the 262 administrative employees, 213 had a performance development plan.

Table 8 shows the uni-variate ANOVAs of all three independent variables. There was a statically significant difference between all the grouping variable means because the p values for all the variables were smaller than 0.05.

The evaluators identified only a single step for this analysis. The only variable they entered was guidance, support and training. This variable had the lowest Wilks’ Lambda value of 0.93 and the highest F value of 19.43. The tolerance value of this variable was 1.00. This suggests that this variable contributed strongly to the analysis (Pallant, 2007). The evaluators found that step 1 accounted for 100% of the variance of whether employees had an existing performance development plan in 2006. In addition, this discriminant function was statistically significant because the p value was 0.00 (p < 0.05). See Pallant (2007). The evaluators also found a canonical correlation of 0.26 and an Eigenvalue of 0.75.

Table 9 shows which criterion groupings have the most accurate classifications when the evaluators used the discriminant functions they developed in the analysis. The analysis was more likely to classify accurately predicted group membership of employees who did not have an existing development plan compared to the employees who did have one.

Ethical considerations

The Ethics in Research Committee of the Faculty of Commerce, University of Cape Town, approved the evaluation.

Potential benefits and hazards
There were no benefits or hazards for participants who took part in the evaluation.

Recruitment procedures
The evaluators distributed the questionnaire electronically to all participants.

Informed consent
On the first page of the questionnaire the purpose, confidentiality of the data and the time it would take to complete were described in detail.

Data protection
The webmaster stripped the data of all identifiers and downloaded the data onto a spreadsheet. The first two evaluators were the only people who had access to the data.

TABLE 7: Linear regression analysis for variables that predict cooperative objective setting.

TABLE 8: Discriminant analysis for variables that predict the existing development plan.

TABLE 9: Classification of results to determine the accuracy of predicted group membership.

Trustworthiness

Reliability
The reliability of the measuring instrument is described on page 6.

Validity
The evaluation focused on a specific programme and did not seek to generalise the findings to other, similar programme.

Discussion

The main aim of the evaluation was to assess whether a pay-for-performance (PFP) system, which a South African university introduced for administrative employees, improved their performance. A secondary aim was to examine whether the university implemented the system as it intended to. This evaluation will contribute to the social science literature on whether PFP systems work, and if they do not, why.

In general, the results showed that there was a discrepancy between the perceptions of the line managers and those of the administrative staff about the PFP system. The line managers tended to be more positive about the system outcomes than the administrative staff were. The evaluators structured the discussion according to the evaluation questions they set earlier.

Question 1: Training coverage
At first glance, the number of staff who received training seems adequate. However, one has to question whether it is acceptable that approximately 40% of administrative staff, who had to complete the complex task of developing measurable objectives, had not had any training. The evaluators recommend that the university presents more training workshops in order to increase the skill levels of administrative staff.

Question 2: Resources
It seems that there were adequate resources for implementing the system. For line managers, senior managers emerged as the resource that was most often helpful. These managers could be involved in the training the evaluators suggest.

Question 3: Short-term outcomes
Understanding:

a. Did the line managers and the administrative staff understand the main components of the system?

From the results, it was clear that line managers understood the system’s goals, operation, measurement and the link between pay and performance better than the administrative staff did. This is quite problematic, as the administrative staff members were the recipients of the system and experienced its consequences. When the university presents follow-up training courses, it should clarify these aspects of the system to the administrative staff.

Skills:

b. Did the line managers and administrative staff have the necessary participative skills to implement parts of the system after training?

Again, line managers rated themselves as more skilled than the administrative staff did in setting performance objectives. Line managers also questioned whether the administrative staff had adequate skills for setting objectives. In the evaluators’ experience, it is fatal simply to prescribe Specific, Measurable, Attainable, Relevant, Timely (SMART) objectives in a manual (the Performance Development Resource Guide, University of Cape Town, 2006) without practising how to set these objectives. The evaluators recommend that any additional training should include the clear guidelines of Kettner, Moroney and Martin’s (1999) for setting measurable objectives.

c. Did line managers have the necessary skills to implement specific aspects of the system?

Line managers’ self-ratings of their skills showed that they thought they had mastered how to develop a performance management plan and how to give constructive feedback. Although more than 70% indicated that they were skilled or highly skilled in rating performance and in differentiating between different levels of performance, additional training in these two aspects might provide better coverage for all skills.

Plans:

d. Did administrative staff apply the training to create development plans?

At face value, the answer to this question is that they did. In 2005, 74% of administrative staff had plans and this percentage increased to 81% in 2006. However, HR performed a quality check of these plans in 2005 and found that in only eight of 21 departments were the plans satisfactory or good. There was no intervention to improve the quality of the development plans.

The evaluators noted earlier that, in the training manuals, the performance objectives of administrative staff did not link to the strategic objectives of the university or the job descriptions of the staff members. One can easily group administrative jobs into job categories (like departmental secretary or finance officer) where there is considerable overlap in performance objectives. One can strengthen the perception that measurement is consistent by clarifying the link between objectives and job descriptions and by emphasising the similarities between objectives in a job category.

Question 4: Longer-term outcomes
a. After two implementation cycles, how did line managers and administrative staff rate the effectiveness of the system?

Too many line managers agreed or strongly agreed that performance management is an administrative task with little pay-off (34.1%) and that it was a lengthy, time-consuming task (54.3%). The clear link between performance objectives and job description or category could change this perception significantly. This link would reduce the number of irrelevant objectives that administrative staff members often create because there are clear parameters for setting objectives. In any event, the evaluators suggest that administrative staff do not set more than between seven and nine objectives. These objectives should focus on the key performance areas of their jobs. Essential objectives rather than irrelevant detail should drive performance measurement.

Of even more concern is the perception of inconsistency in implementing the system, endorsed by 83% of line managers. This poses a threat to the credibility of the system. Again, job categories with comparable objectives might reduce the notion that each person’s objectives and their measurement are unique or arbitrary.

On the positive side, the results showed that line managers reported high commitment to staff development and the required skills to implement pay-for-performance in their departments. However, when the evaluators compared these opinions, about the availability of training for performance improvement, to those of administrative staff members (see Table 5) a different picture emerges. Only 34.7% of administrative staff agreed or strongly agreed with the statement that there are adequate training opportunities for improving performance.

Administrative staff members indicated that, in their opinion, the PFP system had little value.

Research on the standards required for effective PFP systems showed that the recipients should see the system as fair (Barnes-Farrell & Lynch, 2003), perceive a link between performance and pay (Holbeche, 2004; Ulrich & Brockbank, 2005) and perceive differential rewards for poor and excellent performance (Ulrich & Brockbank, 2005).

The results of the evaluation showed that if one took 50% of administrative staff as an arbitrary cut-off point, one could conclude that more than half of the administrative staff:

• did not see the system as fair

• did not work harder because of the system

• felt that the rewards did not reflect their performance

• felt that they did not get credit and recognition for their performance

• felt that financial incentives did not increase their motivation to achieve their objectives.

Where staff had more control over the outcomes of the system, their perceptions were more positive. More than half of them strongly agreed or agreed that their performance objectives were clear enough to show what they were supposed to achieve (58.7%) and that they had agreed on realistic deadlines with their line managers to achieve their objectives (51.6%).

These results showed that the recipients of the system were not well disposed towards it. This could jeopardise the whole PFP endeavour. The evaluators recommend an intervention to change these perceptions. They also recommend that the university creates a clear link between objectives and job descriptions or categories in order to introduce the notion of consistency and reinforce perceptions of fairness. Thereafter, line managers should receive follow-up training in performance-rating skills. Clear performance categories and criteria (standards) are available in current manuals and additional training will strengthen line managers’ confidence in their performance judgements.

Addressing the link between pay and performance seems to be relatively easy. To date, the university has not implemented the PFP as it intended to. In 2005 (a year after the evaluation), the university committed itself to a PFP system but set the lowest end of the range at a 5% increase. In other words, the poorest performers still received a 5% increase. The upper limits of the range depended on the job class, with the highest job class in the bargaining unit getting the highest possible increase, namely 13.4%. In 2006, the university introduced a major pay correction for administrative staff based on salaries in the national job market. The PFP portion of the total increase all but disappeared with this adjustment. In 2007, the first time the university implemented the PFP performance system as it intended to, it fixed the ranges at between 6.2% and 7% (please note that the evaluation did not cover this period).

This range is problematic, as there is too little differentiation between poor and excellent performers. Therefore, it undermined one of the main goals of the system. The university managers need to revisit the uneasy alliance between collective bargaining and PFP, taking into account the tendency for unions to compress pay structures in these systems (White & Druker, 2000).

According to Holbeche (2004), the pay element of PFP systems shows what and who organisations value. In the open-ended questions, too many line managers and administrative staff questioned whether the system delivered meaningful rewards and whether there were fair performance decisions. These results do not hold great promise for one to regard the system as effective: recipients and appraisers alike questioned whether pay showed which employees and behaviours the university valued. They also questioned the fairness of the system. These doubts go to the heart of any performance management system. The university needs to address them to improve the system.

b. Did the PFP system at the university work for some administrative employees and not for others?

c. Are there interventions or variables that can predict for whom the system would work?

Figure 1 presented the outcome variables of an effective PFP system. These are cooperative objective setting and existing development plans. Using the results they obtained from the evaluation, the evaluators completed the diagram.

Figure 2 shows that the system worked for administrative employees who had guidance, support and training. This variable consisted of four items. The evaluators discuss each one below.

FIGURE 2: Programme theory for an effective pay-for-performance system for administrative employees.

I receive guidance from my line manager in setting my performance objectives
Guidance from line managers in setting objectives is important and should aim at defining objectives, in task and behavioural terms, developing observable objectives and setting clear deadlines for achieving these objectives (Eichinger et al., 2004; Harper, 2003). To ensure that line managers are fully competent when they guide their employees to set objectives, they should receive training in how to set objectives and how to guide their employees effectively. In addition to this, the evaluators recommend that this training focuses on guiding employees to set objectives that satisfy the current and future needs of the university (Eichinger et al., 2004). Therefore, the evaluators propose that HR educates line managers in how to turn the university’s strategic goals into practical and measurable objectives for administrative employees (Lawler, 1990).

I receive guidance from my human resources adviser on how to set the performance objectives that are relevant to my job
There is little research on the role of HR in supporting or guiding employees in PFP processes. The lack of this research is because of the notion that some HR professionals believe that their role is strategic and not participative when it comes to implementing PFP systems (Brown & Purcell, 2007). Furthermore, there is often an understanding that HR is responsible for designing programmes, whilst line managers are accountable for implementing programmes. Because of this split in responsibilities, line managers often find themselves in situations where they have insufficient time or financial resources to implement the PFP system effectively (Brown & Purcell, 2007). Therefore, support and guidance from HR is vital for implementing reward systems (Brown & Purcell, 2007). This is consistent with the results of this evaluation because the evaluators found that the PFP system worked for administrative employees who received guidance from their HR advisers on how to set objectives that are relevant to their jobs.

According to Streib and Nigro (1993), this type of process training guides employees to set clear and realistic performance objectives. It also defines the roles and responsibilities of managers and employees when they set performance objectives (Purcell & Hutchinson, 2007). Process training can also provide clear guidelines on how employees and their managers should be contributing to, and interacting in, the process of setting objectives. This will improve the implementation of the system (Streib & Nigro, 1993).

The university’s Performance Development Resource Guide (University of Cape Town, 2006) implied that HR advisers should help administrative employees to set their objectives during the formal training sessions. The evaluators suggest that this supporting role includes assessing the quality of the development plans (Armstrong, 2006). Although HR identified the poor quality of existing development plans during a previous round of implementation, there were no interventions to rectify this.

The effectiveness of process training and quality assurance are not the only aspects that need attention. Another is training coverage. Only 40% of administrative staff indicated that they attended process training. Voluntary attendance is not an option when it comes to understanding a new PFP system. The evaluators also suggest that the university communicates information about training workshops clearly and explicitly to administrative staff because the main reason that they did not attend was that they were unaware of the time or place of the workshops.

I receive regular informal advice from my line manager about how to improve my job performance
Receiving regular informal advice from line managers on how to improve performance was another of the items that contributed to the successful functioning of the PFP system at the university. According to Armstrong (2003), regular performance feedback is one of the most important aspects of effective PFP systems. This feedback reinforces perceptions about the fairness of the system (Armstrong, 2003).

The evaluators recommend that this informal feedback process extends to training for line managers. It could include how to distinguish between good and bad performance, how to give appropriate feedback according to performance and how frequently to give informal feedback (Armstrong, 2003).

There are adequate training opportunities in my department to enable me to achieve my objectives
It is the responsibility of HR and line managers to ensure that employees have access to adequate training opportunities. HR ought to ensure that the training is appropriate for developing administrative ability. Line managers should provide opportunities for administrative staff to attend training programmes during work time and to practise what they have learned at work (Armstrong, 2003). However, only 34.7% of administrative staff members agreed that there were adequate training opportunities for improving performance. Appropriate training, access to this training and opportunities to practise, as well as to apply, the new skills or knowledge in the workplace could change the perception that there are insufficient opportunities to develop.

Human resource’s additional role in a pay-for-performance system
It is clear that the four items the evaluators have discussed are associated with line managers’ support and guidance. However, enabling and eliciting this support from managers is not simple. Line managers often point out that it is difficult to provide the necessary support because of conflicting priorities, heavy workloads as well as lack of training and support (Brown & Purcell, 2007).

Therefore, in order to ensure that managers support their employees properly, the evaluators suggest that managers themselves receive support from their HR department. HR’s support should involve training and regular communication between the two parties, shared accountability for implementing PFP processes as well as giving feedback to line managers about any on-going monitoring processes (Brown & Purcell, 2007).

Conclusions
In conclusion, the administrative staff whose working lives the PFP system affected had a low opinion of its effectiveness. With regard to the goals of the system, which are to improve the performance of employees and to reward excellence, one must conclude that the university has not achieved them. Administrative staff members stated unambiguously that they did not work harder because of the system. Line managers and administrative staff members felt that the reward part of the system did not distinguish between poor and excellent performance.

However, there are positive results that the university could use to improve the system. Line managers seemed to be confident about their understanding and skills to implement the system. A clear link between job description, or category, and performance objectives as well as reducing the number of performance objectives would address line managers’ current perceptions of the amount of time the exercise of setting objectives takes.

After training, line managers could also clarify the range of performance standards and the link between these standards and payment to administrative staff.

Three factors made implementing the PFP system difficult at this university.

Firstly, after decades of incremental increases, administrative staff found themselves in the unfamiliar territory of performance scrutiny. Secondly, a major salary adjustment in 2006 masked the effects of the PFP system. Thirdly, the university introduced the system in a unionised environment where collective bargaining sets pay parameters.

It will take more time to stabilise the system and attend to the unfavourable perceptions of the recipients. With a few more interventions to improve the system, the university could achieve its goals of improving the performance of employees and of rewarding excellent work.

The literature and the findings of this evaluation confirm that guidance, support and training are vital for the PFP system to work at the university. This factor relates directly to programme implementation and its importance. It seems that the PFP system at the university was ineffective because of poor implementation rather than poor design.

The evaluators mentioned earlier that line managers and HR often disagree and deny their responsibilities when implementing the PFP system. As a result, implementing the system usually suffers. Implementers often ignore the human dynamics of implementation. Therefore, most of the recommendations this evaluation makes relate to issues of interaction and communication between administrative employees, line managers and HR. These recommendations are practical and will clarify roles as well as improve implementation. Better implementation seems to be the key to an effective PFP system at the university.

Suggestions for future research
The university used the recommendations in this evaluation to improve the implementation of its PFP system in 2009. The evaluators suggest that the university conducts a systematic process and outcome evaluation to assess whether the system is now achieving its intended outcomes.

Acknowledgements

Competing interests
The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them when they wrote this paper.

Authors’ contributions
L.J. (University of Cape Town) and J.L.-P. (University of Cape Town) completed the first part of the evaluation as contractors. K.E. (University of Cape Town) completed the second part of the evaluation as a requirement for her master’s degree in programme evaluation. J.L.-P. (University of Cape Town) supervised the evaluation.

References

Armstrong, M. (2003). Employee reward. London: Chartered Institute of Personnel and Development.

Armstrong, M. (2006). A handbook of human resource management practice. London: Kogan Page.

Barnes-Farrell, J.L., & Lynch, A.M. (2003). Performance appraisal and feedback programs. In J.E. Edwards, J.C. Scott & N.S. Raju (Eds.), The human resources program-evaluation handbook, (pp. 155–176). Thousand Oaks: Sage.

Belcher, J.G. (1996). How to design and implement a results-orientated variable pay system. New York: AMACOM.

Brown, D., & Purcell, J. (2007). Reward management: On the line. Compensation and Benefits Review, May, 28–34. http://dx.doi.org/10.1177/0886368707302649

Bruns, W.J. (1996). Performance measurement, evaluation, and incentives. Boston: Harvard Business School Press.

Cannell, M., & Wood, S. (1992). Incentive pay: Impact and evolution. London: IPD and NEDO.

Doran, T. (2008). Lessons from early experience with pay for performance. Journal of Disease Management & Health Outcomes, 16, 69–78. http://dx.doi.org/10.2165/00115677-200816020-00001

Dowling, B., & Richardson, R. (1997). Evaluating performance-related-pay for managers in the National Health Service. International Journal of Human Resource Management, 8, 348–366. http://dx.doi.org/10.1080/095851997341685

Edwards, J.E., Scott, J.C., & Raju, N. (2003).The human resources program-evaluation handbook. Thousand Oaks: Sage Publications.

Eichinger, R.W., Lombardo, M.M., & Ulrich, D. (2004). 100 things you need to know: Best people practices for managers and HR. Minneapolis: Lominger Limited.

Gomez-Mejia, L., Wiseman, L.M., & Dykes, B.J. (2005). Agency problems in diverse contexts: A global perspective. Journal of Management Studies, 42, 1507–1517. http://dx.doi.org/10.1111/j.1467-6486.2005.00554.x

Harper, C.S. (2003). Adding purpose to performance reviews. Journal of Training and Development, September, 53–55.

Henderson, I.R. (2006). Compensation management in a knowledge-based world. (10th edn.). New Jersey: Pearson Education.

Holbeche, L. (2004). Aligning human resources and business strategy. Amsterdam: Elsevier.

Kettner, P.M., Moroney, R.M., & Martin, L.L. (1999). Designing and managing programs. An effectiveness-based approach. (2nd edn.). London: Sage Publications.

Lawler, E.E. (1990). Strategic pay: Aligning organizational strategy and pay systems. San Francisco: Jossey-Bass.

Pallant, J. (2007). SPSS: Survival Manual. (3rd edn.). New York: McGraw Hill.

Pennings, J.M. (1993). Executive reward systems: A cross-national comparison. Journal of Management Studies, 30, 261–280. http://dx.doi.org/10.1111/j.1467-6486.1993.tb00304.x

Purcell, J., & Hutchinson, S. (2007). Front-line managers as agents in the HRM-performance causal chain: Theory, analysis and evidence. Human Resource Management Journal, 17, 3–20. http://dx.doi.org/10.1111/j.1748-8583.2007.00022.x

Reilly, P. (2003). New approaches in reward: Their relevance to the public sector. Journal of Management Studies, 30, 245–253.

Rosenthal, M.B., Landon, B.E., Howitt, K., Song, H.R., & Epstein, A.M. (2007). Climbing up the pay-for-performance learning curve: Where are the early adopters now? Journal of Health Affairs, 26, 1674–1682. http://dx.doi.org/10.1377/hlthaff.26.6.1674 , PMid:17978386

Schwab, D.L., & Olson, C.A. (1990). Merit pay practices: Implications for pay-performance relationships. Industrial and Labor Relations Review, 43, 237–253. http://dx.doi.org/10.2307/2523581

Scriven, M. (1991). Evaluation thesaurus. (4th edn.). Newbury Park: Sage.

Streib, G., &. Nigro, L.G. (1993). Pay for performance in local governments: Programmatic differences and perceived utility. Public Productivity & Management Review, 17, 145–159. http://dx.doi.org/10.2307/3380510

Ulrich, D., & Brockbank, W. (2005). The HR value proposition. Boston: Harvard Business School Press.

University of Cape Town. (2006). Salary assessment resource guide. Rondebosch: Cape Town.

University of Cape Town. (2006). Performance development resource guide. Rondebosch: Cape Town.

White, G., & Druker, J. (2000). Rewards management: A critical text. London: Routledge.

Wright, A. (2004). Reward management in context. London: Chartered Institute of Personnel and Development.