Looking good on paper is low indicator of effective lawyering
By Marjorie Shultz and Sheldon Zedeck, Daily Journal
People interested in professional education, preparation and selection are increasingly seeking outcome or performance-oriented measures rather than using input measures or paper credentials alone. For legal employers, that means improved methods for hiring decisions. For professional development managers, that means finding better tools to assess deficiencies and plan for needed training. For those making decisions about compensation and promotion, it means more detailed and targeted measures of actual job performance, as well as better means to predict expected professional growth.
In a 10-year study involving thousands of practicing lawyers and law graduates doing law-related jobs, we (longtime professors at UC Berkeley and UC Berkeley School of Law) empirically developed and validated methods to identify and assess characteristics conducive to effective lawyering. Beginning with interviews of five constituencies (lawyers, law students and faculty, judges, and clients of UC Berkeley School of Law) and continuing through multiple rounds of focus groups in San Francisco, Los Angeles and Washington, D.C., we identified 26 factors important to lawyering effectiveness. (The 26 factors in random order are: analysis and reasoning; creativity; problem solving; practical judgment; writing; interviewing and questioning; listening; managing one's own work; managing work of others; diligence; integrity; stress management; influencing and advocating; self-development; ability to see the world through the eyes of others; negotiation skills; networking and business development; client advising and counseling; building relationships in the profession; passion and engagement; researching the law; speaking; evaluation, development and mentoring; finding and using facts; strategic planning; community involvement and service.)
Next, we used multiple additional focus groups of lawyers to generate detailed examples of lawyer behavior that they described as illustrations of below average, average and above average lawyer performance on each of the 26 factors. We then asked law graduates (over 2000 responded, from various work settings and subject specialties, from graduates ranging from two years out of law school to those 30 plus years out) to rate from scratch the levels of effectiveness of those same behavioral examples on a 1 ("poor") - 5 ("excellent") scale for a particular factor. The survey gave us validation of the effectiveness levels of the behavioral examples initially produced in focus groups.
Using only those examples whose ratings showed a high degree of agreement among survey respondents, we constructed rating scales for each of the 26 factors that could be used to evaluate an individual attorney's job performance. Behavioral examples were arrayed along the 1-5 scale from high to low on the basis of their mean rating by the survey participants. The different behavioral examples and their mean ratings provide "anchors" for each scale. Behaviorally anchored scales provide a common and concrete frame of reference for raters, reducing the impact of "halo effects" and stereotypes that often influence other types of performance ratings.
Next, we looked for tests that we hypothesized could predict individuals' propensity to display the 26 identified attorney competencies. We examined a wide range of existing tests, chose some, adapted others, and created others from scratch, eventually melding eight types of tests into a new online pilot test. Then, in order to validate the new tests, we recruited law graduates (again from all settings and specialties and from a 30 year range of time who graduated from either UC Berkeley School of Law or UC Hastings College of the Law) who were willing to take the new test battery.
In addition, participants gave us permission to access their academic history (UGPA, LGPA, LSAT score) and provide a self assessment as well as the names of two supervisors and two peers who could evaluate their current performance on the job using the scales and factors described above. Just over 1100 law graduates took the test and over 4000 supervisor, peer and self-appraisals of the job performance of those 1100 law graduates were returned.
After all the numbers were crunched, results suggested that our new tests (particularly two of them) predicted almost all (24) of the 26 factors identified in the first phase of research as being important to lawyer effectiveness. By contrast, LSAT scores, UGPA, and Index Scores predicted fewer than 10 of the job-effectiveness factors, with several of the relationships being negative! Positive correlations between LSAT/Index Scores and job performance factors were mainly with those factors that are most academic in nature (analysis and reasoning, writing, researching, etc.).
Improving the racial and ethnic diversity of law school matriculants or of legal employers' staffs are important additional benefits of new tests like those reported here. As most lawyers are aware, academically-oriented tests like the LSAT produce a substantial adverse impact on under-represented racial and ethnic minorities. By contrast, results on tests that aim to predict effectiveness of professional performance show little practical difference between performance of various racial and ethnic groups. This outcome is congruent with much of the employment research literature. To pass legal scrutiny, hiring tests that create an adverse impact on underrepresented minorities must show job-relatedness. Decades of work in the employment sector shows that conducting a job analysis to identify end points that employers seek to predict, then devising tests of the actual job skills, often significantly reduces differences in race and ethnic group performance. We decided to import some of the theory and practice from employment law into legal selection systems.
Because our research and our tests were initially directed to improving admission decision-making in law schools, their use in that context should improve the representation of minorities in the pool of law school matriculants. That in turn should give legal employers better options to improve the diversity of their professional staffs. In addition to greater numbers of minority graduates included in hiring pools, legal recruiters who themselves adopt standards less tied to the academic rank of a school or student and more aimed at effective professional performance might gain the confidence to search deeper into classes or ranks of law schools to find prospective employees.
Legal employers know that grades and test scores don't tell them all that they need to know. Nor do informal interviews yield a significant amount of information for hiring. While those are indicators of certain job-related strengths, they are incomplete at best. Use of some form of job effectiveness evaluation, or of behavioral interviewing based on needed lawyer competencies, are emerging as better methods to assess candidates for jobs that are becoming scarce. Legal organizations can no longer afford to hire based on paper academic credentials and wait to see who emerges as useful and who should be let go months or years down the road.
You may be interested in learning more about the study discussed here. Under the sponsorship of the State Bar's Council on Access and Fairness, we are going to be doing focus groups up and down the state in October in order to talk with representatives of legal organizations (including representatives from law firms, in house corporate counsel, government/public sector offices and public interest organizations) about how our research on lawyer effectiveness can be helpful to legal employers, particularly in connection with their efforts to diversify and retain their professional staffs. Later in the fall, we will be convening symposia both in the northern and the southern parts of California based on the input from the focus groups and to discuss our research. We hope as many of you as possible will participate as invitees in the focus groups and in the symposia as they are scheduled. 8/10/2011