Free legal advice - but does it work?

How do you improve America's resource-poor network of legal services? Harvard professor James Greiner believes you have to put it to the test.

Almost every morning, staffers at Greater Boston Legal Services arrive at work to find a line of people waiting on the street outside their offices. These people are there because they need help: some because they're being evicted, others because they're chasing down child support payments, still others because they're filing for divorce.

Lawyers are expensive, and for millions of low-income people across the United States, nonprofits like Greater Boston Legal Services offer the best - perhaps the only - chance at professional help. Staffed by civic-minded attorneys and paid for with public money and private donations, these organizations represent our society's primary mechanism for making sure that when it comes to civil proceedings, all people, including the very poor, are treated equally before the law.

For decades, it has been an article of faith among those who have devoted their lives to the cause of legal aid that if only the system had more funding, it could do more good and help more people. But lately, a difficult new question is being asked of the legal services community: What evidence do they have that the help they're offering even makes a difference - and that they're allocating their scarce resources as effectively as possible?

That is the challenge being laid down by a group of critics led by James Greiner, a professor at Harvard Law School. Greiner believes passionately in providing free legal assistance to the poor, but he is dismayed by what he sees as a lack of data on how it's delivered and how it affects people's lives. For all the good they think they're doing, Greiner argues, the fact is that legal services providers are working off untested assumptions and operating largely in the dark.

Given the noble intentions and tireless dedication of the individuals who staff legal clinics around the country, it feels almost indecent to question the effectiveness of the help they're providing. But from where Greiner sits, the sanctity of these efforts - and the fact that the system provides for just two attorneys per 10,000 low-income people - is precisely why it matters. In particular, as he sees it, we know very little about how legal service providers ought to determine which clients to take and which to turn down; in many cases, he argues, they may be wasting precious time and money on cases where they're unlikely to have an impact.

"Most folks have concluded that we're never going to be able to give a full attorney-client relationship to every person who has a legal problem," Greiner said last week. "The funding is just never going to be there. So you have to take steps short of that, in an attempt to meet the need."
For Greiner, the solution is one that has already revolutionized other fields, most notably medicine. He is part of a budding movement that wants to introduce randomized experiments and put hard numbers to what have traditionally been treated as unquantifiable social problems. In medicine, he and his allies point out, such tests have revealed that once-common treatments - hormone replacement therapy, for instance - were not only ineffective but harmful. "The mission," Greiner said, "is to make law more evidence-based, more rational and scientific."

Among those who labor in the legal services world, this insistence on randomized testing has been met with skepticism, even defensiveness - particularly after Greiner's first study on the topic found that one group of people who were offered representation by law students working at the Harvard Legal Aid Bureau were no more likely to prevail in their court cases than people who were not.

Those results came as a jarring surprise to some on the bureau's staff, and the study sparked consternation among legal aid workers all over the country. Some worried that Greiner's findings would be used as justification to defund legal aid, while others raised an ethical objection to the very idea of collecting the kind of data he'd like to see: It amounts to experimenting with vulnerable people's lives.

Randomized trials are the gold standard in the world of science, and for good reason: Done right, they can generate highly precise insights about how the world works. To determine the effectiveness of a new drug, researchers give it to some patients and not others; if the people in the first group fare significantly better than the control group, that sends a clear signal about the efficacy of the medicine.

So far, these types of controlled experiments have proven difficult to run on social problems. For Greiner, a soft-spoken man who favors turtleneck sweaters and loves Harry Potter, finding ways to bring the power of quantitative analysis to bear on the legal system has been a calling since 2002, when he decided, after working as a litigator, to get a doctorate in statistics. But it was not until a few years ago that the recently tenured professor turned his attention to legal services, having become convinced that in order for our democracy to truly work, all people - not just those who can afford it - must have "access to justice."

What Greiner found when he began studying legal services was a complex, decentralized, and woefully underfunded endeavor that was helping only a fraction of the people in need. Across the country, hundreds of organizations worked more or less separately, without sharing information in any systematic way about the kinds of people who were coming to them for help and what exactly they needed. Most importantly, Greiner noted, no one had ever done any rigorous tests to see how much good these organizations were actually doing.

When he first approached the Harvard Legal Aid Bureau, an organization staffed by law students, Greiner made his pitch this way: Given that the number of people who come to HLAB looking for help far exceeds the number they can offer representation, why not randomize the sorting process, create a control group, and try to learn something about the organization's impact? Initially, Greiner proposed running the study using either domestic abuse cases or eviction cases.

The answer from HLAB's leaders was a firm no. "We pretty quickly decided we weren't willing to do that," said faculty director David Grossman, explaining that the student volunteers simply had too much confidence that their sorting methods were working to jettison them in favor of a random number generator. These methods, which took into account not only the likelihood that HLAB could make a difference in a particular case, but also the urgency of the case and the number of people affected by its outcome, were based on years of experience and institutional knowledge. Given the high stakes in eviction and domestic abuse cases, Grossman said, "randomizing away" clients HLAB would take under normal circumstances felt deeply immoral.

HLAB's reluctance pointed to one of the main hurdles Greiner is facing as he looks for more legal services providers to work with on experiments: For an organization to do what he's asking involves an admission that what they do normally might be ineffective. "Given our experience, we feel like we can pick the cases where we can make a difference," said Greater Boston Legal Services executive director Jacquelynne Bowman. "But if it's randomized, then we may miss out on providing representation to someone who truly needs it."

Convincing professionals to doubt their methods is never easy, said Greiner, but it might be especially hard in the world of legal services. "Is it surprising that not every single legal aid attorney possesses the kind of self-doubt that might cause her to desire to undergo a rigorous evaluation of her...activities?" he asked. Nevertheless, Greiner believes this kind of self-doubt is important, precisely because people in the trenches can't always look objectively at what they're doing. "I don't deny that...experiential observation is worthwhile," he said. "But my job is to question and test it, because it turns out that human beings sometimes stink at figuring out what works and what doesn't from day-to-day observations."

The Harvard Legal Aid Bureau ultimately agreed to work with Greiner when he proposed running a study on people seeking unemployment benefits. This was an area of their practice they were already unsure of, said Tim Visser, the group's president, with low enough stakes that the moral issues were not prohibitive. According to Visser, the students saw potential in Greiner's study. In theory, it promised to reveal patterns that would tell them what to look for in clients.

Greiner's surprising results, which were later published in the Yale Law Journal, indicated that people who were offered legal help from HLAB were no more likely to prevail in their cases - and worse, had to wait longer for their cases to be resolved - than those who were not. "That right there was a revelation," said Greiner. More than 90 percent of people offered HLAB's help received it; among those turned away, at least 39 percent found representation elsewhere. The fact that both groups scored a win rate of around 70 percent meant that getting help from HLAB was not increasing people's chances as much as had been assumed.

HLAB swiftly responded to the study, sending a letter to colleagues in which they explained their participation and pointed out what they regarded as flaws in Greiner's methodology. The letter was meant in part to address the concerns of legal aid workers around the country who had begun to worry about the study's political implications. "When I first read the HLAB study and when I first heard Jim speak, I thought, ‘Oh, no. This is going to provide all kinds of fodder for people who just hate lawyers, or people who just hate poor people, and it's going to cause all kinds of problems for us,'" said Steven Eppler-Epstein, the executive director of Connecticut Legal Services. But after speaking to Greiner, Eppler-Epstein concluded that the professor had a real, long-range plan to improve the legal services world, not hurt it, and today he is one of Greiner's closest allies.

Greiner himself knows that on the surface, his study looks bad for legal services. But he cautions against jumping to easy conclusions, and insists that what his findings actually indicate is that people who were savvy enough to approach the aid bureau in the first place were already more likely to do well. So the deeper need may lie elsewhere. "There's a whole set of people who were not even contacting HLAB," Greiner said. "And my speculation is that they're the ones who actually needed the help."

Greiner has initiated five more randomized experiments since the HLAB study. One showed that having a lawyer in eviction cases made it much more likely a client would get to keep his home. Others are currently in progress: One is about divorce, one involves people seeking social security disability benefits, and one involves federal prisoners making civil rights complaints. Greiner's hope is to devote his career to running randomized studies across the country; over time, he says, the results will accumulate into a profound source of practical knowledge for service providers. The question is, how much time will it take?

"One of my concerns is he's working at such a level of detail...that while the findings are valuable and, certainly, as I understand it, scientifically unassailable, I don't know how much they tell us about what we should or should not be doing on a day to day basis in legal aid," said Lonnie Powers, head of the Massachusetts Legal Assistance Corporation, which provides funding to providers. "Nor do I know how many decades we'd have to spend doing studies like this to really get effective information that covers the whole scope of what legal services provides."

Ultimately, Greiner and his allies hope to reform the legal services industry from the ground up. One thing Greiner suspects his work will ultimately show is that lawyers, per se, are not needed to handle all legal problems. Instead of trying to raise money to hire more lawyers, he said, we ought tackle the scarcity problem by identifying the types of cases that are less complex and thus could be competently handled by people whose time is less expensive - like paralegals.

Getting there will require a lot more randomized testing, Greiner said. And more testing will require an open mind on the part of the overworked, underfunded industry it is putting under the microscope.

"It really comes down to political will and the desire to find out the objective truth," Greiner said. "It's hard to persuade people who have invested so much time and money into constructing something that they think works...that it may not be doing what they think it's doing."

By Leon Neyfakh

Source: The Boston Globe