Author Manzi tells us that too often we allow policies to be guided by inflexible ideology. His alternatives include software that incorporates the experimental method for guiding business decision-making, and now this book that argues the same methods could be applied to important social issues.
Much of Manzi's material within the book is taken up explaining why simply using historical data, surveys, etc. is inadequate at best, due to the substantial and unquantified impact of factors not included within the 'model.' Similarly, regarding multiple linear regression analyses, the seemingly favorite tool of everyone with access to a computer and SPSS software. (Added problems with regression analysis - the need to incorporate non-linear variables such as binary zero-one to account for the presence/absence of a specific attribute or policy; interactions, and factors better represented as exponential variables, etc.) Thus, Manzi's rationale for instead using controlled experiments.
He cites Capitol One (credit-card firm that reportedly conducts 60,000 experiments/year), Gary Loveman's management at Harrah's Entertainment, and Google's sophisticated programs for evaluating alternative ad wording, etc. Unfortunately, he failed to elaborate on how any of those three leaders have utilized randomized experimentation. 'Uncontrolled' would be stronger if he'd also pointed out that there's a danger in these approaches - simply that the firm/government entity becomes overly focused on incremental improvement instead of breakthroughs. Had Henry Ford only followed these approaches, the best he could have accomplished would have been to breed a faster horse; similarly, Toyota never would have developed its revolutionary Toyota Production System that, unfortunately, helped decimate Detroit while simultaneously and significantly improving quality, response times, and costs.
Manzi's major recommendation is that government mandate randomized experiments to evaluate social policy proposals - especially when waiver approvals are being considered. He envisions as many as 10,000 such experiments/year. An excellent suggestion! Another - that we prioritize science and technology. As for treating immigration as a recruiting opportunity - his elaboration seemed to be simply a ruse for legalizing illegal immigration, a major contributor to today's high unemployment rates.
Good as Manzi's book is, he's still guilty of significantly overselling his positions. For example, his writing makes it abundantly clear that he's on the conservative end of the spectrum - that shouldn't even be mentioned or discernible in a book that must be seen as non-partisan if it is to be adopted to social science use. Secondly, he too quickly dismisses the potential use of thousands of non-random experiments, especially in education. Objective summaries in those areas offer immediate and credible support for pulling the plug on the belief that more money improves education outcomes. More importantly, the availability of such data for at least 50-some years underscores an important problem with even Manzi's ideal of using randomized data - the entrenched simply ignore and/or denigrate the findings. On the conservative side of education, Manzi further errs when he drops in an plug for vouchers - at best, quality experiments and analyses on this topic have found little if any benefit to doing so.
Still another area replete with quasi-experimental data - American health care. We lead the world by far in expenditures (18% GDP, vs. #2 at 12%), but lag in important preventable patient outcomes. Manzi, I presume, would prescribe multiple doses of experimentation to find out why. This is not needed - the differences between the U.S. and other nations are already obvious. Every other developed nation utilizes strong government regulation to control providers' ability to take advantage of the strong inelasticity of demand within that sector. (Same problem exists in education - also largely uncontrolled in America.) On the other hand, when it comes to evaluating treatment efficacy, randomized clinical trials similar to Manzi's thinking are the gold standard for practitioner guidance, as well as defanging demagoguery ala 'Death Panel' rhetoric.
One more point - since Manzi referenced Premier Deng's aphorism about black cats/white cats in 'Uncontrolled,' I'd like to cite another early lesson from Deng. Deng frequently encountered fierce resistance to proposals for major economic reform. Rather than remain stalemated or risk violent confrontation, he proposed, like Manzi, experiments. Such an approach would limit any potential damage, and could be isolated to relatively unpopulated boundary areas with existing local champions. Though these experiments weren't even on the same scale of sophistication as Mr. Manzi's used to and proposes, the improvements were obvious and fast-coming. Six 'Special Economic Zones' were approved in 1980 to conduct major, but non-random, experiments that greatly transcended Maoist thought. My point - further evidence that complex analyses such as Mr. Manzi champions aren't always needed. The keys in this success story were pragmatism, reliance on data, and physical isolation.
Bottom-Line: Some good points, but needing more examples and flexibility, along with less conservative bias.
