Why Experimental Policy

Or Scientific Policy, Policy Hypothesis Testing... naming is hard

A Story to Start

I am a fairly good citizen researcher when it comes to voting on California Propositions. I look at a couple voting guides, usually LA Times, KnockLA, and CADems, look at the endorsements, read the legislative summary, and the case for and against.

But invariably the two most useful thing to do are:

1. Go to CalMatters and see the short non-partisan video on the proposition, read their summary, and take the matching quiz e.g. main page, video, quiz

2. Read the “Fiscal Summary” that is included with each proposition e.g. Prop 33

Now the California Proposition system is incredibly flawed and having to wade through dozens of propositions remains tedious but, I believe, there is the seed here of a better way of engaging in politics. That solution is to focus on intelligible policy and testable hypotheses.

Why Not Something Else?

I have a wise friend who considers that the best we mere mortals can do is throw our support behind the least bad candidate and let policy be determined by inevitable horse trading between elected representatives. This view is not without merit and perhaps near the limit of what most of us can or want to do today but evidence of this approach’s inadequacy abounds. It seems like we’ve collectively succumb to the mass delusion that this is the best we can do— merely select the “great men of history” and leave policy to them.

As I’ve discussed before, I think this is the wrong model. Politics today even in it’s best form is the triumph of tribalism and magical thinking. It is a game won by charismatic storytellers who can craft compelling, but not necessarily realistic, narratives of the present and visions of the future and assemble an identifiable “us” and “them”.

Law as Hypothesis Testing

In Engineering Open Societies, this is why my focus is on policy rather than saying building coalitions or electing the right politicians. Because policy seems to offer a path forward rather than a dead end. The alternatives to policy are people, parties, and vibes none of which seem to offer a path to improvement.

There are no silver bullets, but we can’t make progress without tools and processes that allow for iterative improvement. And I believe the current nature of politics is explicitly and systemically hostile to this sort of iterative refinement.

A Short Glossary

I use the term policy, but it’s imprecise. When I say policy i mean an experiment conducted on society by society with a clearly defined set of hypotheses. I’ve called this “scientific policy” before, “experimental policy” feels more appropriate though.

Law as Hypothesis Testing

It’s a bad sign I need a glossary for my own idea. All terms as always subject to future refinement.

• Prescription: Our “policy” or “law” the change we are proposing making e.g. Repeals Costa-Hawkins Rental Housing Act of 1995, which currently prohibits local ordinances limiting initial residential rental rates for new tenants or rent increases for existing tenants in certain residential properties.

• Hypothesis: Our basic building block tied to a prescription. A testable hypothesis as to what will happen e.g. If approved Prop 33 will reduce property tax revenues in Los Angeles by $30M and San Francisco by $60M annually, will result in a 0.1% decrease in homelessness in the state, will result in a decline of new home construction from 300,000 units to 200,000 units. The key thing to note here is for any given policy prescription there are many hypotheses, we want to capture the holistic impact through many discrete hypotheses..

• Forecasts: So far we are talking about testing hypotheses but really there are always a minimum of four distinct hypotheses (forecasts) put forward (similar to the arguments and rebuttals in the California propositions e.g. Prop 33 arguments and rebuttals). There is the proponents’ positive hypothesis, the contrarians’ positive hypothesis, and the null or status quo hypothesis for each side e.g. if approved, Prop 33 will decrease homelessness (pro argument), Prop 33 will increase homelessness (con rebuttal), if not approved homelessness will decrease (con argument), if not approved homeless will remain the same (pro rebuttal). Honestly, I don’t have this piece figured out yet but in the vein of Superforecasting it seems important to capture the disagreement about what a policy will do for later evaluation.

• Experimental Policy: A set of hypotheses linked to one or more policy prescriptions as part of a unified legislative package or policy agenda. Due to the messy nature of reality these are always net effects e.g. if we enact rent control, change the minimum units required for affordable unit requirements for new constructions, and removing zoning restrictions we have to consider the aggregate effect of these linked policy prescriptions as well as the effect of each separate policy prescription.

Back to Why

The world is always messy, and data from natural experiments will always be subject to caveats, but testable policy hypotheses offer a pathway to iterative improvement that seems so absent from the pseudo-random walk of politics.

Experimental policy offers a better approach. Free of morality, tribalism, resistant to human frailty. It offers the same path of iterative improvement through experimentation that has propelled technology and value creation over the past two centuries.

Thinking in terms of policy frees us from a number of evils that beset the political process:

• Robust: Policy is free from the fragility of individuals. People are subject to death, dishonor, distraction, and dithering. Policies are more robust than their proponents.

• Persuasive: Engagement on policy is the way to transform fixed opinions into flexible curiosity. Ideally, policy should be something that anyone can grasp and hold in their hands if we do X, the outcome will be Y, after doing x the outcome was y. What we know about our own psychology supports this. Don’t try and change peoples’ minds at all but get them thinking about the problem and let their rigid opinions make way for more flexible problem solving as they get engaged.

• Concrete: The availability of high quality policy prescriptions is an antidote to magical thinking. Today when a politician engages in wishful thinking and chooses to make policy out of whole cloth the resistance to this amounts to assembling, de novo, various research publications, small-scale experiments, and historical anecdotes to combat the confabulation. There is no definitive resource and process that can be pointed to that has evaluated and worked through the topic and made a high quality assessment of impact.

• Inclusive: Technology enables global collaboration on policy at various scales with different targeting. There is no need for great men, or special interests to craft policy and estimate impact when that process can occur in full view with a record of results. This also eliminates horse trading and blunts the opaque influence of special interests. They must reveal themselves to participate and defend their reasoning.

• Scalable: Policies can be developed and experiments ran in parallel there is no blockade on the individual legislators bandwidth to craft, review, and evaluate policy.

• Prosaic: High quality policy prescriptions would help to break the cycle of novelty and outrage. You cannot constantly scream “think of the children” when the magnitude of the problem and the impact of the proposed solution are well-documented. Wikipedia cannot eliminate historical lies on social media but it can tamp down their impact, fight the spread of wildfire, and ultimately extinguish the misinformation. Similarly, the existence of “good” experimental policy prescriptions and experiments could not stop the spread of poorly-conceived, misguide, and disingenuous policy proposals and unsupportable hypotheses.

Next

Having address the “why” hopefully to some extent although always incompletely, I will feel on much firmer ground moving next onto the “how”. Our next piece will lay a firm foundation for experimental policy and from there we’ll go into actually crafting a software platform for experimental policy. I hope you join me on the journey.