What empirical social science should aim for: lessons from physics

I frequently complain that empirical social science is obsessed with estimating parameters (eg. demand elasticities) that are rarely compared or generalized.  This makes it hard (or at least unreliable) to use empirical results in theory and prediction.

Recently, I ran into the paper Thermal Conductivity of the Elements: A Comprehensive Review (1974) by Ho, Powell and Liley (don't ask how). It seems like an excellent example of what we, as empirical social scientists, should be aiming for.

The paper is, as advertised, is a comprehensive review (circa 1974) of a specific class of basic physical parameters.  It takes parameter estimates from a huge number of lab studies and tries to synthesize what was known about a these basic parameters.  For a sense of how much research went into uncovering these basic parameters, the paper has 1626 references (this dwarfs anything I've ever seen in the Journal of Economic Literature).

I think that producing an analog of this figure (for essential variables) should be the "near-term" objective of empirical social scientists:

Click to enlarge

Each number denotes the results of a different study.  Note the dark black curve of "Recommended" values that can only be stated with confidence following such a meta-analysis.  How much work went into this single figure of a single parameter? The paper tabulates and summarizes every paper is uses, here's one of the pages the describes the studies from this figure. (The last entry on this page is numbered "190", but it's not the last page in the multi-page table.)

Click to enlarge

One thing that's so impressive about this review is that its not just about copper.  It has a similarly in-depth study of each element. For example, tungsten:

Click to enlarge

One thing that jumps out of this figure is the range of estimates produced by individual studies. Even in the laboratory sciences, it's hard to produce a perfect experiment.  If any one of the estimates above was taken as "perfect," it might produce a very distorted picture of reality.  In economics (the social science I know best), such experimental repetition is discouraged in favor of completely fresh (and clever) studies of novel phenomena. While this setup produces studies that are more entertaining in a seminar, I don't think these incentives are optimal for our science in the long-run.

Finally, its worth noting that it took a lot of hard work by many scientists to obtain the basic parameters pictured above (see the paper for many more).  In social science, where we didn't figure out how to reliably identify parameters in real or quasi-experiments until a few decades ago, we'll need to swallow our pride and recognize that we have a lot of catching up to do. (For example, note that Holland's paper popularizing Rubin's model was published in 1986, more than ten years after this review paper was published.)  I think that if we're honest with ourselves, most of the parameters in social science (besides, maybe, the wage returns to schooling...) are understood about as well as the thermal conductivity of erbium was in 1974.  Following Ho et al., most of our current parameter estimates should probably be explicitly termed "Provisional":

Click to enlarge