Choosing experiments to accelerate collective discovery

How efficient are research agendas?

Abstract: A scientist’s choice of research problem affects his or her personal career trajectory. Scientists’ combined choices affect the direction and efficiency of scientific discovery as a whole. In this paper, we infer preferences that shape problem selection from patterns of published findings and then quantify their efficiency. We represent research problems as links between scientific entities in a knowledge network. We then build a generative model of discovery informed by qualitative research on scientific problem selection. We map salient features from this literature to key network properties: an entity’s importance corresponds to its degree centrality, and a problem’s difficulty corresponds to the network distance it spans. Drawing on millions of papers and patents published over 30 years, we use this model to infer the typical research strategy used to explore chemical relationships in biomedicine. This strategy generates conservative research choices focused on building up knowledge around important molecules. These choices become more conservative over time. The observed strategy is efficient for initial exploration of the network and supports scientific careers that require steady output, but is inefficient for science as a whole. Through supercomputer experiments on a sample of the network, we study thousands of alternatives and identify strategies much more efficient at exploring mature knowledge networks. We find that increased risk-taking and the publication of experimental failures would substantially improve the speed of discovery. We consider institutional shifts in grant making, evaluation, and publication that would help realize these efficiencies.

The paper is Rzhetsky et al.'s 2015 - Choosing experiments to accelerate collective discovery. (via Shanee)

Third Interdisciplinary Ph.D. Workshop in Sustainable Development

Third Interdisciplinary Ph.D. Workshop in Sustainable Development
April 12th-13th, 2013: Columbia University in the City of New York, USA

The graduate students in sustainable development at Columbia University are convening the Third Interdisciplinary Ph.D. Workshop in Sustainable Development (IPWSD); scheduled for April 12th-13th, 2013, at Columbia University in New York City.

The IPWSD is a conference open to graduate students working on or interested in issues related to sustainable development.  It is intended to provide a forum to present and discuss research in an informal setting, as well as to meet and interact with similar graduate student researchers from other institutions.  In particular, we hope to facilitate a network among students pursuing in-depth research across a range of disciplines in the social and natural sciences, to generate a larger interdisciplinary discussion concerning sustainable development.  If your research pertains to the field of sustainable development and the linkages between natural and social systems, we encourage you to apply regardless of disciplinary background.

For details, please see the call for papers, or visit our conference website where a detailed list of topics, conference themes and other information is available. 

website: http://blogs.cuit.columbia.edu/sdds/schedule-events/ipwsd_2013/
contact: cu.sdds.ipwsd@gmail.com

Note that the submission deadline has been extended to February 15th.

If scientists had a Book of Psalms, it would be this book

While wandering through the Princeton bookstore, I stumbled upon this gem. The Oxford Book of Modern Science Writing by Richard Dawkins will become a treasure of the scientific community. Dawkins gathers 83 choice writing excerpts from the "Greats" of scientific writing (e.g. Pinker, Diamond, Turing, Einstein, Sagan, Penrose, Greene, Hawking, Chandrasekhar, Sacks, Oppenheimer, Wilson, Carson, Dyson, Snow... the whole list is here). The excerpts are each short (a few pages) but masterfully chosen, and Dawkins provides a brief discussion of each writer and their style before presenting the text.  The selected excerpts discuss many of the central philosophical questions/insights of science, as well as many of its key contributions -- so readers are educated about actual science in addition to seeing how to write about it beautifully.

The book is thick, and I haven't finished it myself, but I can't recommend it enough for anyone who considers themselves a scientist.  If science were art, this text would be like a distillation of the best masterworks from the world's best museums into a potent liquor that makes you feel guilty when you read from it because it is so rich and amazing -- representing much of humanity's collective accomplishments -- and undeservingly, you're still just sitting on your couch.

If you're looking for a holiday gift for a scientist, I would recommend this. Or if you're a scientist whose annoyed that your loved ones didn't buy you this book for the holidays, you can read a lot of it for free on google here.

An aside: If I ever get the chance, I hope to lead a seminar/clinic for phd students on scientific communication.  I think this book on writing will round out the curriculum alongside Tufte's book on data display and Baron's book on communicating verbally.

Sexism in science persists, it is unacceptable and female mentors exhibit a substantially larger bias than male mentors

This is a recent, elegant and upsetting PNAS paper:

Science faculty’s subtle gender biases favor male students
Corinne A. Moss-Racusin, John F. Dovidio, Victoria L. Brescoll, Mark J. Grahama, and Jo Handelsman

Abstract: Despite efforts to recruit and retain more women, a stark gender disparity persists within academic science. Abundant research has demonstrated gender bias in many demographic groups, but has yet to experimentally investigate whether science faculty exhibit a bias against female students that could contribute to the gender disparity in academic science. In a randomized double-blind study (n = 127), science faculty from research-intensive universities rated the application materials of a student—who was randomly assigned either a male or female name—for a laboratory manager position. Faculty participants rated the male applicant as significantly more competent and hireable than the (identical) female applicant. These participants also selected a higher starting salary and offered more career mentoring to the male applicant. The gender of the faculty participants did not affect responses, such that female and male faculty were equally likely to exhibit bias against the female student. Mediation analyses indicated that the female student was less likely to be hired because she was viewed as less competent. We also assessed faculty participants’ preexisting subtle bias against women using a standard instrument and found that preexisting subtle bias against women played a moder- ating role, such that subtle bias against women was associated with less support for the female student, but was unrelated to reactions to the male student. These results suggest that interventions addressing faculty gender bias might advance the goal of increasing the participation of women in science.

The authors construct a fake job application for a hypothetical undergraduate student who is applying to work as a scientific technician/lab manager in a laboratory (this is a common stepping-stone to entering a doctoral program and becoming a PhD researcher). The authors randomly assign a male or female name to the applicant and distribute the application to principle investigators (PhD scientists who run real labs), asking them to score the applicant on a variety of metrics such as "competence" and "hireability." The only difference between applications is the gender of the student. The results are unambiguous:

Click to enlarge

It is possible that one could construct an explanation for why researchers would mentor male students more, without invoking sexism -- eg. perhaps if the scientist believes a female student is more likely to leave the field, they will feel like there is less personal reward for spending time mentoring female students. But (and this is where I congratulate the authors for a well-designed experiment) there is no way that differences in "competence" scores can be explained without sexism.

The authors then ask these principle investigators how much they would be willing to pay the applicant to work in their lab:

Click to enlarge

The authors write: 

Finally, using a previously validated scale, we also measured how much faculty participants liked the student (see SI Materials and Methods). In keeping with a large body of literature, faculty participants reported liking the female (mean = 4.35, SD = 0.93) more than the male student [(mean = 3.91, SD = 0.1.08), t(125) = −2.44, < 0.05]. However, consistent with this previous literature, liking the female student more than the male student did not translate into positive perceptions of her composite competence or material outcomes in the form of a job offer, an equitable salary, or valuable career mentoring.

Every teacher, research and mentor in the sciences should read the paper (open access here) and do some soul-searching, asking themselves if they consciously or subconsciously discriminate against female students, employees in the lab or colleagues.  In addition, I also think we all have the responsibility to keep one another honest and to make one another aware of situations and decisions when we might mistakenly judge our students, employees or peers based on their sex and not on their scientific merit.

Overall, this paper is carefully designed and convincing with writing that is thoughtful and readable.  Just a few comments before proceeding:

1. The title of the paper describes the results as a "subtle" bias. But as my fiance (a PhD) points out, if effects of this size were found in any other context, they would held up as "big" effects.  I think the use of the word "subtle" is a bit confusing, since I think the authors are referring to the bias being subconscious, rather than referring to the magnitude of the bias (which is not subtle).
2. Even if these biases are subconscious, they are still sexist. I understand why the authors don't use this language in a published paper, but in discussing these results in the context of our own conduct, it seems important to not shy away from what is really going on. Describing these results as a subconscious bias, rather than sexism, may make them seem more excusable.
3. The central contribution of the paper is to simply point out what is going on.  Once we are aware of our own biases, especially if they are subconscious, we can make a conscious effort to correct them. But in addition to each of us reflecting on our own actions, there are some easy institutional mechanisms that can be developed to help us avoid these biases. For example, universities could make it easy for scientists to receive job applications through an electronic system that automatically double-blinds the applicant.  (Many peer-reviewed journals do this when sending papers out to referees.)  Unfortunately, it is harder to protect students and researchers in day-to-day activities, so if sexist treatment persists even after the hiring process, this will be harder to address. But much more certainly could be done. For example, it should be standard that an oversight committee anonymously surveys students and employees regularly to determine if there is statistical evidence of discrimination within departments or individuals laboratories (which tend to behave a bit like small fiefdoms, with little to no oversight of the principle investigator's behavior towards students/employees).  The NSF and various funding agencies frequently award money to labs for "teaching and mentoring," so they should make these anonymous evaluations and their analysis (or something similar) a requirement for this funding.

The authors are careful to check whether sexism is a strictly male phenomenon (i.e. men faculty discriminating against female students). They do this by constructing this table:

Click to enlarge

The authors find that both male and female mentors exhibit sexism. But the authors do not push the data as far as they could as they make no statements about whether the bias of male mentors is larger or smaller than that of female mentors.  The authors write:

In support of hypothesis B, faculty gender did not affect bias (Table 1). Tests of simple effects (all d < 0.33) indicated that female faculty participants did not rate the female student as more competent [t(62) = 0.06, P = 0.95] or hireable [t(62) = 0.41, P = 0.69] than did male faculty. Female faculty also did not offer more mentoring [t(62) = 0.29, P = 0.77] or a higher salary [t(61) = 1.14, P = 0.26] to the female student than did their male colleagues. In addition, faculty participants’ scientific field, age, and tenure status had no effect (all P < 0.53). Thus, the bias appears pervasive among faculty and is not limited to a certain demographic subgroup.

And later in the discussion:

Our results revealed that both male and female faculty judged a female student to be less competent and less worthy of being hired than an identical male student, and also offered her a smaller starting salary and less career mentoring. Although the differences in ratings may be perceived as modest, the effect sizes were all moderate to large (d = 0.60–0.75). Thus, the current results suggest that subtle gender bias is important to address because it could translate into large real-world dis- advantages in the judgment and treatment of female science students (39). Moreover, our mediation findings shed light on the processes responsible for this bias, suggesting that the female student was less likely to be hired than the male student because she was perceived as less competent. Additionally, moderation results indicated that faculty participants’ preexisting subtle bias against women undermined their perceptions and treatment of the female (but not the male) student, further suggesting that chronic subtle biases may harm women within academic science. Use of a randomized controlled design and established practices from audit study methodology support the ecological validity and educational implications of our findings (SI Materials and Methods). 

It is noteworthy that female faculty members were just as likely as their male colleagues to favor the male student. The fact that faculty members’ bias was independent of their gender, scientific discipline, age, and tenure status suggests that it is likely un- intentional, generated from widespread cultural stereotypes rather than a conscious intention to harm women (17). Additionally, the fact that faculty participants reported liking the fe- male more than the male student further underscores the point that our results likely do not reflect faculty members’ overt hostility toward women. Instead, despite expressing warmth to- ward emerging female scientists, faculty members of both genders appear to be affected by enduring cultural stereotypes about women’s lack of science competence that translate into biases in student evaluation and mentoring.

Now here is a bit that I am adding. Looking at Table 1, it seemed like the bias for female mentors was larger, but it was hard to tell based on the layout of the table (you have to hold the differences in your head since they aren't written down). This caught my attention because its an issue that was raised by Anne Marie-Slaughter's recent Atlantic article, which I had discussed extensively with family and friends.

So I copied the data from Table 1 into Excel, reorganized it and explicitly compared the magnitude of the bias based on the gender of the faculty-mentor. In the first two panels, the column "difference" is the magnitude of the bias in favor of male students. In the bottom panel, I compare these biases and take their difference (a difference-in-differences) to see whether male or female mentors are more biased (a positive number means female faculty are more biased). The last column lists how much larger the bias is for female faculty relative to male faculty.



Both male and female faculty exhibit sexism. But across all four measures, female faculty exhibit a larger bias then male faculty (I don't have the raw data, so I can't know if the difference is statistically significant in this sample -- but, the direction of the bias is clearly consistent across measures).  Without any evidence that the male student has more merit, the female faculty member is on average 15% more biased when it comes to evaluating whether the applicant is competent; and the female faculty member offers the male student an additional $920 in salary on top of the $3,400 extra that the male faculty offered him. Now, I am not trying to point the finger at women faculty to distract from the fact that male faculty are sexist. Discrimination by either group is unacceptable. I am simply trying to highlight one additional point that was skipped over in the original analysis and which Anne Marie-Slaughter argues is an important (but under-discussed) obstacle for professional women.

The findings of this study are important, and they indicate that all of us, men and women alike, should take a cold hard look at our own decisions, behaviors and tendencies. Sexism of this magnitude and scale, among some of the most highly educated members of society is unacceptable. If you observe a colleague who treats their male and female students differently, or if you see that they run a lab full of happy young men and miserable young women, take them aside and ask them what is going on.  It is not easy to call a colleague out on these things, but they would probably rather hear it from you than an internal review board -- and more importantly, we owe it to our students and employees who work hard for us and look up to their mentors for education, guidance and leadership.

It is morally indefensible that sexism of this magnitude persists in our scientific communities and that the young women who are discriminated against suffer at the hands of their teachers and mentors.  Moreover, we all lose out every time that a talented young woman, who would have made scientific discoveries benefitting the world, leaves science because of discrimination.

Read this book!

So you have an extra 23 dollars and a few hours to fill? My recommendation: change your life and read this book.

Steven Gaines recommended "Escape from the Ivory Tower" (by Nancy Baron) to me and it has made me a better communicator, a better writer, and probably a better researcher.

Baron is a scientist-turned-science-writer and puts together a quick read that helps us awkward and detail-oriented scientists pretend that we are smooth operators doing research that everyone should care about.

The book basically has two components. First, she helps you understand how journalists, policy-makers and normal humans see the world and, more importantly, how they think about scientific research. This alone helped me dramatically improve how I frame my work.

Second, she then lays out a whole bunch of practical tools to help you think through how you should present your research, from how to structure a paper summary to how to handle telephone/TV interviews and what to expect when talking with policy-types.

And since Baron is a pro on writing, the book is an unsurprisingly snappy and entertaining read full of excellent quotes.

I can't recommend this book enough. If I ever get the chance to teach a class on research methodology, I swear that I will require that everyone read this book.

Other books in the make-yourself-a-better-communicator series: graphics and climate.

An unusual number of goodies in Nature this week

A special Outlook issue reviewing the challenges of malaria control:

Nature Outlook: Malaria (Open access)

The war against the malaria parasite has raged for millennia, and still claims hundreds of thousands of lives each year. Resistance is a growing issue — for both the parasite to current therapy, and the mosquito to pesticides. Past attempts to eradicate malaria have failed. What will it take to finally subdue this deadly disease?

Commentary on the ivory tower:

Global issues: Make social sciences relevant
Luk Van Langenhove

Excerpt:

The social sciences are flourishing. As of 2005, there were almost half a million professional social scientists from all fields in the world, working both inside and outside academia. According to the World Social Science Report 2010 (ref. 1), the number of social-science students worldwide has swollen by about 11% every year since 2000, up to 22 million in 2006. 

Yet this enormous resource is not contributing enough to today's global challenges, including climate change, security, sustainable development and health. These issues all have root causes in human behaviour: all require behavioural change and social innovations, as well as technological development.... 

Despite these factors, many social scientists seem reluctant to tackle such issues. And in Europe, some are up in arms over a proposal to drop a specific funding category for social-science research and to integrate it within cross-cutting topics of sustainable development. This is a shame — the community should be grasping the opportunity to raise its influence in the real world.... 

Today, the social sciences are largely focused on disciplinary problems and internal scholarly debates, rather than on topics with external impact.... 

The main solution, however, is to change the mindset of the social-science community, and what it considers to be its main goal. If I were a student now, I would throw myself at global challenges and social innovations; I hope to encourage today's young researchers to do the same.

Meta-analysis of a famous question:

Comparing the yields of organic and conventional agriculture
Verena Seufert, Navin Ramankutty & Jonathan A. Foley

Abstract: Numerous reports have emphasized the need for major changes in the global food system: agriculture must meet the twin challenge of feeding a growing population, with rising demand for meat and high-calorie diets, while simultaneously minimizing its global environmental impacts1, 2. Organic farming—a system aimed at producing food with minimal harm to ecosystems, animals or humans—is often proposed as a solution3, 4. However, critics argue that organic agriculture may have lower yields and would therefore need more land to produce the same amount of food as conventional farms, resulting in more widespread deforestation and biodiversity loss, and thus undermining the environmental benefits of organic practices5. Here we use a comprehensive meta-analysis to examine the relative yield performance of organic and conventional farming systems globally. Our analysis of available data shows that, overall, organic yields are typically lower than conventional yields. But these yield differences are highly contextual, depending on system and site characteristics, and range from 5% lower organic yields (rain-fed legumes and perennials on weak-acidic to weak-alkaline soils), 13% lower yields (when best organic practices are used), to 34% lower yields (when the conventional and organic systems are most comparable). Under certain conditions—that is, with good management practices, particular crop types and growing conditions—organic systems can thus nearly match conventional yields, whereas under others it at present cannot. To establish organic agriculture as an important tool in sustainable food production, the factors limiting organic yields need to be more fully understood, alongside assessments of the many social, environmental and economic benefits of organic farming systems.

Copyright Nature

And some interesting agent-based modeling from Nature Climate Change:

Emerging migration flows in a changing climate in dryland Africa
Dominic R. Kniveton, Christopher D. Smith & Richard Black

Fears of the movement of large numbers of people as a result of changes in the environment were first voiced in the 1980s (ref. 1). Nearly thirty years later the numbers likely to migrate as a result of the impacts of climate change are still, at best, guesswork2. Owing to the high prevalence of rainfed agriculture, many livelihoods in sub-Saharan African drylands are particularly vulnerable to changes in climate. One commonly adopted response strategy used by populations to deal with the resulting livelihood stress is migration. Here, we use an agent-based model developed around the theory of planned behaviour to explore how climate and demographic change, defined by the ENSEMBLES project3 and the United Nations Statistics Division of the Department of Economic and Social Affairs4, combine to influence migration within and from Burkina Faso. The emergent migration patterns modelled support framing the nexus of climate change and migration as a complex adaptive system5. Using this conceptual framework, we show that the extent of climate-change-related migration is likely to be highly nonlinear and the extent of this nonlinearity is dependent on population growth; therefore supporting migration policy interventions based on both demographic and climate change adaptation.

Interfacing Water, Climate, And Society: A Resource List

The Research Applications Laboratory at NCAR has a nice wiki-like resource list for folks interested in the interface between water, climate and society. The list isn't comprehensive, but its useful and has sections on:

• Undergraduate Level Degree Programs
• Graduate Level Degree Programs
• Post-graduate Opportunities
• Academic Research Groups
• Professional Development and Research Training
• Professional Networks
• Boundary Organizaions
• Journals
• References
• Funding Programs
• Conferences

Check it out here.

Edu-tainment for misanthropic referees

This is very funny, although your amusement will probably be proportional to the number of papers you've reviewed + 4 * the number of papers you've submitted and gotten rejected because of referees.

[The appendix is actually quite serious and addresses a number of real statistical issues, although several are specialized for neuroimaging.]

Ten ironic rules for non-statistical reviewers
Karl Friston

Abstract: As an expert reviewer, it is sometimes necessary to ensure a paper is rejected. This can sometimes be achieved by highlighting improper statistical practice. This technical note provides guidance on how to critique the statistical analysis of neuroimaging studies to maximise the chance that the paper will be declined. We will review a series of critiques that can be applied universally to any neuroimaging paper and consider responses to potential rebuttals that reviewers might encounter from authors or editors.

Excerpt:

There is a perceived need to reject peer-reviewed papers with the advent of open access publishing and the large number of journals available to authors. Clearly, there may be idiosyncratic reasons to block a paper – to ensure your precedence in the literature, personal rivalry etc. – however, we will assume that there is an imperative to reject papers for the good of the community: handling editors are often happy to receive recommendations to decline a paper. This is because they are placed under pressure to maintain a high rejection rate. This pressure is usually exerted by the editorial board (and publishers) and enforced by circulating quantitative information about their rejection rates (i.e., naming and shaming lenient editors). All journals want to maximise rejection rates, because this increases the quality of submissions, increases their impact factor and underwrites their long-term viability. A reasonably mature journal like Neuroimage would hope to see between 70% and 90% of submissions rejected. Prestige journals usually like to reject over 90% of the papers they receive. As an expert reviewer, it is your role to help editors decline papers whenever possible. In what follows, we will provide 10 simple rules to make this job easier: 

Rule number one: dismiss self doubt Occasionally, when asked to provide an expert opinion on the design or analysis of a neuroimaging study you might feel under qualified. For example, you may not have been trained in probability theory or statistics or – if you have – you may not be familiar with topological inference and related topics such as random field theory. It is important to dismiss any ambivalence about your competence to provide a definitive critique. You have been asked to provide comments as an expert reviewer and, operationally, this is now your role. By definition, what you say is the opinion of the expert reviewer and cannot be challenged – in relation to the paper under consideration, you are the ultimate authority. You should therefore write with authority, in a firm and friendly fashion. 

[My favorite: (emphasis added)]

Rule number two: avoid dispassionate statements A common mistake when providing expert comments is to provide definitive observations that can be falsified. Try to avoid phrases like “I believe” or “it can be shown that”. These statements invite a rebuttal that could reveal your beliefs or statements to be false. It is much safer, and preferable, to use phrases like “I feel” and “I do not trust”. No one can question the veracity of your feelings and convictions. Another useful device is to make your points vicariously; for example, instead of saying “Procedure A is statistically invalid” it is much better to say that “It is commonly accepted that procedure A is statistically invalid”. Although authors may be able to show that procedure A is valid, they will find it more difficult to prove that it is commonly accepted as valid. In short, trying to pre-empt a prolonged exchange with authors by centering the issues on convictions held by yourself or others and try to avoid stating facts. 

Rule number three: submit your comments as late as possible It is advisable to delay submitting your reviewer comments for as long as possible – preferably after the second reminder from the editorial office. This has three advantages. First, it delays the editorial process and creates an air of frustration, which you might be able to exploit later. Second, it creates the impression that you are extremely busy (providing expert reviews for other papers) and indicates that you have given this paper due consideration, after thinking about it carefully for several months. A related policy, that enhances your reputation with editors, is to submit large numbers of papers to their journal but politely decline invitations to review other people's papers. This shows that you are focused on your science and are committed to producing high quality scientific reports, without the distraction of peer-review or other inappropriate demands on your time.

[I am definitely guilty of this last one... it goes on]

My own related grievances here.

h/t Matt

It takes a community to define a discipline: the 5th anniversary of Environmental Research Letters

ERL has turned five years old and Dan Kammen has noted some successes and his vision in a short article:

The motivation for founding ERL was initially more focused: to alter the mode of publication and review in the diverse, yet linked fields of environmental and resource studies and to ensure new levels of interaction, inclusion and equity, providing the platform for the world-changing research findings published in ERL. The key driver in this conversation was the issue of access. Specifically the situation that too many research findings were produced by, and for, very specific academic 'clubs', and that the opportunity to engage in discussion and debate over important emerging findings about our world was being severely limited by the process of publication in frequently slow-to-publish and tremendously expensive traditional academic journals. 

The need for change was, and still is, obvious. Environmental and resource studies have been the fastest growing and most diverse nexus of academic research, private sector concern and public sector action. Universities worldwide are adding academic and extension professorships and staff as well as experiencing increasing student interest in this area at a record pace. Corporate social and environmental sustainability has been changing dramatically and, in lurching fits and starts, a mosaic of environmental regulations—both carrots and sticks—are emerging worldwide. The 'Rio + 20' Earth Summit in June 2012 will be a testament to both the dramatic broadening of this interest, and the frustration about the lack of progress at building strong global institutions to permit international cooperation. This is a clear call for an on-going and evolving process of community building.

Some features of ERL that I've liked, and Kammen emphasizes:

• it's open access
• they are now doing "video abstracts"
• impact factor = 3.05
• they actually are very interdisciplinary
• they actually do turn around articles very quickly

Here's the video abstract for Kammen's article:

[FE has hosted an ERL RSS feed in the sidebar for a year now, check it out.]

Unpublished comments should be taxed

I don't think any journal editors read this blog, but I'm posting this just in case one ever does.

In the scientific literature, if a scientist disagrees with a previously published finding they can submit a "comment" to the journal and this comment is handled a bit like a normal paper (eg. there is peer review). However, for some journals (eg. Nature), the editors solicit a "reply" from the original authors, which is basically a comment on the submitted comment.  If the comment-reply pair is considered useful for advancing scientific understanding of the original paper (this is where peer review comes in), then they are published together.

My coauthors and I have just finished dealing with the third comment submitted in reaction to a paper we published in August. Many people dislike our result and have tried to falsify it.  None of these comments have been published, however, because in each case our reply has demonstrated the faulty approaches of the comments.  We have learned many things from the process, so in some sense it represents the interactive component of scientific research at its best, but dealing with comments is exhausting.  When a comment is submitted, we must read it, think about it, conduct simulations and additional analysis to demonstrate its faults, write up the results, edit the write up and compose a formal review.  Given my recent experience, I would guess that each comment consumes about four person-days of work between myself and my coauthors (costing ~$550 at post-doc-like wages).  For time-constrained researchers, this is a lot of time.  And for us, the authors of the original paper being commented on, there is not much flexibility in timing since journals ask for responses to be submitted quickly (Nature gives ten days).  So comments demand a serious time-commitment from the original researchers on short notice.

In cases where the original research really had serious issues that are clarified by comments, then this process produces valuable public goods. But in cases where the response is strong enough to prevent the comment's publication (our recent three experiences), then public confusion is avoided (a public good relative to the counterfactual state) at the expense of the original researchers' time and effort.  This is a clear externality imposed on the original researchers by the commenting authors.  As we know from Econ 101, this will lead to the overproduction of low-quality (rejected) comments in equilibrium, since the full social cost of a comment (the lost time of the original authors) is not borne by the commenting authors.  What should be done here? The answer is obvious from Econ 101: we need a Pigovian tax on low-quality comments.  We should penalize commenting authors if their comment is rejected, but not if it is accepted.  This will force authors with a comment to think hard about the quality of their comment before submitting it and will raise the quality of the average comment that is actually submitted in equilibrium.  Society will still get the public good benefits from the strong comments, but authors of original papers will be less burdened by dealing with the excessive supply of low-quality comments.  In the current no-tax regime, the oversupply of low-quality comments generates a pure deadweight loss in the form of occupying the original researchers' valuable time and slowing down their research.

What should the tax be? Well, based on my earlier estimate, roughly $550 for a comment that is rejected.  This is on par with the marginal cost of publishing an additional color figure (~$450 in Nature).  How can this tax be implemented and enforced? When a comment is submitted, a $550 bond must be paid at the time of submission. If the comment is published, the bond is returned to the authors. If the comment is rejected, the journal keeps the bond. (Or, if they are feeling nice and want to maximize social welfare, they give it to the original/replying authors! Under this model, I would be $1650 dollars richer and less annoyed at spending my time replying...)

Empirical evidence that hard work can make the world a better place

Sometimes when it's the weekend and you need a motivational break from furiously analyzing-data-while-writing-two-papers-while-applying-for-jobs-while-refereeing-stuff-and-washing-dishes, a movie on a historical success does the trick.  Here are some of my favorites flavors of inspiration-caffeine [despite their historical inaccuracies...]:

Amazing Grace - ending slavery in the British Empire
(one of my favorite quotes is Edmund Burke on these events)

Ghandhi - liberating India and Pakistan

The Great Debaters - civil rights in the US
(maybe I like this because I met my fiance on our high-school's science olympiad team...)

Invictus - reconciliation in post-apartheid South Africa
(I remembered to post this because I was jogging in my Springbok's jersey!)

The best research quote ever

 Theories have four stages of acceptance:

1. This is worthless nonsense;
2. This is interesting, but perverse;
3. This is true, but quite unimportant;
4. I always said so. 

-J.B.S. Haldane

Credit to this book, which I'll review when I finish it. More research quotes here.

Climate CoLab

Hannah Lee sent me this interesting project put together by the MIT Center for Collective Intelligence: the Climate CoLab.

What should we do about climate change?
Somehow we have to answer this question. You can help.
The Climate CoLab seeks to harness the collective intelligence of contributors from all over the world to address global climate change.
The Climate CoLab is a forum where teams create proposals for what to do in a series of annual contests.

The current contest is to respond to the prompt: "How should the 21st century economy evolve, bearing in mind the risks of climate change?"  I'm excited to learn what our collective intelligence has to say about this.

In unrelated but hilarious news, see this comic about doing science courtesy of Reed Walker. (If you're not an active researcher then this probably isn't funny to you, sorry.)

Make better graphs

I rarely actually finish reading books. But when I do it means they're useful and interesting, so I'll tell you when that happens.

I finally finished Tufte's famous The Visual Display of Quantitative Information (here on Amazon).  There are gazillions of reviews online, so I'll be brief.

If you are an applied researcher and you plan on making your living doing statistics or theory and writing papers, ask each of your colleagues for $1 so you can buy and read this book.  They will gladly pay up, since it will make deciphering your future papers much more pleasant.

The book is overpriced and Tufte's writing style is patronizing. But the book is still worth reading and if I ever teach a stats class, it will be assigned.  If the book does nothing else, it brings together a wide array of statistical graphics from various fields, graphs I never would have found on my own, and critiques them.  The critiques sound like the kind of thing that photographers, artists and designers do for each other all the time.  But its uncommon for us stats-types to ever really talk about the organization and aesthetics of our graphs. I was surprised how useful the exercise was for me.

Tufte is a designer and he names his rules-of-thumb-for-graphs things that only a designer would come up with.  But he's not dumb and this is a book for applied statisticians.  He doesn't do any fancy math, but he assumes you know what variance, a marginal distribution and r-squared are on page 2.

If you pride yourself on being a pragmatist with no taste for unnecessary glitz, you may be surprised to find that Tufte-the-designer is on your side.  All of his tools and tips help you focus on the data and avoid unnecessary decorations in your graphs.  He even critiques several of the "recommended graph styles" used in top journals as being over-drawn.

Recommendation: buy the book and read it.  It's easy to read and will make all of your future work more valuable.  Constructing a single smoking graph as the centerpiece in each paper/presentation is one of the most useful things you can do as a researcher.  And this book will help you get there.  I didn't like all of his advice, but I'm certain that he altered how I approach my own graphs (and I'm someone who already spent a lot of time thinking about their design).

Besides, even if you end up hating the book, at least you can appreciate its beauty: it's probably the only statistics book that I would consider leaving out on the coffee table when guests are over.

Important questions vs. interesting questions

In a recent conversation over drinks with a colleague, we got into the classic debate of whether researchers should select interesting questions or important questions. There are lots of "research advice" documents out there discussing this tradeoff (eg. this one by Donald Davis, which DRDR recently pointed out), but I don't believe there really is a tradeoff per se.

Here's what I think my opinion is (for now, at least): there are many interesting but un-important questions but there are no important but un-interesting questions.

If a question really is important for society, then it is by definition interesting.  If an important question remains unanswered, it is probably because it is difficult to answer.  But if it really is important, then there will be many people interested in its answer (so long as you can come up with a good one).  Clearly, this means the question is interesting.

I think there are two reasons that important questions sometimes get mistakenly categorized as "uninteresting" by researchers. First, researchers sometimes have a strategic incentive to label a question they do not know how to answer as uninteresting, since it gives them justification for not investigating the problem.  Second, questions may be so difficult to answer that we assume any answer will be unsatisfying and thus uninteresting.  In both these cases, I think we confuse the difficulty of a problem with whether that problem is interesting. 

It's fine with me if some academics want to work on interesting but unimportant questions, I don't see any way to stop this from happening.  But I get angry when researchers thoughtlessly dismiss as uninteresting a question that they are willing to recognize is important.

Quotes for the discouraged researcher

Research is hard.  And keeping oneself inspired to keep at it is even harder.  These are some favorites that keep me going.  (If these aren't enough, see Xavier Sala-i-Martin's hilarious collection of rejected ideas.)

"Somewhere, something incredible is waiting to be known."
- Carl Sagan

"The significant problems we face cannot be solved at the same level of thinking we were at when we created them."
- Albert Einstein

"I have not failed. I have just found 10,000 ways that won't work."
-Thomas Edison on attempting to make an incandescent light bulb

"Good and bad research take the same amount of time."
- Some economist?
(tell me who if you know). Larry Summers

"If you want to make incremental contributions to the literature in your field — let us say, if you want to generalize an existence theorem by relaxing the condition of semi-strict quasi-concavity to one of mere hemi-demi-proper pseudoconcavity — then stick to the technical journals. If you want to change your field in more fundamental ways, then obtain your primary motivation from life, and use it to look for fundamental shortcomings of previous thinking in the field."
- Avinash Dixit describing Thomas Schelling after he won the Nobel Prize
(I owe this one to Matt Notowidigdo)

"He well knows what snares are spread about his path, from personal animosity... and possibly from popular delusion.  But he has put to hazard his ease, his security, his interests, his power, even his... popularity.... He is traduced and abused for his supposed motives.  He will remember that obloquy is a necessary ingredient in the composition of all true glory: he will remember...that calumny and abuse are essential parts of triumph.... He may live long, he may do much.  But here is the summit.  He never can exceed what he does this day."
-Edmund Burke's eulogy of Charles James Fox for his attack upon the tyranny of the East India Company, House of Commons, December 1, 1783
(I owe this one to JFK)

"Good sense is, of all things among men, the most equally distributed; for every one thinks himself so abundantly provided with it, that those even who are the most difficult to satisfy in everything else, do not usually desire a larger measure of this quality than they already posess. And in this it is not likely that all are mistaken: the conviction is rather to be held as testifying that the power of judging aright and of distinguishing truth from error, which is properly what is called good sense or reason, is by nature equal in all men; and that the diversity of our opinions, consequently, does not arise from some being endowed with a larger share of reason than others, but solely from this, that we conduct our thoughts along different ways, and do not fix our attention on the same objects. For to be possessed of a vigorous mind is not enough; the prime requisite is rightly to apply it. The greatest minds, as they are capable of the highest excellences, are open likewise to the greatest aberrations....
I will not hesitate, however, to avow my belief that it has been my singular good fortune to have very early in life fallen in with certain tracks which have conducted me to considerations and maxims, of which I have formed a method that gives me the means, as I think, of gradually augmenting my knowledge, and of raising it by little and little to the highest point which the mediocrity of my talents and the brief duration of my life will permit me to reach."
-Rene Descartes, Discourse on Method, 1637