I am assembling a list of questions/comments I think are unfair to be directed towards seminar speakers (with reasons why they should be considered unfair). If you disagree with my assessment or have questions you want to be added to this list, please reach out!
- “Your effect size is too big. Therefore, there must be something wrong with your identification strategy / approach.”
Explanation: You should assess empirical findings by how credible the identification strategy / approach is. If you only focus on effect size to infer whether identification is credible, publications will eventually contain too many false positives with small effect sizes and lack true positives with large effect sizes. Note that (as David Jaeger pointed out) this is a compound statement: Noting a large effect size per se is okay. But it is not sufficient to dismiss the study’s findings. Paul Huenermund remarked that it is important to also consider factors beyond the identification strategy, such as sample size. That’s why the statement above now includes the word “approach”: to reflect all choices/factors of one’s estimations that may lead to a lack of credibility of the findings. - “What you find is not surprising.”
Explanation: Even if not surprising, as scientists we have a duty to show what is true and what is not and what defies or confirms conventional wisdom. If we focus on surprising findings only, we risk publishing too many articles containing false surprising positives and we will publish too few true, yet unsurprising results. - “Why is this economics?”
Explanation: This question will put the presenter in the uncomfortable position to tell you how the setting contains some sort of allocational inefficiency or influences other parameters that eventually are part of economic decisions. Given that almost every resource is scarce and we can express almost everything in money, the presenter will likely have a good answer ready, such that you don’t learn anything from asking your question. And even if it’s not strictly economics, why do you care? As a researcher, you work in the business of expanding knowledge, irrespective of the label! - “It doesn’t look random.” (when showing raw data, e.g. across space)
Explanation: What “looks” random and doesn’t is subjective. Randomness also is not equal to uniformity. If you doubt the randomness of something, evaluate the quality of the identification strategy instead. - “Will you show us … ?” (anticipatory question in the introduction that will definitely be addressed later in the talk)
Explanation: Do you really need an explanation? - “Why should we care about the results of your study in country X? My home country has a completely different institutional feature…”
Explanation: Economics is a global discipline and institutional differences may shape different behaviors. Only because it is not relevant in your country doesn’t make it irrelevant for everyone on the planet. - “Your model is wrong.”
Explanation: Every model is wrong. Treat the presenter with some courtesy.