replication experimental economics

• Search Menu

Sign in through your institution

• Advance articles
• Featured articles
• Virtual Issues
• Prize-Winning Articles
• Browse content in A - General Economics and Teaching
• Browse content in A1 - General Economics
• A11 - Role of Economics; Role of Economists; Market for Economists
• A12 - Relation of Economics to Other Disciplines
• A13 - Relation of Economics to Social Values
• Browse content in B - History of Economic Thought, Methodology, and Heterodox Approaches
• Browse content in B4 - Economic Methodology
• B41 - Economic Methodology
• Browse content in C - Mathematical and Quantitative Methods
• Browse content in C1 - Econometric and Statistical Methods and Methodology: General
• C10 - General
• C11 - Bayesian Analysis: General
• C12 - Hypothesis Testing: General
• C13 - Estimation: General
• C14 - Semiparametric and Nonparametric Methods: General
• C18 - Methodological Issues: General
• Browse content in C2 - Single Equation Models; Single Variables
• C21 - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions
• C22 - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes
• C23 - Panel Data Models; Spatio-temporal Models
• C26 - Instrumental Variables (IV) Estimation
• Browse content in C3 - Multiple or Simultaneous Equation Models; Multiple Variables
• C31 - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
• C32 - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
• C33 - Panel Data Models; Spatio-temporal Models
• C36 - Instrumental Variables (IV) Estimation
• Browse content in C4 - Econometric and Statistical Methods: Special Topics
• C40 - General
• C43 - Index Numbers and Aggregation
• C44 - Operations Research; Statistical Decision Theory
• C45 - Neural Networks and Related Topics
• C49 - Other
• Browse content in C5 - Econometric Modeling
• C52 - Model Evaluation, Validation, and Selection
• C53 - Forecasting and Prediction Methods; Simulation Methods
• C55 - Large Data Sets: Modeling and Analysis
• Browse content in C6 - Mathematical Methods; Programming Models; Mathematical and Simulation Modeling
• C61 - Optimization Techniques; Programming Models; Dynamic Analysis
• C62 - Existence and Stability Conditions of Equilibrium
• C63 - Computational Techniques; Simulation Modeling
• Browse content in C7 - Game Theory and Bargaining Theory
• C70 - General
• C71 - Cooperative Games
• C72 - Noncooperative Games
• C73 - Stochastic and Dynamic Games; Evolutionary Games; Repeated Games
• C78 - Bargaining Theory; Matching Theory
• Browse content in C8 - Data Collection and Data Estimation Methodology; Computer Programs
• C81 - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
• Browse content in C9 - Design of Experiments
• C90 - General
• C91 - Laboratory, Individual Behavior
• C92 - Laboratory, Group Behavior
• C93 - Field Experiments
• C99 - Other
• Browse content in D - Microeconomics
• Browse content in D0 - General
• D00 - General
• D01 - Microeconomic Behavior: Underlying Principles
• D02 - Institutions: Design, Formation, Operations, and Impact
• D03 - Behavioral Microeconomics: Underlying Principles
• Browse content in D1 - Household Behavior and Family Economics
• D11 - Consumer Economics: Theory
• D12 - Consumer Economics: Empirical Analysis
• D13 - Household Production and Intrahousehold Allocation
• D14 - Household Saving; Personal Finance
• D15 - Intertemporal Household Choice: Life Cycle Models and Saving
• D18 - Consumer Protection
• Browse content in D2 - Production and Organizations
• D21 - Firm Behavior: Theory
• D22 - Firm Behavior: Empirical Analysis
• D23 - Organizational Behavior; Transaction Costs; Property Rights
• D24 - Production; Cost; Capital; Capital, Total Factor, and Multifactor Productivity; Capacity
• Browse content in D3 - Distribution
• D30 - General
• D31 - Personal Income, Wealth, and Their Distributions
• D33 - Factor Income Distribution
• Browse content in D4 - Market Structure, Pricing, and Design
• D40 - General
• D42 - Monopoly
• D43 - Oligopoly and Other Forms of Market Imperfection
• D44 - Auctions
• D47 - Market Design
• Browse content in D5 - General Equilibrium and Disequilibrium
• D50 - General
• D52 - Incomplete Markets
• Browse content in D6 - Welfare Economics
• D60 - General
• D61 - Allocative Efficiency; Cost-Benefit Analysis
• D62 - Externalities
• D63 - Equity, Justice, Inequality, and Other Normative Criteria and Measurement
• D64 - Altruism; Philanthropy
• Browse content in D7 - Analysis of Collective Decision-Making
• D70 - General
• D71 - Social Choice; Clubs; Committees; Associations
• D72 - Political Processes: Rent-seeking, Lobbying, Elections, Legislatures, and Voting Behavior
• D73 - Bureaucracy; Administrative Processes in Public Organizations; Corruption
• D74 - Conflict; Conflict Resolution; Alliances; Revolutions
• D78 - Positive Analysis of Policy Formulation and Implementation
• Browse content in D8 - Information, Knowledge, and Uncertainty
• D80 - General
• D81 - Criteria for Decision-Making under Risk and Uncertainty
• D82 - Asymmetric and Private Information; Mechanism Design
• D83 - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
• D84 - Expectations; Speculations
• D85 - Network Formation and Analysis: Theory
• D86 - Economics of Contract: Theory
• D87 - Neuroeconomics
• D89 - Other
• Browse content in D9 - Micro-Based Behavioral Economics
• D90 - General
• D91 - Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making
• D92 - Intertemporal Firm Choice, Investment, Capacity, and Financing
• Browse content in E - Macroeconomics and Monetary Economics
• Browse content in E0 - General
• E01 - Measurement and Data on National Income and Product Accounts and Wealth; Environmental Accounts
• E02 - Institutions and the Macroeconomy
• Browse content in E1 - General Aggregative Models
• E13 - Neoclassical
• Browse content in E2 - Consumption, Saving, Production, Investment, Labor Markets, and Informal Economy
• E20 - General
• E21 - Consumption; Saving; Wealth
• E22 - Investment; Capital; Intangible Capital; Capacity
• E23 - Production
• E24 - Employment; Unemployment; Wages; Intergenerational Income Distribution; Aggregate Human Capital; Aggregate Labor Productivity
• E25 - Aggregate Factor Income Distribution
• E26 - Informal Economy; Underground Economy
• E27 - Forecasting and Simulation: Models and Applications
• Browse content in E3 - Prices, Business Fluctuations, and Cycles
• E30 - General
• E31 - Price Level; Inflation; Deflation
• E32 - Business Fluctuations; Cycles
• E37 - Forecasting and Simulation: Models and Applications
• Browse content in E4 - Money and Interest Rates
• E43 - Interest Rates: Determination, Term Structure, and Effects
• E44 - Financial Markets and the Macroeconomy
• Browse content in E5 - Monetary Policy, Central Banking, and the Supply of Money and Credit
• E51 - Money Supply; Credit; Money Multipliers
• E52 - Monetary Policy
• E58 - Central Banks and Their Policies
• Browse content in E6 - Macroeconomic Policy, Macroeconomic Aspects of Public Finance, and General Outlook
• E61 - Policy Objectives; Policy Designs and Consistency; Policy Coordination
• E62 - Fiscal Policy
• E63 - Comparative or Joint Analysis of Fiscal and Monetary Policy; Stabilization; Treasury Policy
• E65 - Studies of Particular Policy Episodes
• Browse content in F - International Economics
• Browse content in F0 - General
• F01 - Global Outlook
• F02 - International Economic Order and Integration
• Browse content in F1 - Trade
• F10 - General
• F11 - Neoclassical Models of Trade
• F12 - Models of Trade with Imperfect Competition and Scale Economies; Fragmentation
• F13 - Trade Policy; International Trade Organizations
• F14 - Empirical Studies of Trade
• F15 - Economic Integration
• F16 - Trade and Labor Market Interactions
• F17 - Trade Forecasting and Simulation
• Browse content in F2 - International Factor Movements and International Business
• F20 - General
• F21 - International Investment; Long-Term Capital Movements
• F22 - International Migration
• F23 - Multinational Firms; International Business
• Browse content in F3 - International Finance
• F31 - Foreign Exchange
• F32 - Current Account Adjustment; Short-Term Capital Movements
• F33 - International Monetary Arrangements and Institutions
• F34 - International Lending and Debt Problems
• F35 - Foreign Aid
• Browse content in F4 - Macroeconomic Aspects of International Trade and Finance
• F41 - Open Economy Macroeconomics
• F42 - International Policy Coordination and Transmission
• F43 - Economic Growth of Open Economies
• F45 - Macroeconomic Issues of Monetary Unions
• Browse content in F5 - International Relations, National Security, and International Political Economy
• F50 - General
• F51 - International Conflicts; Negotiations; Sanctions
• F52 - National Security; Economic Nationalism
• F53 - International Agreements and Observance; International Organizations
• F55 - International Institutional Arrangements
• Browse content in F6 - Economic Impacts of Globalization
• F60 - General
• F61 - Microeconomic Impacts
• F66 - Labor
• F68 - Policy
• Browse content in G - Financial Economics
• Browse content in G0 - General
• G01 - Financial Crises
• Browse content in G1 - General Financial Markets
• G10 - General
• G11 - Portfolio Choice; Investment Decisions
• G12 - Asset Pricing; Trading volume; Bond Interest Rates
• G15 - International Financial Markets
• Browse content in G2 - Financial Institutions and Services
• G20 - General
• G21 - Banks; Depository Institutions; Micro Finance Institutions; Mortgages
• G24 - Investment Banking; Venture Capital; Brokerage; Ratings and Ratings Agencies
• G28 - Government Policy and Regulation
• Browse content in G3 - Corporate Finance and Governance
• G30 - General
• G31 - Capital Budgeting; Fixed Investment and Inventory Studies; Capacity
• G32 - Financing Policy; Financial Risk and Risk Management; Capital and Ownership Structure; Value of Firms; Goodwill
• G33 - Bankruptcy; Liquidation
• G34 - Mergers; Acquisitions; Restructuring; Corporate Governance
• Browse content in G4 - Behavioral Finance
• G41 - Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making in Financial Markets
• Browse content in G5 - Household Finance
• G51 - Household Saving, Borrowing, Debt, and Wealth
• G53 - Financial Literacy
• Browse content in H - Public Economics
• Browse content in H0 - General
• H00 - General
• Browse content in H1 - Structure and Scope of Government
• H11 - Structure, Scope, and Performance of Government
• H12 - Crisis Management
• Browse content in H2 - Taxation, Subsidies, and Revenue
• H20 - General
• H21 - Efficiency; Optimal Taxation
• H22 - Incidence
• H23 - Externalities; Redistributive Effects; Environmental Taxes and Subsidies
• H24 - Personal Income and Other Nonbusiness Taxes and Subsidies; includes inheritance and gift taxes
• H25 - Business Taxes and Subsidies
• H26 - Tax Evasion and Avoidance
• Browse content in H3 - Fiscal Policies and Behavior of Economic Agents
• H30 - General
• H31 - Household
• Browse content in H4 - Publicly Provided Goods
• H41 - Public Goods
• H42 - Publicly Provided Private Goods
• H44 - Publicly Provided Goods: Mixed Markets
• Browse content in H5 - National Government Expenditures and Related Policies
• H50 - General
• H51 - Government Expenditures and Health
• H52 - Government Expenditures and Education
• H53 - Government Expenditures and Welfare Programs
• H54 - Infrastructures; Other Public Investment and Capital Stock
• H55 - Social Security and Public Pensions
• H56 - National Security and War
• H57 - Procurement
• Browse content in H6 - National Budget, Deficit, and Debt
• H62 - Deficit; Surplus
• H63 - Debt; Debt Management; Sovereign Debt
• H68 - Forecasts of Budgets, Deficits, and Debt
• Browse content in H7 - State and Local Government; Intergovernmental Relations
• H71 - State and Local Taxation, Subsidies, and Revenue
• H72 - State and Local Budget and Expenditures
• H75 - State and Local Government: Health; Education; Welfare; Public Pensions
• H76 - State and Local Government: Other Expenditure Categories
• H77 - Intergovernmental Relations; Federalism; Secession
• Browse content in H8 - Miscellaneous Issues
• H87 - International Fiscal Issues; International Public Goods
• Browse content in I - Health, Education, and Welfare
• Browse content in I1 - Health
• I10 - General
• I11 - Analysis of Health Care Markets
• I12 - Health Behavior
• I14 - Health and Inequality
• I15 - Health and Economic Development
• I18 - Government Policy; Regulation; Public Health
• I19 - Other
• Browse content in I2 - Education and Research Institutions
• I20 - General
• I21 - Analysis of Education
• I22 - Educational Finance; Financial Aid
• I23 - Higher Education; Research Institutions
• I24 - Education and Inequality
• I25 - Education and Economic Development
• I26 - Returns to Education
• I28 - Government Policy
• Browse content in I3 - Welfare, Well-Being, and Poverty
• I30 - General
• I31 - General Welfare
• I32 - Measurement and Analysis of Poverty
• I38 - Government Policy; Provision and Effects of Welfare Programs
• Browse content in J - Labor and Demographic Economics
• Browse content in J0 - General
• J00 - General
• J01 - Labor Economics: General
• J08 - Labor Economics Policies
• Browse content in J1 - Demographic Economics
• J10 - General
• J11 - Demographic Trends, Macroeconomic Effects, and Forecasts
• J12 - Marriage; Marital Dissolution; Family Structure; Domestic Abuse
• J13 - Fertility; Family Planning; Child Care; Children; Youth
• J14 - Economics of the Elderly; Economics of the Handicapped; Non-Labor Market Discrimination
• J15 - Economics of Minorities, Races, Indigenous Peoples, and Immigrants; Non-labor Discrimination
• J16 - Economics of Gender; Non-labor Discrimination
• J17 - Value of Life; Forgone Income
• J18 - Public Policy
• Browse content in J2 - Demand and Supply of Labor
• J20 - General
• J21 - Labor Force and Employment, Size, and Structure
• J22 - Time Allocation and Labor Supply
• J23 - Labor Demand
• J24 - Human Capital; Skills; Occupational Choice; Labor Productivity
• J26 - Retirement; Retirement Policies
• J28 - Safety; Job Satisfaction; Related Public Policy
• Browse content in J3 - Wages, Compensation, and Labor Costs
• J31 - Wage Level and Structure; Wage Differentials
• J32 - Nonwage Labor Costs and Benefits; Retirement Plans; Private Pensions
• J33 - Compensation Packages; Payment Methods
• J38 - Public Policy
• Browse content in J4 - Particular Labor Markets
• J42 - Monopsony; Segmented Labor Markets
• J43 - Agricultural Labor Markets
• J45 - Public Sector Labor Markets
• J46 - Informal Labor Markets
• J47 - Coercive Labor Markets
• Browse content in J5 - Labor-Management Relations, Trade Unions, and Collective Bargaining
• J50 - General
• J51 - Trade Unions: Objectives, Structure, and Effects
• J53 - Labor-Management Relations; Industrial Jurisprudence
• Browse content in J6 - Mobility, Unemployment, Vacancies, and Immigrant Workers
• J61 - Geographic Labor Mobility; Immigrant Workers
• J62 - Job, Occupational, and Intergenerational Mobility
• J63 - Turnover; Vacancies; Layoffs
• J64 - Unemployment: Models, Duration, Incidence, and Job Search
• J65 - Unemployment Insurance; Severance Pay; Plant Closings
• J68 - Public Policy
• Browse content in J7 - Labor Discrimination
• J71 - Discrimination
• Browse content in K - Law and Economics
• Browse content in K1 - Basic Areas of Law
• K10 - General
• K12 - Contract Law
• K14 - Criminal Law
• Browse content in K2 - Regulation and Business Law
• K21 - Antitrust Law
• Browse content in K3 - Other Substantive Areas of Law
• K36 - Family and Personal Law
• Browse content in K4 - Legal Procedure, the Legal System, and Illegal Behavior
• K40 - General
• K41 - Litigation Process
• K42 - Illegal Behavior and the Enforcement of Law
• Browse content in L - Industrial Organization
• Browse content in L0 - General
• L00 - General
• Browse content in L1 - Market Structure, Firm Strategy, and Market Performance
• L11 - Production, Pricing, and Market Structure; Size Distribution of Firms
• L12 - Monopoly; Monopolization Strategies
• L13 - Oligopoly and Other Imperfect Markets
• L14 - Transactional Relationships; Contracts and Reputation; Networks
• L15 - Information and Product Quality; Standardization and Compatibility
• L16 - Industrial Organization and Macroeconomics: Industrial Structure and Structural Change; Industrial Price Indices
• Browse content in L2 - Firm Objectives, Organization, and Behavior
• L20 - General
• L22 - Firm Organization and Market Structure
• L25 - Firm Performance: Size, Diversification, and Scope
• L26 - Entrepreneurship
• Browse content in L3 - Nonprofit Organizations and Public Enterprise
• L31 - Nonprofit Institutions; NGOs; Social Entrepreneurship
• L32 - Public Enterprises; Public-Private Enterprises
• L33 - Comparison of Public and Private Enterprises and Nonprofit Institutions; Privatization; Contracting Out
• Browse content in L4 - Antitrust Issues and Policies
• L41 - Monopolization; Horizontal Anticompetitive Practices
• L42 - Vertical Restraints; Resale Price Maintenance; Quantity Discounts
• L44 - Antitrust Policy and Public Enterprises, Nonprofit Institutions, and Professional Organizations
• Browse content in L5 - Regulation and Industrial Policy
• L51 - Economics of Regulation
• L52 - Industrial Policy; Sectoral Planning Methods
• Browse content in L6 - Industry Studies: Manufacturing
• L60 - General
• L66 - Food; Beverages; Cosmetics; Tobacco; Wine and Spirits
• L67 - Other Consumer Nondurables: Clothing, Textiles, Shoes, and Leather Goods; Household Goods; Sports Equipment
• Browse content in L8 - Industry Studies: Services
• L81 - Retail and Wholesale Trade; e-Commerce
• L82 - Entertainment; Media
• L83 - Sports; Gambling; Recreation; Tourism
• L86 - Information and Internet Services; Computer Software
• Browse content in L9 - Industry Studies: Transportation and Utilities
• L91 - Transportation: General
• L94 - Electric Utilities
• L96 - Telecommunications
• L98 - Government Policy
• Browse content in M - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics
• Browse content in M1 - Business Administration
• M10 - General
• M12 - Personnel Management; Executives; Executive Compensation
• M13 - New Firms; Startups
• Browse content in M2 - Business Economics
• M21 - Business Economics
• Browse content in M3 - Marketing and Advertising
• M30 - General
• M31 - Marketing
• Browse content in M5 - Personnel Economics
• M50 - General
• M51 - Firm Employment Decisions; Promotions
• M52 - Compensation and Compensation Methods and Their Effects
• M55 - Labor Contracting Devices
• Browse content in N - Economic History
• Browse content in N1 - Macroeconomics and Monetary Economics; Industrial Structure; Growth; Fluctuations
• N10 - General, International, or Comparative
• N12 - U.S.; Canada: 1913-
• N13 - Europe: Pre-1913
• N14 - Europe: 1913-
• N15 - Asia including Middle East
• Browse content in N2 - Financial Markets and Institutions
• N20 - General, International, or Comparative
• N23 - Europe: Pre-1913
• N26 - Latin America; Caribbean
• Browse content in N3 - Labor and Consumers, Demography, Education, Health, Welfare, Income, Wealth, Religion, and Philanthropy
• N30 - General, International, or Comparative
• N32 - U.S.; Canada: 1913-
• N33 - Europe: Pre-1913
• N34 - Europe: 1913-
• N35 - Asia including Middle East
• Browse content in N4 - Government, War, Law, International Relations, and Regulation
• N40 - General, International, or Comparative
• N41 - U.S.; Canada: Pre-1913
• N42 - U.S.; Canada: 1913-
• N43 - Europe: Pre-1913
• N44 - Europe: 1913-
• N45 - Asia including Middle East
• Browse content in N5 - Agriculture, Natural Resources, Environment, and Extractive Industries
• N50 - General, International, or Comparative
• N51 - U.S.; Canada: Pre-1913
• N53 - Europe: Pre-1913
• N55 - Asia including Middle East
• N57 - Africa; Oceania
• Browse content in N6 - Manufacturing and Construction
• N63 - Europe: Pre-1913
• Browse content in N7 - Transport, Trade, Energy, Technology, and Other Services
• N70 - General, International, or Comparative
• N71 - U.S.; Canada: Pre-1913
• N72 - U.S.; Canada: 1913-
• N73 - Europe: Pre-1913
• N75 - Asia including Middle East
• Browse content in N9 - Regional and Urban History
• N90 - General, International, or Comparative
• N92 - U.S.; Canada: 1913-
• N94 - Europe: 1913-
• N95 - Asia including Middle East
• Browse content in O - Economic Development, Innovation, Technological Change, and Growth
• Browse content in O1 - Economic Development
• O10 - General
• O11 - Macroeconomic Analyses of Economic Development
• O12 - Microeconomic Analyses of Economic Development
• O13 - Agriculture; Natural Resources; Energy; Environment; Other Primary Products
• O14 - Industrialization; Manufacturing and Service Industries; Choice of Technology
• O15 - Human Resources; Human Development; Income Distribution; Migration
• O16 - Financial Markets; Saving and Capital Investment; Corporate Finance and Governance
• O17 - Formal and Informal Sectors; Shadow Economy; Institutional Arrangements
• O18 - Urban, Rural, Regional, and Transportation Analysis; Housing; Infrastructure
• O19 - International Linkages to Development; Role of International Organizations
• Browse content in O2 - Development Planning and Policy
• O22 - Project Analysis
• O24 - Trade Policy; Factor Movement Policy; Foreign Exchange Policy
• O25 - Industrial Policy
• Browse content in O3 - Innovation; Research and Development; Technological Change; Intellectual Property Rights
• O30 - General
• O31 - Innovation and Invention: Processes and Incentives
• O32 - Management of Technological Innovation and R&D
• O33 - Technological Change: Choices and Consequences; Diffusion Processes
• O34 - Intellectual Property and Intellectual Capital
• O38 - Government Policy
• O39 - Other
• Browse content in O4 - Economic Growth and Aggregate Productivity
• O40 - General
• O41 - One, Two, and Multisector Growth Models
• O43 - Institutions and Growth
• O44 - Environment and Growth
• O47 - Empirical Studies of Economic Growth; Aggregate Productivity; Cross-Country Output Convergence
• Browse content in O5 - Economywide Country Studies
• O50 - General
• O52 - Europe
• O53 - Asia including Middle East
• O55 - Africa
• Browse content in P - Economic Systems
• Browse content in P0 - General
• P00 - General
• Browse content in P1 - Capitalist Systems
• P10 - General
• P11 - Planning, Coordination, and Reform
• P14 - Property Rights
• P16 - Political Economy
• Browse content in P2 - Socialist Systems and Transitional Economies
• P26 - Political Economy; Property Rights
• Browse content in P3 - Socialist Institutions and Their Transitions
• P39 - Other
• Browse content in P4 - Other Economic Systems
• P48 - Political Economy; Legal Institutions; Property Rights; Natural Resources; Energy; Environment; Regional Studies
• Browse content in P5 - Comparative Economic Systems
• P50 - General
• P51 - Comparative Analysis of Economic Systems
• Browse content in Q - Agricultural and Natural Resource Economics; Environmental and Ecological Economics
• Browse content in Q1 - Agriculture
• Q12 - Micro Analysis of Farm Firms, Farm Households, and Farm Input Markets
• Q14 - Agricultural Finance
• Q15 - Land Ownership and Tenure; Land Reform; Land Use; Irrigation; Agriculture and Environment
• Q16 - R&D; Agricultural Technology; Biofuels; Agricultural Extension Services
• Q17 - Agriculture in International Trade
• Q18 - Agricultural Policy; Food Policy
• Browse content in Q2 - Renewable Resources and Conservation
• Q23 - Forestry
• Q28 - Government Policy
• Browse content in Q3 - Nonrenewable Resources and Conservation
• Q32 - Exhaustible Resources and Economic Development
• Q33 - Resource Booms
• Browse content in Q4 - Energy
• Q41 - Demand and Supply; Prices
• Q48 - Government Policy
• Browse content in Q5 - Environmental Economics
• Q51 - Valuation of Environmental Effects
• Q52 - Pollution Control Adoption Costs; Distributional Effects; Employment Effects
• Q53 - Air Pollution; Water Pollution; Noise; Hazardous Waste; Solid Waste; Recycling
• Q54 - Climate; Natural Disasters; Global Warming
• Q55 - Technological Innovation
• Q56 - Environment and Development; Environment and Trade; Sustainability; Environmental Accounts and Accounting; Environmental Equity; Population Growth
• Q58 - Government Policy
• Browse content in R - Urban, Rural, Regional, Real Estate, and Transportation Economics
• Browse content in R1 - General Regional Economics
• R10 - General
• R11 - Regional Economic Activity: Growth, Development, Environmental Issues, and Changes
• R12 - Size and Spatial Distributions of Regional Economic Activity
• R13 - General Equilibrium and Welfare Economic Analysis of Regional Economies
• R15 - Econometric and Input-Output Models; Other Models
• Browse content in R2 - Household Analysis
• R21 - Housing Demand
• R23 - Regional Migration; Regional Labor Markets; Population; Neighborhood Characteristics
• Browse content in R3 - Real Estate Markets, Spatial Production Analysis, and Firm Location
• R30 - General
• R31 - Housing Supply and Markets
• Browse content in R4 - Transportation Economics
• R40 - General
• R41 - Transportation: Demand, Supply, and Congestion; Travel Time; Safety and Accidents; Transportation Noise
• Browse content in R5 - Regional Government Analysis
• R52 - Land Use and Other Regulations
• R58 - Regional Development Planning and Policy
• Browse content in Z - Other Special Topics
• Browse content in Z1 - Cultural Economics; Economic Sociology; Economic Anthropology
• Z10 - General
• Z12 - Religion
• Z13 - Economic Sociology; Economic Anthropology; Social and Economic Stratification
• Z19 - Other
• Browse content in Z2 - Sports Economics
• Z20 - General
• Author Guidelines
• Submission Site
• Open Access
• Self-Archiving Policy
• About The Economic Journal
• About the Royal Economic Society
• Editorial Board
• Advertising and Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic
• < Previous

To Replicate or Not to Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study

• Article contents
• Figures & tables
• Supplementary Data

Zacharias Maniadis, Fabio Tufano, John A. List, To Replicate or Not to Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study, The Economic Journal , Volume 127, Issue 605, October 2017, Pages F209–F235, https://doi.org/10.1111/ecoj.12527

• Permissions Icon Permissions

The sciences are in an era of an alleged ‘credibility crisis’. In this study, we discuss the reproducibility of empirical results, focusing on economics research. By combining theory and empirical evidence, we discuss the import of replication studies and whether they improve our confidence in novel findings. The theory sheds light on the importance of replications, even when replications are subject to bias. We then present a pilot meta‐study of replication in experimental economics, a subfield serving as a positive benchmark for investigating the credibility of economics. Our meta‐study highlights certain difficulties when applying meta‐research to systematise the economics literature.

Royal Economic Society members

Personal account.

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Short-term Access

To purchase short-term access, please sign in to your personal account above.

Don't already have a personal account? Register

Email alerts

Citing articles via.

• Recommend to your Librarian

Affiliations

• Online ISSN 1468-0297
• Print ISSN 0013-0133
• Copyright © 2024 Royal Economic Society
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

(Stanford users can avoid this Captcha by logging in.)

• Send to text email RefWorks EndNote printer

Replication in experimental economics

Available online.

• EBSCO Academic Comprehensive Collection

More options

• Find it at other libraries via WorldCat
• Contributors

Description

Creators/contributors, contents/summary.

• Four Classic Public Goods Experiments: A Replication Study. The Impact of Social Information on the Voluntary Provision of Public Goods: A Replication Study. On Replication and Perturbation of the McKelvey and Palfrey Centipede Game Experiment. The Impact of Financial Histories on Individuals and Societies: A Replication of and Extension of Berg et al. (1995). Revisiting the Effect of Voter Isolation. Nonverbal Feedback, Strategic Signaling, and Nonmonetary Sanctioning: New Experimental Evidence from a Public Goods Game. When and Why Matches Are More Effective Subsidies than Rebates. How to Make Experimental Economics Research More Reproducible: Lessons from Other Disciplines and a New Proposal. Replication in Experimental Economics. Research in experimental economics. Replication in Experimental Economics. Copyright page. List of Contributors. Encouraging Replication of Economics Experiments. About the Editors.
• (source: Nielsen Book Data)

Bibliographic information

• Stanford Home
• Maps & Directions
• Search Stanford
• Emergency Info
• Terms of Use
• Non-Discrimination
• Accessibility

© Stanford University , Stanford , California 94305 .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 27 August 2018

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

• Colin F. Camerer 1   na1 ,
• Anna Dreber 2   na1 ,
• Felix Holzmeister   ORCID: orcid.org/0000-0001-9606-0427 3   na1 ,
• Teck-Hua Ho 4   na1 ,
• Jürgen Huber 3   na1 ,
• Magnus Johannesson   ORCID: orcid.org/0000-0001-8759-6393 2   na1 ,
• Michael Kirchler 3 , 5   na1 ,
• Gideon Nave 6   na1 ,
• Brian A. Nosek   ORCID: orcid.org/0000-0001-6797-5476 7 , 8   na1 ,
• Thomas Pfeiffer   ORCID: orcid.org/0000-0002-0592-577X 9   na1 ,
• Adam Altmejd   ORCID: orcid.org/0000-0002-4248-0677 2 ,
• Nick Buttrick 7 , 8 ,
• Taizan Chan 10 ,
• Yiling Chen 11 ,
• Eskil Forsell 12 ,
• Anup Gampa 7 , 8 ,
• Emma Heikensten 2 ,
• Lily Hummer 8 ,
• Taisuke Imai   ORCID: orcid.org/0000-0002-0610-8093 13 ,
• Siri Isaksson 2 ,
• Dylan Manfredi 6 ,
• Julia Rose 3 ,
• Eric-Jan Wagenmakers 14 &
• Hang Wu 15  

Nature Human Behaviour volume  2 ,  pages 637–644 ( 2018 ) Cite this article

66k Accesses

823 Citations

2300 Altmetric

Metrics details

Being able to replicate scientific findings is crucial for scientific progress 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 . We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 . The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 12 digital issues and online access to articles

111,21 € per year

only 9,27 € per issue

Buy this article

• Purchase on SpringerLink
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

High replicability of newly discovered social-behavioural findings is achievable

Low replicability can support robust and efficient science

The natural selection of good science

McNutt, M. Reproducibility. Science 343 , 229 (2014).

CAS   PubMed   Google Scholar  

Baker, M. Is there a reproducibility crisis? Nature 533 , 452–454 (2016).

Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1 , 0021 (2017).

Google Scholar  

Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2 , e124 (2005).

PubMed   PubMed Central   Google Scholar  

Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10 , 712 (2011).

Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483 , 531–533 (2012).

Lawrence, M. S. et al. Mutational heterogeneity in cancer and the search for new cancer-associated genes. Nature 499 , 214–218 (2013).

CAS   PubMed   PubMed Central   Google Scholar  

Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14 , 365–376 (2013).

Maniadis, Z., Tufano, F. & List, J. A. One swallow doesn’t make a summer: new evidence on anchoring effects. Am. Econ. Rev. 104 , 277–290 (2014).

Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The economics of reproducibility in preclinical research. PLoS Biol. 13 , e1002165 (2015).

Klein, R. A. et al. Investigating variation in replicability: a ‘many labs’ replication project. Soc. Psychol. 45 , 142–152 (2014).

Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349 , aac4716 (2015).

Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351 , 1433–1436 (2016).

Ebersole, C. R. et al. Many Labs 3: evaluating participant pool quality across the academic semester via replication. J. Exp. Soc. Psychol. 67 , 68–82 (2016).

Klein, R. A. et al. Many Labs 2: investigating variation in replicability across sample and setting. Adv. Methods Prac. Psychol. Sci. (in the press).

Ackerman, J. M., Nocera, C. C. & Bargh, J. A. Incidental haptic sensations influence social judgments and decisions. Science 328 , 1712–1715 (2010).

Aviezer, H., Trope, Y. & Todorov, A. Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science 338 , 1225–1229 (2012).

Balafoutas, L. & Sutter, M. Affirmative action policies promote women and do not harm efficiency in the laboratory. Science 335 , 579–582 (2012).

Derex, M., Beugin, M.-P., Godelle, B. & Raymond, M. Experimental evidence for the influence of group size on cultural complexity. Nature 503 , 389–391 (2013).

Duncan, K., Sadanand, A. & Davachi, L. Memory’s penumbra: episodic memory decisions induce lingering mnemonic biases. Science 337 , 485–487 (2012).

Gervais, W. M. & Norenzayan, A. Analytic thinking promotes religious disbelief. Science 336 , 493–496 (2012).

Gneezy, U., Keenan, E. A. & Gneezy, A. Avoiding overhead aversion in charity. Science 346 , 632–635 (2014).

Hauser, O. P., Rand, D. G., Peysakhovich, A. & Nowak, M. A. Cooperating with the future. Nature 511 , 220–223 (2014).

Janssen, M. A., Holahan, R., Lee, A. & Ostrom, E. Lab experiments for the study of social-ecological systems. Science 328 , 613–617 (2010).

Karpicke, J. D. & Blunt, J. R. Retrieval practice produces more learning than elaborative studying with concept mapping. Science 331 , 772–775 (2011).

Kidd, D. C. & Castano, E. Reading literary fiction improves theory of mind. Science 342 , 377–380 (2013).

Kovacs, Á. M. & Téglás, E. & Endress, A. D. The social sense: susceptibility to others’ beliefs in human infants and adults. Science 330 , 1830–1834 (2010).

Lee, S. W. S. & Schwarz, N. Washing away postdecisional dissonance. Science 328 , 709 (2010).

Morewedge, C. K., Huh, Y. E. & Vosgerau, J. Thought for food: imagined consumption reduces actual consumption. Science 330 , 1530–1533 (2010).

Nishi, A., Shirado, H., Rand, D. G. & Christakis, N. A. Inequality and visibility of wealth in experimental social networks. Nature 526 , 426–429 (2015).

Pyc, M. A. & Rawson, K. A. Why testing improves memory: mediator effectiveness hypothesis. Science 330 , 335 (2010).

Ramirez, G. & Beilock, S. L. Writing about testing worries boosts exam performance in the classroom. Science 331 , 211–213 (2011).

Rand, D. G., Greene, J. D. & Nowak, M. A. Spontaneous giving and calculated greed. Nature 489 , 427–430 (2012).

CAS   Google Scholar  

Shah, A. K., Mullainathan, S. & Shafir, E. Some consequences of having too little. Science 338 , 682–685 (2012).

Sparrow, B., Liu, J. & Wegner, D. M. Google effects on memory: cognitive consequences of having information at our fingertips. Science 333 , 776–778 (2011).

Wilson, T. D. et al. Just think: the challenges of the disengaged mind. Science 345 , 75–77 (2014).

Bohannon, J. Replication effort provokes praise—and ‘bullying’ charges. Science 344 , 788–789 (2014).

Gilbert, D. T., King, G., Pettigrew, S. & Wilson, T. D. Comment on "Estimating the reproducibility of psychological science". Science 351 , 1037 (2016).

Anderson, C. J. et al. Response to comment on "Estimating the reproducibility of psychological science". Science 351 , 1037 (2016).

Ioannidis, J. P. A. Why most discovered true associations are inflated. Epidemiology 19 , 640–648 (2008).

PubMed   Google Scholar  

Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22 , 1359–1366 (2011).

Etz, A. & Vandekerckhove, J. A Bayesian perspective on the Reproducibility Project: Psychology. PLoS One 11 , e0149794 (2016).

Gelman, A. & Stern, H. The difference between “significant” and “not significant” is not itself statistically significant. Am. Stat. 60 , 328–331 (2006).

Cumming, G. Replication and P intervals: P values predict the future only vaguely, but confidence intervals do much better. Psychol. Sci. 3 , 286–300 (2008).

Verhagen, J. & Wagenmakers, E.-J. Bayesian tests to quantify the result of a replication attempt. J. Exp. Psychol. Gen. 143 , 1457–1475 (2014).

Simonsohn, U. Small telescopes: detectability and the evaluation of replication results. Psychol. Sci. 26 , 559–569 (2015).

Patil, P., Peng, R. D. & Leek, J. T. What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 , 539–544 (2016).

Wagenmakers, E.-J. et al. Bayesian inference for psychology. Part II: example applications with JASP. Psychon. Bull. Rev. 25 , 58–76 (2017).

PubMed Central   Google Scholar  

Lee, M. D. & Wagenmakers, E.-J. Bayesian Cognitive Modeling: A Practical Course (Cambridge Univ. Press, Cambridge, 2013).

Dreber, A. et al. Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl Acad. Sci. USA 112 , 15343–15347 (2015).

Benjamin, D. et al. Redefine statistical significance. Nat. Hum. Behav. 2 , 6–10 (2018).

Jeffreys, H. Theory of Probability (Oxford Univ. Press, Oxford, 1961).

Kass, R. E. & Raftery, A. E. Bayes factors. J. Am. Stat. Assoc. 90 , 773–795 (1995).

Arrow, K. J. et al. The promise of prediction markets. Science 320 , 877–878 (2008).

Nosek, B. A., Ebersole, C. R., DeHaven, A. & Mellor, D. M. The preregistration revolution. Proc. Natl Acad. Sci. USA 115 , 2600–2606 (2018).

Nosek, B. A. et al. Promoting an open research culture: author guidelines for journals could help to promote transparency, openness, and reproducibility. Science 348 , 1422–1425 (2015).

Download references

Acknowledgements

Neither Nature Human Behaviour nor the publisher had any involvement with the conduct of this study prior to its submission to the journal. For financial support we thank: the Austrian Science Fund FWF (SFB F63, START-grant Y617-G11), the Austrian National Bank (grant OeNB 14953), the Behavioral and Neuroeconomics Discovery Fund (C.F.C.), the Jan Wallander and Tom Hedelius Foundation (P2015-0001:1 and P2013-0156:1), the Knut and Alice Wallenberg Foundation (Wallenberg Academy Fellows grant to A.D.), the Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1), the Netherlands Organisation for Scientific Research (Vici grant 016.Vici.170.083 to E.-J.W.), the Sloan Foundation (G-2015-13929) and the Singapore National Research Foundation’s Returning Singaporean Scientists Scheme (grant NRF-RSS2014-001 to T.-H.H.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. We thank the following people for assistance with the experiments and analyses: D. van den Bergh, P.-C. Bindra, J. van Doorn, C. Huber, A. Ly, M. Marsman and J. Zambre.

Author information

These authors contributed equally: Colin F. Camerer, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A. Nosek, Thomas Pfeiffer.

Authors and Affiliations

California Institute of Technology, Pasadena, CA, USA

Colin F. Camerer

Department of Economics, Stockholm School of Economics, Stockholm, Sweden

Anna Dreber, Magnus Johannesson, Adam Altmejd, Emma Heikensten & Siri Isaksson

Department of Banking and Finance, University of Innsbruck, Innsbruck, Austria

Felix Holzmeister, Jürgen Huber, Michael Kirchler & Julia Rose

NUS Business School, National University of Singapore, Singapore, Singapore

Teck-Hua Ho

Centre for Finance, Department of Economics, University of Göteborg, Göteborg, Sweden

Michael Kirchler

The Wharton School, University of Pennsylvania, Philadelphia, PA, USA

Gideon Nave & Dylan Manfredi

Department of Psychology, University of Virginia, Charlottesville, VA, USA

Brian A. Nosek, Nick Buttrick & Anup Gampa

Center for Open Science, Charlottesville, VA, USA

Brian A. Nosek, Nick Buttrick, Anup Gampa & Lily Hummer

New Zealand Institute for Advanced Study, Auckland, New Zealand

Thomas Pfeiffer

Office of the Senior Deputy President and Provost, National University of Singapore, Singapore, Singapore

Taizan Chan

John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA

Yiling Chen

Spotify Sweden AB, Stockholm, Sweden

Eskil Forsell

Department of Economics, LMU Munich, Munich, Germany

Taisuke Imai

Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands

Eric-Jan Wagenmakers

School of Management, Harbin Institute of Technology, Harbin, China

You can also search for this author in PubMed   Google Scholar

Contributions

C.F.C., A.D., F.H., J.H., T.-H.H., M.J., M.K., G.N., B.A.N. and T.P. designed the research. C.F.C., A.D., F.H., T.-H.H., J.H., M.J., M.K., D.M., G.N., B.A.N., T.P. and E.-J.W. wrote the paper. T.C., A.D., E.F., F.H., T.-H.H., M.J., T.P. and Y.C. helped to design the prediction market part. F.H. and E.-J.W. analysed the data. A.A., N.B., A.G., E.H., F.H., L.H., T.I., S.I., D.M., J.R. and H.W. carried out the replications (including re-estimating the original estimate with the replication data). All authors approved the final manuscript.

Corresponding author

Correspondence to Brian A. Nosek .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

Supplementary Methods, Supplementary References, Supplementary Tables 1–7 and Supplementary Figures 1–9

Reporting Summary

Rights and permissions.

Reprints and permissions

About this article

Cite this article.

Camerer, C.F., Dreber, A., Holzmeister, F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav 2 , 637–644 (2018). https://doi.org/10.1038/s41562-018-0399-z

Download citation

Received : 06 March 2018

Accepted : 06 July 2018

Published : 27 August 2018

Issue Date : September 2018

DOI : https://doi.org/10.1038/s41562-018-0399-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Evidence of direct and indirect reciprocity in network-structured economic games.

• Daniel Redhead
• Matthew Gervais
• Cody T. Ross

Communications Psychology (2024)

Realizing the full potential of behavioural science for climate change mitigation

• Kristian S. Nielsen
• Viktoria Cologna
• Kimberly S. Wolske

Nature Climate Change (2024)

Trust but verify

Nature Materials (2024)

Optimally generate policy-based evidence before scaling

• John A. List

Nature (2024)

An evaluation of the replicability of analyses using synthetic health data

• Khaled El Emam
• Lucy Mosquera
• Alaa El-Hussuna

Scientific Reports (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

The replication crisis, the rise of new research practices and what it means for experimental economics

• Original Paper
• Published: 07 October 2021
• Volume 7 , pages 210–225, ( 2021 )

Cite this article

• Lionel Page   ORCID: orcid.org/0000-0002-6020-9733 1 ,
• Charles N. Noussair 2 &
• Robert Slonim 1  

1226 Accesses

14 Citations

13 Altmetric

Explore all metrics

The replication crisis across several disciplines raises challenges for behavioural sciences in general. In this report, we review the lessons for experimental economists of these developments. We present the new research methods and practices which are being proposed to improve the replicability of scientific studies. We discuss how these methods and practices can have a positive impact in experimental economics and the extent to which they should be encouraged.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Origins of Experimental Economics

Large-scale Social Experiments in Experimental Ethics

High replicability of newly discovered social-behavioural findings is achievable

Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016). Star wars: the empirics strike back. American Economic Journal: Applied Economics, 8 (1), 1–32.

Google Scholar  

Brodeur, A., Lé, M., Sangnier, M., Zylberberg, Y., (2013). Star wars: the empirics strike back. IZA Discussion Paper No. 7268.

Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., & Heikensten, E. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351 (6280), 1433–1436.

Article   Google Scholar  

Christensen, G., & Miguel, E. (2018). Transparency, reproducibility, and the credibility of economics research. Journal of Economic Literature, 56 (3), 920–980.

Coffman, L. C., & Niederle, M. (2015). Pre-analysis plans have limited upside, especially where replications are feasible. Journal of Economic Perspectives, 29 (3), 81–98.

Dufwenberg, M., Martinsson, P. (2014). Keeping researchers honest: The case for sealed-envelopesubmissions. IGIER (Innocenzo Gasparini Institute for Economic Research), 533.

Duyx, B., Urlings, M. J., Swaen, G. M., Bouter, L. M., & Zeegers, M. P. (2017). Scientific citations favor positive results: a systematic review and meta-analysis. Journal of Clinical Epidemiology, 88 , 92–101.

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2 (8), e124.

Kasy, M. (2021). Of forking paths and tied hands: Selective publication of findings, and what economists should do about it. Journal of Economic Perspectives, 35 (3), 175–192.

Maniadis, Z., Tufano, F., & List, J. A. (2015). How to make experimental economics research more reproducible: Lessons from other disciplines and a new proposal. Replication in experimental economics. Emerald Group Publishing Limited.

Olken, B. A. (2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29 (3), 61–80.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2018). False-positive citations. Perspectives on Psychological Science, 13 (2), 255–259.

Download references

Acknowledgements

This report benefited from comments, feedback and suggestions from a large number of ESA members. Among those, we would like to thank particularly Edward Cartwright, Noemi Peter, Alex Roomets, Egon Tripodi, Irenaeus Wolff and the JESA editors Maria Bigoni and Dirk Engelmann.

Author information

Authors and affiliations.

University of Technology Sydney, Sydney, NSW, Australia

Lionel Page & Robert Slonim

Department of Economics, University of Arizona, Tucson, AZ, USA

Charles N. Noussair

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Lionel Page .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Page, L., Noussair, C.N. & Slonim, R. The replication crisis, the rise of new research practices and what it means for experimental economics. J Econ Sci Assoc 7 , 210–225 (2021). https://doi.org/10.1007/s40881-021-00107-7

Download citation

Received : 20 August 2021

Revised : 21 August 2021

Accepted : 24 August 2021

Published : 07 October 2021

Issue Date : December 2021

DOI : https://doi.org/10.1007/s40881-021-00107-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Replication crisis
• Pre-registration
• Registered report
• Replication

JEL Classifications

• Find a journal
• Publish with us
• Track your research

Economic Science Association Repository for Replication Packages

In 2021 the Economic Science Association implemented a new Data and Replication Policy . The policy requires authors of manuscripts published in the ESA journals ( Experimental Economics and Journal of the Economic Science Association ) to publish, in a trusted online repository, any documentation necessary to reproduce or replicate their study.

On this webpage, we provide a log of the replication packages published since the policy was implemented. In addition, we aim to log any replications of these articles that we are aware of.

Examples of repositories used by authors of papers in Experimental Economics and JESA include the Harvard Dataverse , OSF , Zenodo , and OpenICPSR .

Useful links:

• Current version of the ESA Data and Replication Policy
• A curated list of trusted repositories

Index of replication materials

• A-Z Publications

Annual Review of Economics

Volume 14, 2022, review article, experimental economics: past and future.

• Guillaume R. Fréchette 1 , Kim Sarnoff 2 , and Leeat Yariv 2,3,4
• View Affiliations Hide Affiliations Affiliations: 1 Department of Economics, New York University, New York, NY, USA; email: [email protected] 2 Department of Economics, Princeton University, Princeton, New Jersey, USA; email: [email protected] [email protected] 3 Centre for Economic Policy Research, London, United Kingdom 4 National Bureau of Economic Research, Cambridge, Massachusetts, USA
• Vol. 14:777-794 (Volume publication date August 2022) https://doi.org/10.1146/annurev-economics-081621-124424
• Copyright © 2022 by Annual Reviews. All rights reserved

Over the past several decades, lab experiments have offered economists a rich source of evidence on incentivized behavior. In this article, we use detailed data on experimental papers to describe recent trends in the literature. We also discuss various experimentation platforms and new approaches to the design and analysis of the data they generate.

Article metrics loading...

Full text loading...

Literature Cited

• Agranov M , Tergiman C. 2014 . Communication in multilateral bargaining. J. Public Econ. 118 : 75– 85 [Google Scholar]
• Agranov M , Yariv L. 2018 . Collusion through communication in auctions. Games Econ. Behav. 107 : 93– 108 [Google Scholar]
• Anauati MV , Galiani S , Gálvez RH. 2020 . Differences in citation patterns across journal tiers: the case of economics. Econ. Inq. 58 : 3 1217– 32 [Google Scholar]
• Anderson LR , Holt CA. 1997 . Information cascades in the laboratory. Am. Econ. Rev. 87 : 5 847– 62 [Google Scholar]
• Arechar AA , Kraft-Todd GT , Rand DG. 2017 . Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk. J. Econ. Sci. Assoc. 3 : 1 1– 11 [Google Scholar]
• Baker M. 2016 . 1,500 scientists lift the lid on reproducibility. Nat. News 553 : 7604 452 – 54 [Google Scholar]
• Baranski A , Morton R. 2021 . The determinants of multilateral bargaining: a comprehensive analysis of Baron and Ferejohn majoritarian bargaining experiments. Exp. Econ. 2021 : https://doi.org/10.1007/s10683-021-09734-7 [Crossref] [Google Scholar]
• Baron DP , Ferejohn JA. 1989 . Bargaining in legislatures. Am. Political Sci. Rev. 83 : 4 1181– 206 [Google Scholar]
• Benjamin DJ , Berger JO , Johannesson M , Nosek BA , Wagenmakers EJ et al. 2018 . Redefine statistical significance. Nat. Hum. Behav. 2 : 1 6– 10 [Google Scholar]
• Berinsky AJ , Huber GA , Lenz GS. 2012 . Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Anal . 20 : 3 351– 68 [Google Scholar]
• Berinsky AJ , Margolis MF , Sances MW. 2014 . Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. Am. J. Political Sci. 58 : 3 739– 53 [Google Scholar]
• Bradfield AJ , Kagel JH. 2015 . Legislative bargaining with teams. Games Econ. Behav. 93 : 117– 27 [Google Scholar]
• Camerer CF , Dreber A , Forsell E , Ho TH , Huber J et al. 2016 . Evaluating replicability of laboratory experiments in economics. Science 351 : 6280 1433– 36 [Google Scholar]
• Charness G , Fréchette GR , Kagel JH. 2004 . How robust is laboratory gift exchange?. Exp. Econ. 7 : 2 189– 205 [Google Scholar]
• Chen Y , Plott CR. 1996 . The Groves-Ledyard mechanism: an experimental study of institutional design. J. Public Econ. 59 : 3 335– 64 [Google Scholar]
• Christensen G , Miguel E 2018 . Transparency, reproducibility, and the credibility of economics research. J. Econ. Lit. 56 : 3 920– 80 [Google Scholar]
• Coffman LC , Niederle M. 2015 . Pre-analysis plans have limited upside, especially where replications are feasible. J. Econ. Perspect. 29 : 3 81– 98 [Google Scholar]
• Cooper DJ , Dutcher EG. 2011 . The dynamics of responder behavior in ultimatum games: a meta-study. Exp. Econ. 14 : 4 519– 46 [Google Scholar]
• Dal Bó P , Fréchette GR. 2018 . On the determinants of cooperation in infinitely repeated games: a survey. J. Econ. Lit. 56 : 1 60– 114 [Google Scholar]
• Dal Bó P , Fréchette GR , Kim J 2021 . The determinants of efficient behavior in coordination games Work. Pap., Brown Univ. Providence, RI: [Google Scholar]
• Dreber A , Johannesson M. 2019 . Statistical significance and the replication crisis in the social sciences. Oxford Research Encyclopedia of Economics and Finance Oxford, UK: Oxford Univ. Press https://doi.org/10.1093/acrefore/9780190625979.013.461 [Crossref] [Google Scholar]
• Ellison G. 2002 . The slowdown of the economics publishing process. J. Political Econ. 110 : 5 947– 93 [Google Scholar]
• Ellison G. 2013 . How does the market use citation data? The Hirsch index in economics. Am. Econ. J. Appl. Econ. 5 : 3 63– 90 [Google Scholar]
• Embrey M , Fréchette GR , Yuksel S. 2018 . Cooperation in the finitely repeated prisoner's dilemma. Q. J. Econ. 133 : 1 509– 51 [Google Scholar]
• Engel C. 2011 . Dictator games: a meta study. Exp. Econ. 14 : 4 583– 610 [Google Scholar]
• Fehr E , Kirchsteiger G , Riedl A. 1993 . Does fairness prevent market clearing? An experimental investigation. Q. J. Econ. 108 : 2 437– 59 [Google Scholar]
• Franklin A , Allan F 1990 . Experiment, Right or Wrong Cambridge, UK: Cambridge Univ. Press [Google Scholar]
• Fréchette GR 2015 . Laboratory experiments: professionals versus students. Handbook of Experimental Economic Methodology A Fréchette, A Schotter 360– 90 Oxford, UK: Oxford Univ. Press [Google Scholar]
• Fréchette GR. 2016 . Experimental economics across subject populations. The Handbook of Experimental Economics , Vol. 2 JH Kagel, AE Roth 435– 80 Princeton, NJ: Princeton Univ. Press [Google Scholar]
• Fréchette GR , Kagel JH , Lehrer SF. 2003 . Bargaining in legislatures: an experimental investigation of open versus closed amendment rules. Am. Political Sci. Rev. 97 : 2 221– 32 [Google Scholar]
• Fréchette GR , Kagel JH , Morelli M. 2005a . Behavioral identification in coalitional bargaining: an experimental analysis of demand bargaining and alternating offers. Econometrica 73 : 6 1893– 937 [Google Scholar]
• Fréchette GR , Kagel JH , Morelli M. 2005b . Nominal bargaining power, selection protocol, and discounting in legislative bargaining. J. Public Econ. 89 : 8 1497– 517 [Google Scholar]
• Fréchette GR , Vespa E. 2017 . The determinants of voting in multilateral bargaining games. J. Econ. Sci. Assoc. 3 : 1 26– 43 [Google Scholar]
• Fudenberg D , Peysakhovich A. 2016 . Recency, records, and recaps: learning and nonequilibrium behavior in a simple decision problem. Proceedings of the Fifteenth ACM Conference on Economics and Computation 971– 86 New York: ACM [Google Scholar]
• Gillen B , Snowberg E , Yariv L. 2019 . Experimenting with measurement error: techniques with applications to the Caltech cohort study. J. Political Econ. 127 : 4 1826– 63 [Google Scholar]
• Goeree JK , Yariv L. 2011 . An experimental study of collective deliberation. Econometrica 79 : 3 893– 921 [Google Scholar]
• Goodman JK , Cryder CE , Cheema A. 2013 . Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples. J. Behav. Decis. Mak. 26 : 3 213– 24 [Google Scholar]
• Guarnaschelli S , McKelvey RD , Palfrey TR. 2000 . An experimental study of jury decision rules. Am. Political Sci. Rev. 94 : 2 407– 23 [Google Scholar]
• Gupta N , Rigotti L , Wilson AJ. 2021 . The experimenters' dilemma: inferential preferences over populations. arXiv:2107.05064 [econ.GN]
• Hauser D , Paolacci G , Chandler J 2019 . Common concerns with MTurk as a participant pool: evidence and solutions. Handbook of Research Methods in Consumer Psychology FR Kardes, PM Herr, N Schwarz 319– 37 New York: Routledge [Google Scholar]
• Healy PJ. 2006 . Learning dynamics for mechanism design: an experimental comparison of public goods mechanisms. J. Econ. Theory 129 : 1 114– 49 [Google Scholar]
• Heckman JJ , Moktan S. 2020 . Publishing and promotion in economics: the tyranny of the top five. J. Econ. Lit. 58 : 2 419– 70 [Google Scholar]
• Horton JJ , Rand DG , Zeckhauser RJ. 2011 . The online laboratory: conducting experiments in a real labor market. Exp. Econ. 14 : 3 399– 425 [Google Scholar]
• Kagel JH , Levin D. 1993 . Independent private value auctions: bidder behaviour in first-, second- and third-price auctions with varying numbers of bidders. Econ. J. 103 : 419 868– 79 [Google Scholar]
• Kagel JH , Roth AE , eds. 1995 . The Handbook of Experimental Economics Princeton, NJ: Princeton Univ. Press [Google Scholar]
• Kagel JH , Roth AE , eds. 2020 . The Handbook of Experimental Economics , Vol. 2 Princeton, NJ: Princeton Univ. Press [Google Scholar]
• Kagel JH , Sung H , Winter E 2010 . Veto power in committees: an experimental study. Exp. Econ. 13 : 2 167– 88 [Google Scholar]
• Kübler D , Weizsäcker G. 2004 . Limited depth of reasoning and failure of cascade formation in the laboratory. Rev. Econ. Stud. 71 : 2 425– 41 [Google Scholar]
• Lane T. 2016 . Discrimination in the laboratory: a meta-analysis of economics experiments. Eur. Econ. Rev. 90 : 375– 402 [Google Scholar]
• Lehmann S , Jackson AD , Lautrup BE. 2006 . Measures for measures. Nature 444 : 7122 1003– 4 [Google Scholar]
• Mengel F. 2018 . Risk and temptation: a meta-study on prisoner's dilemma games. Econ. J. 128 : 616 3182– 209 [Google Scholar]
• Nikiforakis N , Slonim R. 2019 . Editors' preface: trends in experimental economics (1975–2018). J. Econ. Sci. Assoc. 5 : 143– 48 [Google Scholar]
• Paolacci G , Chandler J , Ipeirotis PG. 2010 . Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. 5 : 5 411– 19 [Google Scholar]
• Plott CR , Smith VL , eds. 2008 . Handbook of Experimental Economics Results , Vol. 1 Amsterdam: Elsevier [Google Scholar]
• Reuben E , Li SX , Suetens S , Svorenčík A , Turocy T , Kotsidis V. 2021 . Trends in the publication of experimental economics articles . Work. Pap., New York Univ. New York: [Google Scholar]
• Samuelson W , Bazerman MH. 1984 . The winner's curse in bilateral negotiations Work. Pap. 1513-84 Sloan Sch. Manag., Mass. Inst. Technol. Cambridge: [Google Scholar]
• Simmons JP , Nelson LD , Simonsohn U. 2011 . False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22 : 11 1359– 66 [Google Scholar]
• Smith VL. 1982 . Microeconomic systems as an experimental science. Am. Econ. Rev. 72 : 5 923– 55 [Google Scholar]
• Smith VL. 1994 . Economics in the laboratory. J. Econ. Perspect. 8 : 1 113– 31 [Google Scholar]
• Snowberg E , Yariv L. 2021 . Testing the waters: behavior across participant pools. Am. Econ. Rev. 111 : 2 687– 719 [Google Scholar]

Data & Media loading...

• Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, power laws in economics and finance, the gravity model, the china shock: learning from labor-market adjustment to large changes in trade, microeconomics of technology adoption, financial literacy, financial education, and economic outcomes, gender and competition, corruption in developing countries, the economics of human development and social mobility, the roots of gender inequality in developing countries, weak instruments in instrumental variables regression: theory and practice.

To read this content please select one of the options below:

Please note you do not have access to teaching notes, four classic public goods experiments: a replication study.

Replication in Experimental Economics

ISBN : 978-1-78560-351-8 , eISBN : 978-1-78560-350-1

Publication date: 13 October 2015

This paper replicates four highly cited, classic lab experimental studies in the provision of public goods. The studies consider the impact of marginal per capita return and group size; framing (as donating to or taking from the public good); the role of confusion in the public goods game; and the effectiveness of peer punishment. Considerable attention has focused recently on the problem of publication bias, selective reporting, and the importance of research transparency in social sciences. Replication is at the core of any scientific process and replication studies offer an opportunity to reevaluate, confirm or falsify previous findings. This paper illustrates the value of replication in experimental economics. The experiments were conducted as class projects for a PhD course in experimental economics, and follow exact instructions from the original studies and current standard protocols for lab experiments in economics. Most results show the same pattern as the original studies, but in all cases with smaller treatment effects and lower statistical significance, sometimes falling below accepted levels of significance. In addition, we document a “Texas effect,” with subjects consistently exhibiting higher levels of contributions and lower free-riding than in the original studies. This research offers new evidence on the attenuation effect in replications, well documented in other disciplines and from which experimental economics is not immune. It also opens the discussion over the influence of unobserved heterogeneity in institutional environments and subject pools that can affect lab results.

• Replication
• Lab experiment
• Public goods
• Experimental economics
• Research transparency

Acknowledgements

Acknowledgments.

Thanks to the staff of the Center for Behavioral and Experimental Economic Science (CBEES), especially Wendy Mak Lee, who guided the students through the programming of the experiments; Eric McLester, who handled the IRB applications and communication; Sheheryar Banuri, who was TA for the course; and the many undergraduate research assistants who helped conduct the sessions: Ravi Hanumara, Tanushree Jhunjhunwala, Nick Lafferty, Nathan Eacret, Anthony Tantillo, Addison Ziegler. Zhengzheng Wang and Billur Aksoy assisted with manuscript preparation. Funding was provided by CBEES and the Negotiations Center at UT Dallas.

The course papers on which this paper is based are as follows:

Manca, Simone, Sabrina Ren, and Noha Sobi, “A Replication of: Group Size Effects in Public Good Provision: The Voluntary Contribution Mechanism.”

George, Justin, and Parneet Pahwa, “A Replication of Cooperation in Public-Goods Experiment: Kindness or Confusion.”

Holcomb, Alex, Victoria Obiadeze, and Shidi Wang, “Framing, It Really Does Matter.”

Alzahrani, Ahmed, Haley Harwell, Caitlin McKillop, and Zhengzheng Wang “A Replication of ‘Cooperation and Punishment in Public Goods Experiments.”

Eckel, C.C. , Harwell, H. and Castillo G., J.G. (2015), "Four Classic Public Goods Experiments: A Replication Study", Replication in Experimental Economics ( Research in Experimental Economics, Vol. 18 ), Emerald Group Publishing Limited, Leeds, pp. 13-40. https://doi.org/10.1108/S0193-230620150000018001

Emerald Group Publishing Limited

Copyright © 2015 Emerald Group Publishing Limited

All feedback is valuable

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

The replication crisis has engulfed economics

Professor, UNSW Sydney

Disclosure statement

Andreas Ortmann does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

UNSW Sydney provides funding as a member of The Conversation AU.

View all partners

A sense of crisis is developing in economics after two Federal Reserve economists came to the alarming conclusion that economics research is usually not replicable.

The economists took 67 empirical papers from 13 reputable academic journals. Without assistance from the original researchers they were only able to get the same result in a third of cases.

With the original researchers’ assistance, that percentage increased to about half, suggesting reporting practices and requirements are seriously deficient.

The replication crisis in psychology is well-documented. Science recently published a stunning report by the Open Science Collaboration. Almost 300 researchers were involved in trying to directly replicate the results of 100 papers published in 2008. This followed earlier exercises involving many labs (such as here, here and here. )

The researchers did not succeed in the clear majority of cases. On average they found the mean effect size to be only half of what was reported in the original studies. While the report has been questioned ( here and here, ) there is growing concern that a cornerstone of the scientific edifice is in serious need of renovation.

What’s the problem?

Researchers are too often granted inappropriate degrees of freedom , and some are just fraudulent . But that said, some of these distressing replication results are because good science is messy. It involves hard work and reasonable people can reasonably disagree on the various calls that have to be made.

A good illustration is this just-published study by Raphael Silberzahn and Eric Uhlmann . The researchers engaged in methodological debates with well-known data sleuth Uri Simohnson .

Simohnson questioned the results of an earlier study from the pair that suggested noble-sounding German names could boost careers. Re-running the analysis with a better analytical approach, Simonsohn did not confirm the effect. Silberzahn and Uhlmann eventually conceded the point in a joint paper with Simonsohn.

In their new study, the researchers provided a data set and asked more than two dozen teams of researchers to contribute. They sought to determine, based on the data set, whether skin colour of soccer players from four major leagues (England, France, Germany, and Spain) influenced how often they were given a red card.

Somewhat shockingly, the answers were rather diverse. Of the 29 teams, 20 found a statistically significant correlation with the median, suggesting dark-skinned players were 1.3 times more likely than light-skinned players to be sent off.

But the researchers reported:

“Findings varied enormously, from a slight (and non-significant) tendency for referees to give more red cards to light-skinned players to a strong trend of giving more red cards to dark-skinned players.”

Interestingly, this diversity of results survived even after the researchers debated the methodological approach.

The upshot is that even under the best of circumstances – one data set, what seems like a straightforward question to answer, and an exchange of ideas on the best method – arriving at consensus can be extraordinarily difficult. And it surely becomes even more difficult with multiple data sets and many teams.

Further scrutiny

That, of course, is hardly news to most social scientists, who largely accept that any single study is worth only so much. This is why replication efforts and meta-analyses are as important as the recent focus on publication bias and underpowered studies . There is tantalising evidence that many experimental economics studies are severely under-powered (although the evidence so far has been established only for a very simple class of games) .

It will be interesting to see the result of a current collaborative effort by economists to replicate eighteen laboratory economics studies from 2011 to 2014.

It is not just the social sciences that are in the grip of replication crises. The extent and consequences of p-hacking, and publication biases (studies that report no effect not being published) in science, are well-documented and have been known for a while .

So, where to from here? With a number of journals (including the Journal of the Economic Science Association , Experimental Economics , Journal of Experimental Social Psychology , Journal of Personality and Social Psychology , Psychological Science , Perspectives on Psychological Science ) opening their doors to replication in various guises, we can expect more results to seemingly discredit the social sciences.

Hopefully in the long run it will up the ante on what it takes for a study to be reliable. Replication studies can inflict considerable damage on individuals’ productivity and reputation . There’s a need for minimal reporting standards and acceptable replication etiquette to be clarified, such as whether original authors have to be invited or consulted. Journals should become more serious about their data set collection efforts, when not prevented by confidentiality.

• Replication

University Relations Manager

2024 Vice-Chancellor's Research Fellowships

Head of Research Computing & Data Solutions

Community member RANZCO Education Committee (Volunteer)

Director of STEM

• Business & Money

Sorry, there was a problem.

Download the free Kindle app and start reading Kindle books instantly on your smartphone, tablet, or computer - no Kindle device required .

Read instantly on your browser with Kindle for Web.

Using your mobile phone camera - scan the code below and download the Kindle app.

Image Unavailable

• To view this video download Flash Player

Replication in Experimental Economics (Research in Experimental Economics, 18) Hardcover – October 20, 2015

• Print length 240 pages
• Language English
• Publisher Emerald Publishing Limited
• Publication date October 20, 2015
• Dimensions 5.98 x 0.67 x 9.02 inches
• ISBN-10 1785603515
• ISBN-13 978-1785603518
• See all details

Editorial Reviews

About the author, product details.

• Publisher ‏ : ‎ Emerald Publishing Limited (October 20, 2015)
• Language ‏ : ‎ English
• Hardcover ‏ : ‎ 240 pages
• ISBN-10 ‏ : ‎ 1785603515
• ISBN-13 ‏ : ‎ 978-1785603518
• Item Weight ‏ : ‎ 1.05 pounds
• Dimensions ‏ : ‎ 5.98 x 0.67 x 9.02 inches

Customer reviews

• 5 star 4 star 3 star 2 star 1 star 5 star 0% 0% 0% 0% 0% 0%
• 5 star 4 star 3 star 2 star 1 star 4 star 0% 0% 0% 0% 0% 0%
• 5 star 4 star 3 star 2 star 1 star 3 star 0% 0% 0% 0% 0% 0%
• 5 star 4 star 3 star 2 star 1 star 2 star 0% 0% 0% 0% 0% 0%
• 5 star 4 star 3 star 2 star 1 star 1 star 0% 0% 0% 0% 0% 0%

Customer Reviews, including Product Star Ratings help customers to learn more about the product and decide whether it is the right product for them.

To calculate the overall star rating and percentage breakdown by star, we don’t use a simple average. Instead, our system considers things like how recent a review is and if the reviewer bought the item on Amazon. It also analyzed reviews to verify trustworthiness.

No customer reviews

• About Amazon
• Investor Relations
• Amazon Devices
• Amazon Science
• Sell products on Amazon
• Sell on Amazon Business
• Sell apps on Amazon
• Become an Affiliate
• Advertise Your Products
• Self-Publish with Us
• Host an Amazon Hub
• › See More Make Money with Us
• Amazon Business Card
• Shop with Points
• Reload Your Balance
• Amazon Currency Converter
• Amazon and COVID-19
• Your Account
• Your Orders
• Shipping Rates & Policies
• Returns & Replacements
• Manage Your Content and Devices

• Conditions of Use
• Privacy Notice
• Consumer Health Data Privacy Disclosure
• Your Ads Privacy Choices

Our systems are now restored following recent technical disruption, and we’re working hard to catch up on publishing. We apologise for the inconvenience caused. Find out more: https://www.cambridge.org/universitypress/about-us/news-and-blogs/cambridge-university-press-publishing-update-following-technical-disruption

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• < Back to search results
• Experimental Economics

Experimental Economics

• Submit your article

This journal utilises an Online Peer Review Service (OPRS) for submissions. By clicking "Continue" you will be taken to our partner site https://www.editorialmanager.com/eec/default.aspx . Please be aware that your Cambridge account is not valid for this OPRS and registration is required. We strongly advise you to read all "Author instructions" in the "Journal information" area prior to submitting.

• Information
• Journal home
• Journal information
• Latest issue
• Open access

We’re delighted to announce that all articles accepted for Volume 28 (2025) of   Experimental Economics will be ‘open access’; published with a Creative Commons licence and freely available to read online (see the journal’s Open Access Options  page for available licence options). We have an OA option for every author: The costs of open access publication will be covered through agreements between the publisher and the author’s institution , payment of APCs from grant or other funds, or else waived entirely, ensuring every author can publish and enjoy the benefits of OA. 

Please see the journal's Open Access Options  page for instructions on how to request an APC waiver.

See this FAQ for more information.

• You have access: full Access: Full

• Forthcoming
• ISSN: 1386-4157 (Print) , 1573-6938 (Online)
• Editors: Friederike Mengel Department of Economics, University of Essex, UK , Andreas Ortmann UNSW Business School, University of South Wales, Sydney, Australia , Ragan Petrie Department of Economics, Texas A&M University, USA , Arno Riedl Department of Microeconomics and Public Economics, Maastricht University, The Netherlands , and Marta Serra Garcia Rady School of Management, University of California, San Diego, USA
• Editorial board