Innovation Contests: How Economic Theory Helps in Solving Technological Problems

Innovation Contests: How Economic Theory
Helps in Solving Technological Problems
Karim R. Lakhani | [email protected] | @klakhani | Harvard University
Crowd Innovation Lab | NASA Tournament Lab
Contests are a Historically Important Institution for Driving
Innovation....
The Duomo - Florence
1418 - Up to 2,000 Florins
The Longitude Prize
1714 - Up to £20,000
Invention of Food Canning
1800 - Up to 12,000 Francs
....Currently Popular as Well
Ansari X-Prize – Space Travel
1996 – $10,000,000
Netflix Prize - Movie Rec.
2006 - 2009
Over 5000 Teams - $1M
Local Motors – Car Design
2008 – Over 35000 Submits
“Crowdsourcing” & Contests-based
Platforms Proliferating in the Economy:
Hundreds of Thousands of Participants
“Crowds” Can Be Organized as Contests or Communities
(Boudreau & Lakhani 2009, 2013, 2016, King & Lakhani 2013)
“Crowds” Can Be Organized as Contests or Communities
(Boudreau & Lakhani 2009, 2013, 2016, King & Lakhani 2013)
Contests/Competition
Innovation problem requires
diversity of approaches and
broad experimentation
Sponsor not sure what
combination of skills and
approaches might be useful in
solution generation
Clear rules for participation
and winning
Communities/Collaboration
Innovation problem requires
cumulative knowledge building
and aggregation of diverse
inputs
Contributions range from mix
& match to co-production with
modular tasks and functions
Informal, norms-based
governance
America COMPETES Act 2010
Proposal for Prizes at NSF (2006)
Legislative and Policy Interest in
Encouraging Prize-based Contests to
Elicit Innovation
Well Established Theoretical
Foundations for Contest Design
Empirical Evidence Lags Theory
“Owing to the limited experience
with innovation prizes, relatively
little is known about how they
work in practice or how effective
they may be as compared with,
for example, R&D grants and
contracts, or tax incentives.”
Similar concerns by scholars
(Brunt, Lerner and Nicholas 2011;
Murray, Stern, Campbell and
MacCormack 2012; Williams
2012)
Mission for Crowd Innovation Lab
Lab partners: NASA, Harvard Medical School & TopCoder + additional
partners
Over 650 contests completed - for a variety of software applications.
Executed 17 computational algorithm development challenges (14 exceed
benchmarks): Computational biology, space sciences and advanced analytics
Managed four large-scale HMS grant funding processes ($25,000 to
$800,000)
Dual objectives - solve innovation problems & drive causal inference
Challenge 1: Data Explosion - From
15
18
Petabytes (10 ) to Exabytes (10 )
Challenge 2:
Labor Market Shortage: 1.8 Million
“Missing” Data Scientists
Intense Competition
for Data Science Talent
Challenge 3:
Rapid Change in Approaches to Solve
Data Challenges
Contest to Solve Complex Problem
for NASA
Contest to Solve Complex Problem
for NASA
Broad Engagement
(459 Competitors & 2000 Code Submissions)
Broad Engagement (459 Competitors & 2000
Code Submissions) & High Performance
Solar Power Output W/h
120000
90000
60000
30000
0
Broad Engagement (459 Competitors & 2000
Code Submissions) & High Performance
Solar Power Output W/h
120000
90000
60000
30000
0
Internal NASA Solution
Planetary Defense Top Priority for NASA:
556 Asteroids Have Penetrated the Earth’s
Atmosphere Over the Last 20 Years
Contest to Improve Asteroid Detection
Algorithm (Catalina Sky Survey - Arizona)
47 Competitors - 256 Code Submissions - 4 Weeks
High Performance: 10% Increase in
Detectability | Order of Magnitude
Reduction in False Positives
High Performance: 10% Increase in
Detectability | Order of Magnitude
Reduction in False Positives
InnoCen6ve&Pilot:&
Challenge&Data&and&Sta6s6cs&
Challenge(Title(
Ctr(
Posted(
Deadline(
Proj(
Rms(
Sub(
Award(
Date(
Award(
Amount(
Improved(Barrier(Layers(…(
Keeping(Food(Fresh(in(Space(
JSC(E(
12/18/2009( 2/28/2010( 174(
SLSD(
22(
5/7/2010(
$11,000((
Mechanism&for&a&Compact&
Aerobic&Resis6ve&Exercise&
Device(
JSC&C&
12/18/2009& 2/28/2010&
SLSD&
564&
95&
5/14/2010&
$20,000&&
DataEDriven(ForecasQng(of(
Solar(Events(
JSC(E(
12/22/2009( 3/22/2010( 579(
SLSD(
11(
5/13/2010(
$30,000((
Coordina6on&of&Sensor&
Swarms&for&Extraterrestrial&
Research&&
LRC&
2/27/2010& 4/26/2010&
423&
37&
6/4/2010& $18,000&(3)&
GRC(
5/17/2010( 7/27/2010( 365(
56(
in(progress( $15,000((3)(
JSC&C&
SLSD&
5/27/2010& 7/27/2010&
229&
18&
9/20/2010&
$10,000&&
5/27/2010( 7/27/2010( 598(
108(
9/21/2010(
$7,500((
Medical(Consumables(
Tracking(
Augmen6ng&the&Exercise&
Experience&
Simple(Microgravity(Laundry( JSC(E(
System(
EA(
Space&Life&Sciences&
Exploring&Space&|&Enhancing&Life&
22&
2900&Solvers&–&80&countries&
NASA&Glenn&
Research&Center&
NASA&Langley&
Research&Center&
NASA&Johnson&Space&
Center&SLSD&–&SA&
&
Over%2,900%Solvers%from%80%Countries%Par8cipated%
Space&Life&Sciences&
Exploring&Space&|&Enhancing&Life&
21&
Are Crowds Smarter than Harvard
Medical School?
Objective: Improve on NIH MegaBlast algorithm for
nucleotide sequence alignment for immunogenomics
Experiment: Generate and evaluate external solver
participation in development of gene-sequencing tools
applied to immunoglobulin and antibody genomics
Two week long competition - $2000 prize pot x 3 on
TopCoder.com
Contest Results Shows the
Discovery of Extreme Value
Outcomes Relatively Quickly
(Lakhani et al., 2013)
Contest Results Shows the
Discovery of Extreme Value
Outcomes Relatively Quickly
122 coders submitted 654
submissions
(Lakhani et al., 2013)
Contest Results Shows the
Discovery of Extreme Value
Outcomes Relatively Quickly
122 coders submitted 654
submissions
34 coders exceeded state of the
art by 102 - 105
(Lakhani et al., 2013)
Contest Results Shows the
Discovery of Extreme Value
Outcomes Relatively Quickly
122 coders submitted 654
submissions
34 coders exceeded state of the
art by 102 - 105
89 different approaches to solve
problem identified
(Lakhani et al., 2013)
Contest Results Shows the
Discovery of Extreme Value
Outcomes Relatively Quickly
122 coders submitted 654
submissions
34 coders exceeded state of the
art by 102 - 105
89 different approaches to solve
problem identified
Winners from Russia, France,
Egypt, Belgium & US
(Lakhani et al., 2013)
Contest Results Shows the
Discovery of Extreme Value
Outcomes Relatively Quickly
122 coders submitted 654
submissions
34 coders exceeded state of the
art by 102 - 105
89 different approaches to solve
problem identified
Winners from Russia, France,
Egypt, Belgium & US
Annotate 10 million sequences
in < 3 mins; Quarter billion
sequences in ~ 1 hour on laptop
(Lakhani et al., 2013)
Contest Results Shows the
Discovery of Extreme Value
Outcomes Relatively Quickly
122 coders submitted 654
submissions
34 coders exceeded state of the
art by 102 - 105
89 different approaches to solve
problem identified
Winners from Russia, France,
Egypt, Belgium & US
Annotate 10 million sequences
in < 3 mins; Quarter billion
sequences in ~ 1 hour on laptop
(Lakhani et al., 2013)
Antibody Sequence Clustering - Scripps Research Institute
($7500 - 10 Days - 40 People)
Antibody Sequence Clustering - Scripps Research Institute
($7500 - 10 Days - 40 People)
Scripps Solution: 100K sequences, 170GB RAM Server, 1.7 hrs
Antibody Sequence Clustering - Scripps Research Institute
($7500 - 10 Days - 40 People)
Scripps Solution: 100K sequences, 170GB RAM Server, 1.7 hrs
Contest Solution: 2.3M sequences, 1.1GB RAM, ~30s
Antibody Sequence Clustering - Scripps Research Institute
($7500 - 10 Days - 40 People)
Scripps Solution: 100K sequences, 170GB RAM Server, 1.7 hrs
Contest Solution: 2.3M sequences, 1.1GB RAM, ~30s
20X Capacity, 10,000X speed, 10X Memory Efficiency
Lab has Completed 8 Field Experiments
with Innovation Contests (Boudreau & Lakhani 2016)
Lab has Completed 8 Field Experiments
with Innovation Contests (Boudreau & Lakhani 2016)
Incentives
Prizes vs signals - NASA/TopCoder - Autonomous Robots
~ 1200 coders
Incentives for “internal” public goods - HMS/MGH-Idea
Competition ~ 350 employees
Contests versus tournaments - Scripps/TopCoder ~300
coders
Selection vs treatment effects in contests - NASA/
TopCoder - Space Medical Kit Development ~ 900 coders
Lab has Completed 8 Field Experiments
with Innovation Contests (Boudreau & Lakhani 2016)
Incentives
Prizes vs signals - NASA/TopCoder - Autonomous Robots
~ 1200 coders
Incentives for “internal” public goods - HMS/MGH-Idea
Competition ~ 350 employees
Contests versus tournaments - Scripps/TopCoder ~300
coders
Selection vs treatment effects in contests - NASA/
TopCoder - Space Medical Kit Development ~ 900 coders
Knowledge
Knowledge disclosure in contests & communities - HMS/
TopCoder- Computational Biology ~ 700 coders
Intellectual Distance and Scientific Evaluation - HMS Grant
Process - 150 Submissions/142 Evaluators
Lab has Completed 8 Field Experiments
with Innovation Contests (Boudreau & Lakhani 2016)
Incentives
Prizes vs signals - NASA/TopCoder - Autonomous Robots
~ 1200 coders
Incentives for “internal” public goods - HMS/MGH-Idea
Competition ~ 350 employees
Contests versus tournaments - Scripps/TopCoder ~300
coders
Selection vs treatment effects in contests - NASA/
TopCoder - Space Medical Kit Development ~ 900 coders
Knowledge
Knowledge disclosure in contests & communities - HMS/
TopCoder- Computational Biology ~ 700 coders
Intellectual Distance and Scientific Evaluation - HMS Grant
Process - 150 Submissions/142 Evaluators
Search
Search costs in finding collaborators - HMS-Advanced
Imaging Grant Program ~ 450 researchers
Self-organization in collaboration - NASA/TopCoderImaging/OCR in Documents ~ 900 coders
Innovation Contests Well Suited for High
Uncertainty Problems (Boudreau, Lacetera & Lakhani 2011)
Innovation Contests Well Suited for High
Uncertainty Problems (Boudreau, Lacetera & Lakhani 2011)
Key question in contest design is
about how many competitors
should enter?
Innovation Contests Well Suited for High
Uncertainty Problems (Boudreau, Lacetera & Lakhani 2011)
Key question in contest design is
about how many competitors
should enter?
Lots of entry means lower
probability of winning - less
incentives to work hard
Innovation Contests Well Suited for High
Uncertainty Problems (Boudreau, Lacetera & Lakhani 2011)
Key question in contest design is
about how many competitors
should enter?
Lots of entry means lower
probability of winning - less
incentives to work hard
Innovation Contests Well Suited for High
Uncertainty Problems (Boudreau, Lacetera & Lakhani 2011)
Key question in contest design is
about how many competitors
should enter?
Lots of entry means lower
probability of winning - less
incentives to work hard
But this could be offset by finding
an outlier response as more
people come on
Innovation Contests Well Suited for High
Uncertainty Problems (Boudreau, Lacetera & Lakhani 2011)
Key question in contest design is
about how many competitors
should enter?
Lots of entry means lower
probability of winning - less
incentives to work hard
But this could be offset by finding
an outlier response as more
people come on
Problem uncertainty moderates
outcomes
Innovation Contests Well Suited for High
Uncertainty Problems (Boudreau, Lacetera & Lakhani 2011)
Key question in contest design is
about how many competitors
should enter?
Lots of entry means lower
probability of winning - less
incentives to work hard
But this could be offset by finding
an outlier response as more
people come on
Problem uncertainty moderates
outcomes
Heterogenous Responses to Increased
Competition in Contests (Boudreau, Lakhani & Menietti 2016)
Heterogenous Responses to Increased
Competition in Contests (Boudreau, Lakhani & Menietti 2016)
If competitors are heterogenous in
skills then we should expect
differential responses to increased
competition (Moldovanu & Sela 2001,
2006)
Heterogenous Responses to Increased
Competition in Contests (Boudreau, Lakhani & Menietti 2016)
If competitors are heterogenous in
skills then we should expect
differential responses to increased
competition (Moldovanu & Sela 2001,
2006)
Heterogenous Responses to Increased
Competition in Contests (Boudreau, Lakhani & Menietti 2016)
If competitors are heterogenous in
skills then we should expect
differential responses to increased
competition (Moldovanu & Sela 2001,
2006)
Low skills — no impact; High skills —
rivalry driven increased performance;
Mid skills - incentive driven
decreased performance
Heterogenous Responses to Increased
Competition in Contests (Boudreau, Lakhani & Menietti 2016)
If competitors are heterogenous in
skills then we should expect
differential responses to increased
competition (Moldovanu & Sela 2001,
2006)
Low skills — no impact; High skills —
rivalry driven increased performance;
Mid skills - incentive driven
decreased performance
Heterogenous Responses to Increased
Competition in Contests (Boudreau, Lakhani & Menietti 2016)
If competitors are heterogenous in
skills then we should expect
differential responses to increased
competition (Moldovanu & Sela 2001,
2006)
Low skills — no impact; High skills —
rivalry driven increased performance;
Mid skills - incentive driven
decreased performance
Structural estimation recovers bid
values —> showing salience of
monetary and non-monetary
incentives
Unconventional Individuals Win in
Innovation Contests (Jeppesen & Lakhani 2010)
Study of 166 problems involving
over 12000 scientists from
InnoCentive
Focus on what predicts winners
What explains who creates a
winning solution?
o Technical Marginality: Increasing
distance between solver’s own field
of expertise and the problem field
o Social Marginality: Women
scientists, when they enter, more
likely to win
Key Insight: Contests Provide
Incentives and Enable Parallel Search
Probability Density
A Simple Statistical Explanation of
Contest Performance (with Michael Menietti)
0.06
0.05
0.04
0.03
0.02
0.01
0
-20
-10
0
10
20
30
40
“Value” of Innovation Outcomes
50
60
A (Very) Rudimentary Model (I)
Suppose quality of outcome produced is given by distribution F
If a single crowd member works the expected quality is
If n members engage in production then the best outcome is distributed as
Fn
If internal/non-crowd innovation process (counterfactual) is expected to
produce a solution of qth quantile of quality distribution.
We want a crowd solution at least as good as the expected alternative
version with 1− α probability
A (Very) Rudimentary Model (II)
Simple to calculate number of crowd solutions needed to meet quality
threshold
For Any Distribution F - We Can Calculate Number of Draws
Needed to Achieve a Quality Objective
The Quantile of Expected Value of n Draws from Several
Distributions
But Internal Experts May Still on
Average Be Smarter than the Crowd
Expected Value of Max Under Normal Distribution
What Motivates People to
Participate in Contests?
Extrinsic
Cash, Job Market Signals,
Community Prestige
Intrinsic
Fun, Enjoyment, Learning,
Autonomy, Taste
Prosocial
Community Belonging, Identity
Extrinsic
Cash, Job Market Signals,
Community Prestige
Intrinsic
Fun, Enjoyment, Learning,
Autonomy, Taste
Prosocial
Community Belonging, Identity
Extrinsic
Cash, Job Market Signals,
Community Prestige
Intrinsic
Fun, Enjoyment, Learning,
Autonomy, Taste
Prosocial
Community Belonging, Identity
Extrinsic
Cash, Job Market Signals,
Community Prestige
Intrinsic
Fun, Enjoyment, Learning,
Autonomy, Taste
Prosocial
Community Belonging, Identity
Extrinsic
Cash, Job Market Signals,
Community Prestige
Intrinsic
Fun, Enjoyment, Learning,
Autonomy, Taste
Prosocial
Community Belonging, Identity
Recall: Most People Lose in Contests
Interesting Organizational Economics Issues in
Employees Using External Contests
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Find the “Right” Workers
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Find the “Right” Workers
Incentivize Effort
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Pray for Performance
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Pray for Performance
Contest
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Contest
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Pray for Performance
Define the Problem
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Contest
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Pray for Performance
Define the Problem
Develop Criteria for Evaluation
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Contest
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Pray for Performance
Define the Problem
Develop Criteria for Evaluation
Set Prize
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Contest
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Pray for Performance
Define the Problem
Develop Criteria for Evaluation
Set Prize
Attract Solvers
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Contest
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Pray for Performance
Define the Problem
Develop Criteria for Evaluation
Set Prize
Attract Solvers
Test Solutions
Interesting Organizational Economics Issues in
Employees Using External Contests
Internal Development
Contest
Define the Problem
Find the “Right” Workers
Incentivize Effort
Monitor Effort
Motivate and Energize Workers
Redefine the Problem
Develop Criteria for Evaluation
Pray for Performance
Define the Problem
Develop Criteria for Evaluation
Set Prize
Attract Solvers
Test Solutions
Pay for Performance
Advancing Macro Research and Practice
via Contests on Improving DSGE
Collaboration between MFM, BFI, LFE & CIL to prototype application of contests
for Macro research and practice communities
Focus on improving DSGE macro as implemented within Dynare
Two objectives:
Significantly improve speed of computation
Generate/Explore/Develop alternative estimation approaches
Contest and problem statement design phase in progress
Launch ~ end of March 2016
First results ~ end of April 2016
Thanks!
[email protected] | @klakhani