A CUSTOMER-PREFERENCE UNCERTAINTY MODEL FOR DECISION-ANALYTIC CONCEPT SELECTION
by user
Comments
Transcript
A CUSTOMER-PREFERENCE UNCERTAINTY MODEL FOR DECISION-ANALYTIC CONCEPT SELECTION
Proceedings of the 4th Annual ISC Research Symposium ISCRS 2010 April 21, 2010, Rolla, Missouri A CUSTOMER-PREFERENCE UNCERTAINTY MODEL FOR DECISION-ANALYTIC CONCEPT SELECTION ABSTRACT Analysis of customer preferences is one of the most important tasks in new product development. How customers come to appreciate and decide to purchase a new product impacts market share and, therefore, the success of the new product. Unfortunately, when designers select a product concept early in the product development process, the market share of the new product is unknown. Conjoint analysis is a statistical methodology that has been used to estimate the market share of a product concept from customer survey data. Although conjoint analysis has been increasingly incorporated in design engineering as a tool to estimate market share of a new product design, it has not been fully employed to model market share uncertainty. This paper presents two approaches, which use conjoint analysis data to model market share uncertainty: bootstrap and binomial inference. Demonstration and comparison of the two approaches are presented using an illustrative example. 1. INTRODUCTION In product development, engineers select a product concept before they develop detailed designs and prototypes [1]. At the time of concept selection therefore, future market size, market share, competition, warranty cost, and product cost are uncertain. Uncertainties directly relevant to concept selection are modeled. These uncertainties include market share, warranty cost, and product cost. The present research addresses market share uncertainty modeling as a means to select a concept with the maximum expected utility of profit. Conjoint Analysis (CA), a method to measure consumer judgments, has been used in a variety of applications and has received considerable attention since the early 1970s. This method has most often been applied in the fields of applied psychology, decision theory, and economics [2]. Robinson [3] reports a multinational conjoint study of North Atlantic air travel involving airfare, discounts and travel restrictions. This study indicates that CA can accurately predict market shares. Benbenisty [4] published a conjoint study involving AT&T’s entry into the data terminal market. His simulator forecasted an 8% share for AT&T four years after launch and obtained an actual share of just under 8%. These developments have been discussed by Paul E. Green and V. Srinivasan [5]. Hildebrand [6] has used CA for market definition, which is instrumental for the assessment of market power and central to competition policy. Currently however CA generates only one value for the percentage of market share for a particular concept or concepts. To overcome this limitation, the first approach, we propose uses bootstrap [7], which is a sampling with a replacement method that permits calculation of sample statistics. Researchers have applied bootstrap to make statistical inferences in clustering analysis and in phylogenetic trees [813]. Felsenstien [8] and Efron et al. [9] incorporated k-by-p data matrix consisting of k species and p sites. They generated bootstrapped samples of data matrices by sampling columns with replacement. By applying bootstrap to the results obtained from CA, a continuous distribution is obtained, which improves results. Furthermore, the continuous distribution can be discretized to obtain probabilities by using the Extended Pearson-Tukey method [14]. The second approach, which we propose as an approximation to bootstrap, is binomial inference. Binomial inference may be explained using a coin-flipping analogy [15]. In coin flipping, if we observe H heads and T tails in H+T flips, an uncertainty of probability of head is modeled by a beta distribution with parameter (H, T). Applying a similar analogy, this work proposes that the uncertainty of market share of a concept selected by M customers out of N total customers is modeled by a beta (M, N–M) distribution. This paper is organized as follows: Section 2 briefly describes conjoint analysis, bootstrap, and a framework to model market share uncertainty by integrating the two. Section 3 demonstrates and compares the proposed two approaches (bootstrap and binomial inference) in an illustrative example. Section 4 concludes the paper with a discussion of future work. 2. METHODOLOGY 2.1. Conjoint analysis Conjoint analysis for estimating the market share of a new product concept involves the following steps: 1 Copyright © 2010 by ISC Concept definition: Identify product attributes and their levels that are important for customers to make purchasing decisions. Define new product concepts and competitors’ products as combinations of attributes and their levels. Product attributes may be identified by interviewing customers and translating the customer needs into product attributes. Attribute level identification: Benchmark existing products in the marketplace or forecast future customer needs for a new product to identify attribute levels included in conjoint analysis survey. Conjoint survey design: Determine which conjoint method will be used. That is, select the respondents’ concept evaluation method (rating/ranking or choice), completeness of profile to be used to describe concepts (full or partial profile), and decide whether or not to use a Bayesian approach (Hierarchical Bayes or non-Hierarchical Bayes). Create concept profiles to be displayed to respondents according to the chosen method. Market share estimation: Estimate the market share of the concepts by analyzing respondents’ concept evaluation results. 2.2. Bootstrap Bootstrap is a computer-based sampling-with-replacement method that has been used to obtain a confidence interval of an estimate as illustrated using a simple example in Fig. 1. Suppose we wish to estimate a population average from a randomly sampled data set {1, 2, 3, 4, 5, 6, 7}. We calculate a sample average 4 and use this as an estimate of population average. Because this is a point estimate of population average, no confidence interval of this estimate can be obtained from the initial data set alone; bootstrap samples of the same data size must be generated by sampling with replacements from the initial data set to obtain confidence intervals. For example, the first bootstrap sample may be {5, 1, 7, 1, 2, 4, 4}, the second Sample statistics Data (average) Sampling with replacements Initial sample : { 1 , 2 , 3 , 4 , 5 , 6 , 7 } → 4 Inference (95% confidence interval) Bootstrap samples 1 st : { 5,1,7,1,2,4,4 } → 3.4 2 nd : { 6,4,2,5,7,3,6 } → 4.7 3 rd : { 2,3,5,2,1,1,1 } → 2.1 200 th : { 4,6,4,7,3,4,2 } → 4.3 Distribution 1 4 7 bootstrap sample may be {6, 4, 2, 5, 7, 3, 6}, and so forth. Fig. 1 Bootstrap procedure In the bootstrap samples, the same data may appear more than once or do not appear at all due to the sampling-withreplacement procedure. Each of these bootstrap samples provides a sample statistic (i.e., average). If 200 bootstrap samples are generated, 200 sample averages will be created. These sample averages permit a construction of a confidence interval or a distribution of sample statistics. For example, the 5th percentile and the 95th percentile of these bootstrap averages provide a range of 95% confidence interval of the point estimate 4, and a histogram of bootstrap sample averages provides distribution of sample statistics, as illustrated in Fig. 1. 2.2. Bootstrap application to conjoint analysis In conjoint analysis, respondents are randomly selected from populations of customers as illustrated in Fig. 2. Although conjoint analysis yields a concept’s true market share if it is applied to the entire population of customers, surveying the entire population is not feasible. By asking randomly selected customers to evaluate product concepts and competitors’ products, a point estimate of the market share of a concept is obtained, as illustrated by the middle flow from left to right in Fig. 2. Using a point estimate for the analysis is equivalent to assigning a probability of one to the point estimate. Population Customer {1, 2, 3, …, N} Conjoint analysis Probability 1 "True" market share, S True 0.5 Random sampling Sample Customer {1, 2, 3, …, n} STrue Probability Conjoint analysis Point estimate of market share, s Estimate Bootstrap { 1st sample} Conjoint analysis { 2nd sample} Market share, s 2 { B-th sample} Market share, s B Market share, s 1 50% Market 100% share 1 0.5 s Estimate 50% Probability Market 100% share 1 0.5 50% Market 100% share Figure 2 Application of bootstrap to conjoint analysis In contrast, if bootstrap is applied to conjoint analysis, bootstrap samples are generated from an initial set of randomly sampled customers, as illustrated in the bottom flow in Fig. 2. In the bootstrapped samples, a customer may appear more than once or may not appear at all because of the sampling-withreplacement procedure. By applying conjoint analysis to the bootstrap samples, market share estimates are obtained from which a distribution of market share can be constructed as illustrated in Fig. 2. 3. Illustrative example This section illustrates and compares two market share uncertainty modeling approaches: the application of bootstrap and binomial inference to conjoint analysis. This example presents non-Bayesian full-profile conjoint analysis with customers’ evaluations in rating scales using automobile concept selection as an illustrative example; however, both approaches can be applied to any conjoint analysis approach. 3.1.Concepts and competition For an illustrative purpose, this example assumes that a firm wishes to estimate the future market share of a new automobile (N) that will compete with two competitor vehicles (C1 and C2). The concept of a new automobile is defined by its type and fuel efficiency. Type refers to its form and the maximum number of passengers that it can accommodate, and 2 Copyright © 2010 by ISC fuel efficiency is associated by an engine type (a gasoline engine or a hybrid engine). Furthermore, the firm selects a basic warranty and a price both of which influence market share. The firm selects a sport utility vehicle (SUV) as a type of the concept, 25 miles per gallon as a fuel efficiency, 5/60,000 (years/miles) as a basic warranty, and $35,000 as a price as summarized in Fig. 3. Concept Warranty Type SUV Fuel Efficiency 25 5 / 60,000 8 passengers passengers (miles/gallon) (years/miles) warranties are selected for the conjoint analysis study: 3/36000, 4/50000, and 5/60000 years/miles. Warranty SUV Convertible Sedan 3/36,000 58 13 28 4/50,000 20 14 21 5/60,000 8 2 6 4/60,000 1 0 2 (Years/Miles) Price Table 2 Basic Warranty Frequency N $ 35,000 Figure 3 Selected combination of concept, warranty, and price Finally, Table 3 shows the minimum, average, median, and maximum price of the benchmarked vehicles. Based on the benchmarking results, three price levels are selected for the conjoint analysis study: $20,000, $35,000, and $50,000. Figure 4 summarizes the features of the two competitor vehicles. The first competitor car (C1) is a convertible that gets 10 miles per gallon (gasoline engine). It has a basic warranty of 3 years/36,000 miles and a price of $20,000. The second competitor car (C2) is a sedan that gets 40 miles per gallon (hybrid engine). It has a basic warranty of 4 years/50,000 miles and a price of $50,000. Type Convertible Fuel Efficiency Non-hybrid Warranty Price 10 3 / 36,000 $20,000 C1 2 passengers (miles per gallon) (years/miles) Hybrid Sedan C2 40 5 passengers 4 / 50,000 $50,000 (miles per gallon) (years/miles) Figure 4 Competitor cars C1 and C2 3.2.Market share point estimate To estimate market share by applying non-Bayesian fullprofile conjoint analysis with customer evaluations in rating scales, the firm first identifies possible levels of fuel efficiency, warranty, and price to be analyzed in conjoint analysis. Based on the automobiles introduced to the market between 2003 and 2009 (87 SUVs, 29 convertibles, and 57 sedans), fuel efficiency, warranty, and price are benchmarked as summarized in Tables 1 through 3. Table 1 shows the minimum, average, median, and the maximum fuel efficiency of the benchmarked vehicles. Based on the benchmarking results, three levels of fuel efficiency are selected for the conjoint analysis study: 10, 25, and 40 miles per gallon. SUV Convertible Sedan Min 13 17 17 Average 17 23 22 Median 17 22 21 Max 24 33 44 Table 1 Fuel Efficiency (Miles per Gallon) Table 2 summarizes the frequency of basic warranties offered for the benchmark vehicles. Based on the benchmarking results, the three most widely offered basic SUV Convertible Sedan Min 20,972 17,365 20,342 Average 36,900 41,579 36,798 Median 36,105 35,513 28,045 Max 67,820 105,855 90,200 Table 3 Price ($) To estimate market share, the firm must identify 81(=3×3×3×3) part worths of three levels of four attributes (type, fuel efficiency, warranty, and price) for each respondent. Based on the part worths, the firm predicts each respondent’s utilities of the concept (as well as warranty and price) in Fig. 3 and two competing vehicles in Fig. 4. By comparing the utility of the concept (and warranty and price) against utilities of competing vehicles, the firm can predict which of the concept and competitors’ vehicles each respondent will choose. From the predictions of respondents’ choices, the firm estimates the market share of a concept as well as that of each competitor vehicle. Part worths can be identified by asking respondents to rate a total of 81(=3×3×3×3) combinations of attribute levels and analyze the rating results by, for example, ordinary least square regression analysis. However, evaluations of 81 alternatives may impose a significant burden on respondents. To reduce that burden, the firm may use an L9 orthogonal array so that respondents evaluate only the nine combinations. From 10 respondents’ evaluations of these nine combinations using a 9-point rating scale (with 1 being the least preferred and 9 being the most preferred), the firm can obtain the respondents’ part worths. Figure 5 shows the part worth of a respondent. This individual prefers an SUV over a convertible or sedan, as indicated by its part worth (1.3), which is larger than that of the convertible (–2.3) or the sedan (1.0). Similarly, the highest fuel efficiency of 40 miles/gallon, the best warranty of 5 years/60,000 miles, and the lowest price of $20,000 are the preferred levels, with the highest part worth within each attribute. The part worths in Fig. 5 are added to obtain the utilities of concept N in Fig. 3 and the two competitor vehicles C1 and C2 3 Copyright © 2010 by ISC in Fig. 4. For example, the utility of C1 (a convertible with a fuel efficiency of 10 miles/gallon, a warranty of 3 years/36,000 miles, and a price of $20,000) is calculated by adding the part worths of a convertible vehicle type, a fuel efficiency of 10 miles/gallon, a 3 years/36,000 miles warranty, a price of $20,000, and the intercept: – 2.3 – 1.0 – 0.7 + 2.3 + 4.7 = 3. Part Worth 3 1.3 1.0 0 -2.3 -3 Convertible SUV Type Sedan 3 Part Worth 1 0 0 -1 10 25 40 Fuel Efficiency (Miles per Gallon) Part Worth 3 -0.7 1st 2st 3rd 4th 5th · · · 200th bootstrap sample bootstrap sample bootstrap sample bootstrap sample bootstrap sample = = = = = { { { { { R7 R1 R3 R8 R3 , , , , , R10 R8 R10 R6 R4 , , , , , R8 R7 R10 R10 R5 , , , , , R10 R6 R9 R1 R10 , , , , , R9 R2 R4 R8 R4 , , , , , R6 R7 R5 R9 R3 , , , , , R6 R5 R2 R8 R9 , , , , , R9 R6 R3 R10 R10 , , , , , R9 R8 R4 R10 R3 , , , , , R6 R1 R2 R8 R7 } } } } } bootstrap sample = { R5 , R1 , R9 , R9 , R2 , R4 , R1 , R10 , R4 , R5 } Fig. 7 Bootstrap samples of respondents -3 0 3.3. Market Share Uncertainty Modeling: Bootstrap Bootstrap distributions of market share can be obtained by applying bootstrap to the conjoint analysis data. Using the original sample of 10 respondents (R1-R10), the samplingwith-replacement procedure is applied to generate 200 bootstrap samples. Each bootstrap sample consists of data from 10 respondents; however, each respondent may appear more than once or may not appear at all as shown in Fig. 7. 0.7 0.0 By mapping respondents in 200 bootstrap samples to their predicted choices, as illustrated in Fig. 8, the firm obtains 200 market share estimates for concept N and for the competitor vehicles C1 and C2. From these market share estimates, the firm constructs distributions of market share for its concept N and for the competitor vehicles; as illustrated in Fig. 9. -3 3/36,000 4/50,000 5/60,000 Warranty (Years/Miles) Part Worth 3 Bootstrap samples 1st 2st 3rd 4th 5th • • • 200th 2.3 0.7 0 -3.0 -3 20,000 35,000 Price ($) 50,000 = = = = = { { { { { C2 N C1 C2 C1 , , , , , C2 C2 C2 C2 C2 , , , , , C2 C2 C2 C2 N C2 C2 C2 N C2 , , , , , C2 N C2 C2 C2 , , , , , C2 C2 N C2 C1 , , , , , C2 N N C2 C2 , , , , , C2 C2 C1 C2 C2 , , , , , C2 C2 C2 C2 C1 , , , , , C2 N N C2 C2 } } } } } = { N , N , C2 , C2 , N , C2 , N , C2 , C2 , N } Fig. 8 Predicted choices Relative Frequency Figure 5 Part worth example Attribute levels Fuel efficiency Warranty Type Price R1 R2 (miles/gallon) (years/miles) N SUV 25 5/60,000 $35,000 7.3 7.3 C1 Convertible 10 3/36,000 $20,000 3.0 2.0 C2 Sedan 40 4/50,000 $50,000 3.7 6.3 , , , , , Market share (%) N C1 C2 0 0 100 40 0 60 30 20 50 10 0 90 10 30 60 • • • • • • • • • 50 0 50 Utility R3 R4 R5 R6 R7 R8 R9 R10 5.7 3.3 5.7 6.0 3.7 3.0 4.7 4.7 9.0 3.0 3.0 2.0 4.0 5.0 2.0 2.0 4.0 4.0 4.0 7.0 5.7 5.3 7.3 5.7 1 0.8 0.6 0.4 0.2 0 0 10 20 30 40 50 60 70 80 90100 Market Share (%) Figure 6 Utilities of N, C1, and C2 for the ten respondents, R1–R10 (a) N Relative Frequency After calculating utilities of N, C1, and C2 for each respondent, the firm can estimate market share of the concept N. Figure 6 summarizes the utilities of concept N and the competitor vehicles C1 and C2 for all 10 respondents; the largest utility of each respondent is highlighted. Respondents 1, 2, and 5 choose N because it has the highest utility of all three alternatives. On the same basis, respondent 3 chooses C1 and respondents 4, 6, 7, 8, 9, and 10 choose C2. Based on these results, the firm would estimate the market shares of these three alternatives to be 30% for N, 10% for C1, and 60% for C2. 1 0.8 0.6 0.4 0.2 0 0 10 20 30 40 50 60 70 80 90100 Market Share (%) (b) C1 4 Copyright © 2010 by ISC 0.6 0.4 0.2 0 0 10 20 30 40 50 60 70 80 90100 Market Share (%) (c) C2 Fig. 9 Market share distributions 3.4. Market share uncertainty modeling: binomial inference Based on the binomial inference, the market share distributions of N, C1, and C2 may be approximated by Beta(3,7), Beta(1,9), and Beta(6,4) distributions; as illustrated in Fig. 11. The probability distribution shows in Fig. 10 (b) was obtained by discretizing the cumulative distribution in Fig. 10 (a) into 11 brackets (i.e., [–0.05, 0.05], [0.05, 0.15], …, and [0.95, 1.05]), calculating the probability of market share (m) in each bracket, and assigning the probability to the middle value of the bracket: For example, Pr(m=0.10)=Pr(m≤0.15) – Pr(m≤0.05). Cumulative probability N: Beta(3,7) C1: Beta(1,9) 3.5. Comparison of bootstrap and binomial inference An important goal of decision analysis is the approximation (or discretizations) of continuous distributions by discrete probabilities. The extended Pearson-Tukey method is a simple yet accurate three-point approximation in which a continuous distribution is represented by the 5th percentile, 50th percentile (i.e., median), and 95th percentile of the distribution, with probabilities of 0.185, 0.63, and 0.185 respectively. Figure 11 compares the distributions obtained from bootstrap with those from binomial inference (i.e., beta distributions). Table 4 compares statistics of the bootstrap and beta distributions: three points (5th percentile, 50th percentile, and 95th percentile), the means, and the standard deviations of the discretized distributions. The differences of statistics (the percentiles, means, and standard deviations) are small, and differ only by a maximum of 3.4 percentage points. Cumulative probability Relative Frequency 1 0.8 C2: Beta(6,4) 1 0.8 N: Beta(3,7) C1: Beta(1,9) C2: Beta(6,4) N: Bootstrap C1: Bootstrap C2: Bootstrap 1 0.8 0.6 0.4 0.2 0 0 0.6 0.2 0.4 0.6 0.8 1 Market share 0.4 Fig. 11 Cumulative distributions 0.2 0 0 0.2 0.4 0.6 0.8 1 Market share (a) N: Beta(3,7) C1: Beta(1,9) N C1 C2 Beta Bootstrap Diff. Beta Bootstrap Diff. Beta Bootstrap Diff. -0.002 0.006 0.006 0.345 0.045 5th percentile 0.098 0.1 0 0.3 -0.014 0.074 -0.026 0.607 0.007 50th percentile 0.286 0.3 0.1 0.6 -0.050 0.283 -0.017 0.831 -0.069 95th percentile 0.550 0.6 0.3 0.9 -0.018 0.10 -0.018 0.6 0 Mean 0.30 0.32 0.12 0.6 -0.015 0.09 -0.003 0.15 -0.034 St. dev. 0.14 0.15 0.09 0.18 C2: Beta(6,4) Table 4 Statistics of discretized distributions 1 Table 5 compares distributions in Figure 11 using a chisquare goodness-of-fit test with a 5% confidence level. According to the results of this test, the null hypothesis that two distributions have a good fit cannot be rejected for all three cases (N, C1, and C2); as shown by the p-values larger than 0.05 in Table 5. Probability 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 p-value Chi-square test Test statistic 1 Market share N 0.27 3.92 C1 0.09 2.81 C2 0.18 6.23 Table 5 Comparison of distributions (b) Fig. 10 Binomial inference 5 Copyright © 2010 by ISC 7. CONCLUSION AND FUTURE WORK Conjoint analysis is a statistical methodology that has been increasingly incorporated in design engineering as a tool to estimate the market share of a new product design; however, the use of conjoint analysis data to model market share uncertainty has not been fully explored in the past design engineering research. This paper has presented two approaches (bootstrap and binomial inference as an approximation of bootstrap distribution) to model market share uncertainty using conjoint analysis data obtained from non-Bayesian full-profile conjoint analysis with customers’ evaluations in rating scales. Once discretized using the extended Pearson-Tukey method, the beta distributions (obtained from binomial inference) were compared with the bootstrap distributions. The results indicated that there are small differences in the statistics (the 5th percentiles, 50th percentiles, 95th percentiles, means, and standard deviations) of discretized distributions. Furthermore, chi-square goodness-of-fit tests of beta distributions and bootstrap distributions indicated that these distributions have good fits. These results support the use of a beta distribution, which is obtained from a binomial inference of conjoint analysis data, as a means to approximate bootstrap distribution; thus, to model market share uncertainty. Continuation of this avenue of research, which is to support this preliminary finding with a larger number of respondents, is future work. This paper used conjoint analysis data obtained from nonBayesian full-profile conjoint analysis with customers’ evaluations in rating scales. Market share uncertainty modeling using data obtained from other conjoint analysis methodologies, in particular, non-Bayesian and Bayesian choice-based conjoint analysis, is a topic for future work. The accuracies of a market share forecast depend on competitors’ products simulated in the conjoint analysis. Furthermore, the future actions of competitors in response to firm’s new product influence the accuracy of market share forecasts [16, 17]. Future work, therefore, should study the integration of competition uncertainty modeling into market share uncertainty modeling in conjoint analysis. 8. ACKNOWLEDGMENTS We would like to thank the Intelligent Systems Center at Missouri University of Science and Technology for supporting this research. 9. REFERENCES [1] Ulrich, K. T., and Eppinger, S.D., 2004, “Product Design and Development”, McGraw-Hill, New York. [2] Green, Paul. E., and Srinivasan., V., 1978, “Conjoint Analysis in Consumer Research: Issues and Outlook” Journal of Consumer Research, Vol. 5, pp. 103-120. [3] Robinson, P. J. 1980, “Application of Conjoint Analysis to Pricing Problems,” in Proceedings of the First ORSA/TIMS Special Interest Conference on Market Measurement and Analysis, D. B. Montgomery and D. R. [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] 6 Wittink, eds. Cambridge, MA: Marketing Science Institute, pp. 183-205. Benbenisty, R, L., 1983, “Attitude Research, Conjoint Analysis Guided Ma Bell's Entry Into Data Terminal Market,”Marketing News (May 13), 12. Green, P. V., and Srinivasan, V., 1990, “Conjoint Analysis in Marketing: New Developments with Implications for Research and Practice,” Journal of Marketing (October 1990), pp. 3-20. Doris, H., 2006, “Using Conjoint Analysis for Market Definition: Application of Modern Market Research Tools to Implement the Hypothetical Monopolist Test.” World Competition 29(2), pp. 315-336. Efron, B., and Tibashirani, R., 1993, “An Introduction to the Bootstrap”, Chapman & Hall, London. Felsenstien, J.,1985,’Confidence Limits on Phylogenies: An Approach Using the Bootstrap,” Evolution, 39(4), July, pp. 783-791 Efron, B., Holloran, E., and Holmes, S., 1996, Bootstrap Confidence Levels for Phylogenetic Trees,” Proceedings of the National Academy of Sciences of the United States of America, 93(23), pp. 1342913434. Kerr, M. K., and Churchill, G. A., 2001, “Bootstrapping Cluster Analysis: Assessing the Reliability of Conclusions from Microarray Experiments,” Proceedings of the National Academy of Sciences of the United States of America, 98(16), pp. 8961-8965. Holmes, S., 1999, “Phylogenies: An Overview,” in Halloran, M. E., and Geisser, S., eds., Statistics and Genetics, IMA Volumes in Mathematics and its Applications, 112, Springer Verlag, New York, NY. Holmes, S.P., 2002, “Statistics for Phylogenies,” Theoretical Population Biology, 63, pp. 17-32. Holmes, S., 2003, “Bootstrapping Phylogenies,” Statistical Science, 18(2), pp. 241-255. Reilly, T., 2002, “Estimating Moments of Subjectively Assessed Distributions,” Decision Sciences, 33(1), pp. 133-138 Howard AR (1970) Decision analysis: Perspectives on inference, decision, and experimentation. In: Proceedings of the IEEE 58(5): 632-643 Choi SC, Desarbo WS (1993) Game theoretic derivations of competitive strategies in conjoint analysis. Marketing Letters 4(4): 337-348 Green PE, Krieger AM (1997) Using conjoint analysis to view competitive interaction through the customer’s eyes. In: Wharton on Dynamic Competitive Strategy, Day GS, Reibstein DJ, Gunther RE (eds) John Wiley, New York Copyright © 2010 by ISC