Potato chip smackdown: Winners and losers

When I was growing up in the Midwest (Columbia, MO, to be precise), flavored potato chips were a favorite of mine. Though I preferred sour cream and onion, barbecue would do in a pinch. Imagine my delight, then, when my colleague Ryan Lekivetz informed me that our neighbors to the North had an entire new range of chip flavors to try!

And imagine my disappointment when I found out that two of the most popular flavors were ketchup and dill pickle.

“Blech,” I told Ryan.

“You’re just saying that because of the name,” he replied.

JMP staff volunteered to taste 10 different potato chip flavors in our choice experiment.

JMP staff volunteered to taste 10 different potato chip flavors in our choice experiment.

So, as statisticians will, we agreed to find out the answer based on a designed experiment.

We selected 10 potato chip flavors, including Canadian Dill Pickle and Canadian Ketchup, and asked people to compare them. Because we were working with volunteers from the US, we decided we’d mix in a few US favorites like barbecue and my beloved sour cream and onion, but we’d also use the Lay’s Do Us a Flavor chips to see how the Canadian chips would fare against other unfamiliar flavors. In the end, we had 10 different types of chips for our volunteers to taste:

  • New York Reuben
  • Southern Biscuits and Gravy
  • West Coast Truffle Fries
  • Greektown Gyro
  • Ketchup
  • All Dressed
  • Dill Pickle
  • Southern Heat Barbecue
  • Barbecue
  • Sour Cream and Onion

Like with the chocolate smackdown, we felt that asking people to taste 10 chips and rank them best to worst would be too difficult. Instead, we needed a designed experiment to break the 10 chips into manageable groups. The twist here would be that we’d not only ask for the “best” or “favorite” in each group, but we’d also ask for the “worst” or “least favorite” in each group. This would allow us to perform a MaxDiff (or Maximum Difference Analysis), another type of Choice analysis that would enable us to get more information from each choice task.

Ryan explains how he developed the experimental design in his blog post: Potato Chip Smackdown: US vs. Canada. Ryan’s design gave us four sets of potato chips, each with three chips to taste. As before, there were three different surveys, giving us a larger total number of comparisons. We gathered 30 volunteers to do the tasting (which took more coaxing than our last experiment, where we asked our volunteers to taste chocolate). Then, it was time to analyze.

The analysis

The MaxDiff model can be analyzed as if the respondents are making two choices. First, they pick the best of the offered choices. Second they pick the worst. Each one is treated like a separate choice. Each respondent, then, makes two choices for every group of chips he or she is shown. In the data, we represent this with two sets of entries for each choice set:

Data Table

If we focus on Choice Set ID 1, we see that there are three rows for the best choice and three rows for the worst choice. We use indicator variables to tell JMP which chips were in the set. For the rows that represent the best choice, the indicators are coded with a 1. For the rows that represent the worst choice, the indicators are coded with a -1. Each of the indicator columns is entered as a continuous variable in the choice model, so JMP is forced to make the parameters associated with the “Worst Choice” rows equal to the negative of the parameters associated with the “Best Choice” rows.

In the model dialog, we make sure that the “Best or Worst Selection” column (the column that tells whether the rows are associated with the best choice or the worst choice) is added to the grouping columns:

Choice Dialog

We also have to select one flavor as our comparison flavor and leave its indicator out of the model (just like you would with any set of dummy-coded columns). In this case, I made sour cream and onion our comparison.

In order to get a ranking of best flavors to worst flavors, I sorted the resulting report (right-click and select “Sort by column”). Barbecue and Southern Barbecue are near the top. The Likelihood ratio tests are testing whether the parameters are significantly different from 0, so in this case, whether flavors are significantly different from sour cream and onion. It looks like the two familiar barbecue flavors are favorites, while Gyro definitely trails the others.

Visualizing the results

I also created a graph of the results (Right-click the estimates table and select “Make into Data Table." Then I opened Graph Builder and followed the steps described in this blog post.


If we just use the graph, Truffle Fries is the next most popular after the barbecue flavors, followed by Biscuits and Gravy. The Canadian flavors are near the bottom, but the statistical tests tell us that if we’d sampled different people (or the same people on a different day), those flavors might sort higher.

So, who's right?

Maybe we were both right. Maybe I was just reacting to the name “ketchup chips."  Maybe if I tasted one of them in a blind test, I’d like them just as much as sour cream and onion. Maybe if I’d grown up in Canada, I’d like them better than barbecue.

I’d be really surprised, though, if Gyro won the “Do Us a Flavor” challenge.

Post a Comment

Potato chip smackdown: US vs. Canada

Four rows of three potato chips

Each taster in the choice experiment got a set of potato chips like this. Can you guess which one is the ketchup-flavored chip?

I grew up in Canada, where ketchup potato chips were a staple at most children's birthday parties. As a huge fan of these ketchup chips, I was unsure whether my enjoyment of the wonderful flavor was simply nostalgia and whether other people unfamiliar with the flavor would show any love for ketchup chips. Similarly, I’ve noticed that dill pickle flavored chips and the ever-popular "all dressed" flavor are uncommon here in the US compared with Canada.

After our chocolate smackdown was such a success with the JMP group, Melinda Theilbar and I were looking for another opportunity to conduct a choice experiment. To make it fun, we wanted the experiment to contain a number of flavors new to the tasters. The fortuitous timing of Lay’s chips new “Do Us a Flavor” campaign and a visit from my parents in Canada presented the perfect opportunity: potato chip smackdown!

While we figured that tasting chips would be not as popular as tasting chocolate, we also did not have as many worries about limiting the number of tasters due to limited resources -- the chip bags contained plenty of chips. The ideal mix of flavors for the experiment would be a combination of mostly unfamiliar flavors, with a couple of more familiar flavors thrown in. Although the chocolate experiment had six different chocolates among two factors (with two origins and three cacao contents), this time we have just one factor with more levels. We ultimately ended up with 10 flavors:

  • New York Reuben
  • Southern Biscuits and Gravy
  • West Coast Truffle Fries
  • Greektown Gyro
  • Ketchup
  • All Dressed
  • Dill Pickle
  • Southern Heat Barbecue
  • Barbecue
  • Sour Cream and Onion

How to get the response?

What we want is a general ranking of the desirability of each of the flavors. One way to do this is to get tasters to provide their own ranking of the flavors. However, with tasting so many different flavors (and the fact that we didn’t want to stuff people with potato chips), this type of forced ranking seemed like it would be a difficult task requiring lots of little bites.

Picking the favorite among a smaller group of choices is much less demanding to the tasters, but carries less information. There are 45 (10 choose 2) different pairwise comparisons for the 10 potato chips. While we don’t require all of these pairs to appear in the survey, if we give tasters five or six pairs of chips, it doesn’t sound like we’re getting much information.

Another option: Size 3

Instead of choice sets of size 2, we decided to use choice sets of size 3. Doesn’t sound like a big deal? Consider this: In picking a favorite, we actually get information comparison information for the favorite versus each of those that were not chosen.

That’s not all

In addition to reporting the favorite from each choice set, we also had the tasters report their least favorite flavor from each choice set. If we just asked for the favorite in each choice set, we don’t get any information about the two flavors not chosen in relation to each other. By asking for the favorite and least favorite, we gain that information.

OK, but can you still analyze it?

Analysis is going to require some extra work, but it’s certainly doable within JMP. Melinda will go through the analysis next time (sorry, you’ll still have to wait for the results – no hints yet!).

The design

We can use the Custom Designer to create a design in much the same way that we did for the chocolate experiment. It was decided that each taster would get four choice sets, and four different surveys would be created.

1. Add a false VHTC and false HTC factor to set up random blocks corresponding to surveys and choice sets, along with a 10-level factor for flavor.


2. Remove the false VHTC and HTC effects from the model.


3. Set up the number of whole plots for the total number of surveys (4), subplots the total number of choice sets (16 = 4*4), and the run size as the number of choice sets times the number of chip flavors per choice set (48 = 16*3).


Final thoughts

Melinda and I have completed this experiment, but I don't know the results yet. The choice sets of size 3 did seem to be a good choice in this case, as the tasters found it reasonably easy to make the ranking for each set they were presented with. I didn’t have a notion that there was a clear-cut winner as the results were being collected, but there’s one particular flavor that I suspect will not fare very well. Thanks for reading!

Post a Comment

Gap analysis of JMP for marketing and consumer research

One of the key application areas for JMP is consumer and market research. As I was invited to give a presentation on JMP for marketing analytics, I was curious about how complete the capabilities of JMP are. I wondered whether there are any significant gaps.

At a recent American Marketing Association conference I discovered a popular textbook titled Modern Marketing Research: Concepts, Method and Cases. The book is written by three marketing professors at the Ross School of Business, University of Michigan and has been used by many top business programs. So, the book is suitable as a benchmark reference for my gap analysis that compares what’s being taught at leading MBA programs and what’s available in JMP.

The textbook is divided into three parts. The first two parts discuss the principles of marketing research, as well as survey designs, sampling and data collection. Part 3 covers the analytical and statistical approaches for the data analysis, which is my focus of the gap analysis.

I found essentially no gaps.

The textbook covers a comprehensive set of traditional statistical methods used in marketing and consumer research such as descriptive statistics, hypothesis testing, multiple regression and multivariate methods. It then goes on to discuss choice models (aka conjoint analysis) and multidimensional scaling, and it ends with a chapter introducing several emerging analytical techniques and concepts like heterogeneity, hierarchical models, etc.

JMP users perhaps know the Distribution, Fit Y by X, Fit Model and Multivariate Methods platforms well, but might be less familiar with the random coefficient models in JMP Pro through Mixed Model personality (please check out my JMP Pro for Linear Mixed Models blog posts). You can use random coefficient models and nesting random coefficients to model “slope heterogeneity” and multi-level structures as discussed in the book’s chapter 12.

For choice models, JMP provides both modeling and design of marketing experiments involving product features and consumer attributes. My colleague Melinda Thielbar has written about the latest development in the Choice Model platform. I used the Dial Readout Scales case study from the textbook, and JMP not only produces the part worth estimates and likelihood ratio tests, but creates the interactive probability and multiple choice profilers that enable you to visualize the consumer preferences for alternative new products.

Probability and multiple choice profilers that enables you to visualize the comparison between competing products.

Probability and Multiple Choice Profilers enable you to visualize how competing products compare.

As for the multidimensional scaling, another colleague of mine, Kelci Miclaus, has created a JMP add-in  that performs the analysis through SAS or R integration.

What’s more, JMP Pro has other advanced analytical methods such as uplift models and data mining that are useful to marketing analysts and consumer researchers.

For those of you who would like to play with the data and cases from the textbook in JMP, I have created a JMP add-in that collects all of the data as JMP tables.

Post a Comment

Discovery Summit 2015 live blog: Doug Montgomery

Arizona State University professor Doug Montgomery gives a keynote speech at Discovery Summit 2015 in San Diego, California, titled "The Flight of the Phoenix."

Montgomery, a professor of engineering and statistics, discusses the reasons that some people believed that design of experiments was no longer of interest, and new developments and applications that have made this one of the most active and important fields of applied statistics.

View the live blog of this speech.

See photos, tweets and other live blogs from the conference at jmp.com/live.

Post a Comment

Discovery Summit 2015 live blog: Xan Gregg

JMP Director of R&D and data visualization expert Xan Gregg gives a keynote speech at Discovery Summit 2015 in San Diego, California, titled "All Graphs Are Wrong, But Yours Can Be Useful."

Gregg, creator of Graph Builder in JMP, talks about how understanding visual perception can help us make more effective data visualizations.

View the live blog of this speech.

See photos, tweets and other live blogs from the conference at jmp.com/live.

Post a Comment

Discovery Summit 2015 live blog: Phil Simon

Technology expert and author Phil Simon gives a keynote speech at Discovery Summit 2015 in San Diego, California, titled "How to Create a Culture of Analytics."

Simon contends that pockets of analytic excellence in an organization are often suboptimal and potentially dangerous. What is truly powerful is an organization in which analytical thinking pervades the culture – where experimenting is valued and decisions are based on data as opposed to gut instincts.

View the live blog of this speech.

See photos, tweets and other live blogs from the conference at jmp.com/live.

Post a Comment

Discovery Summit 2015 live blog: John Sall

SAS co-founder and JMP creator John Sall gives a keynote speech at Discovery Summit 2015 in San Diego, California, titled "A Few of My Favorite Things."

Sall takes a deep dive into a few aspects of what he likes best about JMP and JMP Pro. You may be surprised and delighted at Sall’s choices.

View the live blog of this speech.

See photos, tweets and other live blogs from the conference at jmp.com/live.

Post a Comment

Creating graphs of muscle quality and body fat in Graph Builder

In my previous post, I shared several Graph Builder graphs of the muscle quality and body fat percentage data I’ve been getting from my Skulpt Aim. However, since the post was already long, I didn’t have a chance to include any instructions on how to recreate those graphs. In case you have similar data or want to create a similar type of graph, here’s how to do it! I have also included animated gifs that capture the main steps in the process of creating each graph. (Click the animated gifs and graphs to see them enlarged.)

Weight Graph with Annotations

This graph shows all the weight data points that I have gleaned from medical record requests, notebooks, and wireless scale data from birth through about 2014. My table includes columns with weight measurements in rows and the date of each in m/d/y format.

Weight history purple and aqua2

In Graph Builder, I used the following steps to create this graph:

  • Dragged Weight to the Y axis (I deleted the label intentionally).
  • Dragged Date to the X axis.
  • Right-clicked in the graph to choose Smoother > Change to > Line.
  • Right-clicked on the line in the legend to change the Line Color to black.
  • Unchecked the Show Legend option under the red triangle next to Graph Builder.
  • Right-clicked on the X axis and increased the axis label nesting to 3, and then turned off labels and tick marks on Level 1 and Level 2 tab (corresponding to day and month) and tick marks on Level 3 tab (year).
  • Added a Y-axis reference range by right-clicking on the Y axis to open the Axis Settings dialog, clicking the Allow Ranges checkbox, and specifying a range extending from 130-135.
  • Added text annotations by clicking on the toolbar icon, typing in my text, and right-clicking to change the background color, font color and style. I color-coded the annotation for my childhood weight range in a different color.
  • Added vertical line annotations to indicate each of the weight ranges for childhood and adult years.

Lifelong weight history short

Graphs of Body Fat Percentage and Muscle Quality

The stacked table I used to create this graph included columns for Date, % fat, MQ, Body part, and Side, with one row for each observation.

To create this graph, I took the following steps in Graph Builder:

  • Dragged % fat to the Y axis drop zone.
  • Dragged MQ to the X axis drop zone.
  • Clicked on the smoother element icon to remove it.
  • Added a regression line by dragging that element from the Graph Builder toolbar into the graph.

To create the version shown below with a separate color and regression element for each body part, I dragged Body Part to the Overlay variable drop zone. You can see my steps in the animated gif below.

fat vs MQ and overlay

Bar Graph of Replicates by Side

To create this graph, I started with my Skulpt Aim results table, and then created a summary table from that data by clicking Tables > Summary, choosing the Replicate column, selecting N as the Statistic, and adding Body part and Side as Group variables.

Starting from the Summary results table, I performed the following steps:

  • Dragged N(Replicates) to the Y-axis drop zone.
  • Clicked the Bar element on the Graph Builder toolbar.
  • Changed the Summary Statistic to Sum in the Bar options on the left.
  • Dragged Body part to the X-axis drop zone.
  • Dragged Side to the Group X drop zone.

Bar graph of replicates by side and body part short

Body Shape Graph of MQ and % fat

I made my next two graphs using custom female muscle shape files I created with the free Custom Map Creator add-in. I described how I used the add-in to create these files in a previous post. By the way, I uploaded copies of my map files to the JMP user community here. Apologies to the guys -- you are going to have to create your own shape files! To make my custom map visible to JMP, I placed the shape files created by the add-in into the folder C:\Users\~myusername\AppData\Roaming\SAS\JMP\Maps.

Comparing my shape files to the areas measured by the Aim, I noted that the Back measurement area for the Aim was in the area I call Trapezius in my shape files. I created a copy of my Body part variable in my Aim results table and named it Body shape, then changed all occurrences of Upper Back to Trapezius in that column so that the colors would show up in the correct area of my body map. I did wish that I had done a better job with my Trapezius shape.

Then, I performed the following steps:

  • Dragged in MQ as a color variable.
  • Right-clicked on the axis and chose gradient.
  • In the Gradient Settings dialog, clicked the Discrete Colors checkbox and clicked on the Color Theme bar to choose Muted Yellow to Red as the Current Theme.
  • Right-clicked on the X and Y axis to open the Axis Settings dialog and uncheck the boxes for tick marks and labels.

I performed the same set of steps for my % fat graph which appeared below the MQ one, except that I also clicked the Reverse Color checkbox so that lower body fat areas would show up colored red rather than yellow. As you’ll recall, MQ and % fat are inversely related, and the Skulpt app also uses color gradients that color high MQ and low % fat similarly.

Body map by MQ

Graphing My Measurement Variability

My final graph required calculations of daily Standard Deviation by body part and side for both MQ and % fat. I initially used Summary to calculate these statistics, dragging the SD columns from the linked Summary table into my main table. While this is a handy shortcut for a one-time calculation, I preferred for the calculation to happen automatically when I added new data to the table rather than repeating the Summary step each day.

Thanks to a suggestion from Xan Gregg, I decided to use a Column Standard Deviation formula column and specify my Body Part/Side concatenated variable and Date as the two optional By Vars in the formula. This gave me values for SD(MQ) and SD(% fat) for each body part, side and day.

Formula SD MQ

To create a graph of these values, I performed the following steps:

  • Dragged SD[MQ] to the Y axis drop zone, unchecked the Jitter option, and chose Mean as my Summary Statistic.
  • Dragged SD[% fat] to the Y axis drop zone below MQ (in a separate section of the graph), unchecked Jitter, and chose Mean as my Summary Statistic.
  • Added Body part as an Overlay variable.
  • Added smoothers to both graphs by dragging that element in from the Graph Builder toolbar.
  • Dragged Month, then Day to the X axis to create a nested axis layout.
  • Added Side as a Group X variable.
  • Added a Local Data Filter from the toolbar icon, choosing a subset of Body parts (Abs, Bicep, Tricep, Quadricep, and 4 site average) and the Right Side in the filter.

Reducing my measurement variability short

I also made some final adjustments to ranges and decimals in the Axis Setting dialog, added shaded axis reference ranges between 0 and 1, and increased the Graph Spacing by right-clicking on the division between the graph sections and changing the number to 5.

I’ll be giving a talk on using my custom body map file with my workout data at JMP Discovery Summit in San Diego next week. I hope to see you there! If you won’t be in San Diego, you can see my previous blog posts on creating custom maps and workout data visualization in my Fitness and Food series on the JMP Blog.

Post a Comment

New to Discovery Summit? Let me connect you

If you’ve been to a Discovery Summit, you know what you have to look forward to. One of the best parts of the conference is meeting other smart, curious people who are interested in discovering cool things in data.

Even the introverts come out of their shells when around hundreds of other like-minded people. When smart, curious and interested people get together, they talk. They talk a lot. And when a large cadre of attendees return year after year, they have a lot to share and catch up on.

So what about people who have just recently started playing with data in JMP or who are novices to the conference?

To those people, I say: Come find me.

I’ll introduce you to other people in your industry who have similar business pains, to users who apply JMP in the ways that you do so you can share best practices, and to the Steering Committee members who select the papers and posters. I’ll hook you up with the developers who work on your favorite JMP platforms. I’ll even introduce you to John Sall, SAS co-founder and JMP creator. Like you, he’s smart, curious and interested in learning new things.

Malcolm Gladwell (a speaker at a previous Discovery Summit) would call me a “Connector.” Per Wikipedia, “Connectors are the people in a community who know large numbers of people and who are in the habit of making introductions.” That’s me. There’s a good chance I’ll know somebody who knows somebody who knows somebody. By the end of the Summit, you’ll feel like you know a whole lot of somebodies too.

So, when you get to San Diego on Sept. 14, come find me so I can start connecting you.

Post a Comment

Creating a covering array when you can't test some factors together

I’ve written before about how we use covering arrays created by JMP to test JMP itself. A recent example that came from my colleagues Wenjun Bao and Joseph Morgan was so intriguing that I wanted to share it with others.

To test a particular platform in JMP Genomics, there are 14 different factors that can be varied, ranging from 14 levels to two levels. What makes this case different from the testing software preferences example I discussed previously is that for three of the factors (Interactive Hierarchical Clustering Options, Automated Hierarchical Clustering Options, Minimum Recombination Grouping Options), only one can be set for any given test run. This restriction arises because of the behavior of the following radio control:


In this example, we have a factor for Linkage Grouping Method that has three possible levels. Each of the grouping methods has one of the above mentioned factors associated with it. Essentially, these three associated factors break into three separate cases to consider. If we wanted a strength 2 covering array, couldn’t we just create a strength 2 covering array for each possibility? This would ensure that each possible (allowable) pair occurs in our testing.

We could, but…

The two factors with the largest number of levels have 14 and 9 levels, respectively. If we didn’t have any restrictions, the smallest possible strength 2 covering array has 126 = 14*9 runs (and the covering array platform can find such a design). However, if we create 3 separate covering arrays, that means each will be 126 runs, for a total of 378 = 126*3 runs. A strength 2 covering array ensures each possible pair occurs at least once, but by breaking it up into three covering arrays, we end up having more coverage than we need. By combining the three covering arrays, each of the pairs not involved in the restricted factors is actually occurring at least three times.

What would be nice is a design that has missing values for factors that cannot occur based on the other settings.

Really? Missing values?

When you think of it from a traditional DOE standpoint, creating a design with missing values sounds silly. But for covering arrays, where we’re looking at combinations of factors, it makes perfect sense: If a factor has a missing value in a row, it means it’s not relevant for that particular test. This also means that if we see a failure for that test, we know the missing factor is not involved in the cause. Fortunately, our Analysis tool recognizes missing values as well.

OK, so how can I create this design?

Consider one of the three grouping methods, Automated Hierarchical Clustering. When it appears, another factor, Automated Hierarchical Clustering Options, can take on three different levels, while the factors Interactive Hierarchical Clustering Options and Minimum Recombination Grouping Options should be missing. We can use the handy disallowed combinations filter, and when Linkage Grouping Method is Automated Hierarchical Clustering, disallow all values for Interactive Hierarchical Clustering Options, join with an OR and do the same thing with Automated Hierarchical Clustering and Minimum Recombination Grouping Options.


We could then follow a similar procedure for the other two grouping methods linking these with OR statements from the Data Filter. So we should now be ready to create the design…

Not quite yet

We’ve overlooked one thing in our disallowed combinations that is very easy to overlook – the designer will still try to make pairs of those restricted factors show up in rows with Grouping Method missing, which doesn’t make any sense for our design. So, we have to disallow all possible combinations between those columns from occurring. For example, if we choose Interactive Hierarchical Clustering Options and Automated Hierarchical Clustering Options from the filter, we would get (with the earlier disallowed combinations cropped from the top):


After we’ve done those combinations and the previously mentioned disallowed combinations, all connected by using OR in the filter, we can create the design.

The design

The Covering Array platform finds us a 140-run design (more on this in a bit, but this is, in fact, the smallest possible run size), with only one of Interactive Hierarchical Clustering Options, Automated Hierarchical Clustering Options, Minimum Recombination Grouping Options set for each row and the other two missing. I’ve put the data table on the File Exchange in the JMP User Community, where you can see the final result and take a peek at what the resulting disallowed combinations looked like.

Final thoughts

While some extra work was needed to set up the disallowed combinations, the savings in the number of tests (378 vs. 140) was dramatic. With the help of the Disallowed Combinations Filter, once it was determined what should be disallowed, it was easy to input them, even with a number of different combinations.

The keen reader may have noticed I could collapse all the possible values from the three variables into one factor with 10 levels (this is how I knew that 140 = 14*10 runs was the lower bound). While it’s certainly possible to construct the design in such a way, it’s easy to lose the context as to what that variable is trying to describe.

Do you run across similar types of problems when using covering arrays? If there’s interest, we can see about adding the capability to easily generate the specialized disallowed combination when some factors cannot be tested together. Thanks for reading!

Post a Comment