Coming in JMP 12: Query Builder

We hear frequently from JMP users that they spend a great deal of time preparing their data before they can start analyzing it. We added Query Builder to JMP 12 to streamline the process of getting data from relational databases into JMP and ready for analysis. JMP has had the Open Table feature for opening tables from relational databases for many releases, but Open Table has some notable limitations: it does not help with joins or filtering, and the queries it does produce can be difficult to share.

Query Builder has been designed to help you build much more complex database queries without having to become an expert in SQL syntax. It can automatically join database tables based on primary and foreign key relationships and help you interactively filter the data. Also, with a few clicks, you can turn the filter into a prompt that can help others use the query you built to get the data they need.

Figure 1

Selecting tables to include in a query

 

Figure 2

Query Builder example with filters

 

While building your query, you can specify the modeling types and formats that variables should have once they are in JMP, eliminating the need to do this manually or write a JSL script to do it. You can even include Recode or other cleanup scripts to be run after data is returned by the query. The result is a single, easily shareable file that delivers analysis-ready data to you or your colleagues in one step.

Query Builder also has a “Custom SQL” mode. If you get to a point building a query interactively where you need something that Query Builder does not provide, you can switch into “Custom SQL” mode and modify the SQL directly to get exactly what you need.

Look for more details on how to use Query Builder in future posts here when JMP 12 is released.

Editor's note: This post is part of a series of previews of JMP 12 written by the people who develop the software.

Post a Comment

Visualizing pregnancy and post-baby weight loss with Graph Builder

My first favorite visualization from my JMP Discovery Summit poster was a multiple-element Graph Builder creation that tracked my weight trends through pregnancy, post-baby weight loss, and two and half years of weight maintenance. This seems like an especially appropriate time of year to share this graph since  so many New Year's resolutions include weight loss goals!

While BodyMedia provides a weight graph in their software and you can export weekly or monthly summaries of calorie-related measures in PDF format, it doesn't offer a unified graphic to track long-term weight changes along with deficit and macronutrient information. Although the final graph may look complicated, it's actually pretty easy to create it in Graph Builder once you know how. Xan Gregg, lead developer of Graph Builder in JMP and head of our Data Visualization group, actually recreated my final graph during part of his "Unlocking the Power of Graph Builder" talk at Discovery Summit 2014. (If you missed the conference, you can see the recording in the JMP User Community.)

In the early version of this graph shown below, I used points for weight measurements, with multiple weight measures stacked within weeks. I added bar elements to display average weekly percentages for the various macronutrients I tracked. I added a heat map to show the number of days out of each week that I ate in a calorie surplus or deficit, with red indicating more days in a particular range and blue indicating fewer. I used a smoother on the heat map and Y axis range lines to show the deficit and surplus ranges in 500 calorie increments. An X axis reference line showed the end date of my pregnancy.

Early version of main plot

Prior to creating the final version of this graph for my poster, I met with Xan, who recommended simplifying the presentation of this graph by:

  • Lightening axis reference lines so that they did not dominate the data.
  • Superimposing macronutrient percentage values in the same section of the graph using lines rather than showing them as separate bar graphs and simplifying the tick mark display.
  • Changing the variable type for macronutrient percentages so that they were displayed as percentages and not ratios.
  • Moving X axis labels to a horizontal orientation.

When we changed the macronutrient percentages to display as lines, I noticed a big problem. Connecting through the missing values for alcohol percentage gave the mistaken impression that I was drinking through my pregnancy! To fix this, Xan showed me that I could right click in the macronutrient line graph to display bars along with the lines. We then used the Variables section in the left side property pane of the Graph Builder window to adjust which graph elements were shown for each variable. I turned off the bars for protein, carbs and fat, and turned off the line for alcohol, and then adjusted the color of the alcohol bar to purple. We labeled each element with annotations from the Tools menu, right clicking to remove the annotation box outline, change its background color to white, and update the text color of the annotation to match each graph element. We also removed the bar display for total calories since it didn't add much to the overall story.

Xan also recommended choosing a continuous color theme for the heat map element to replace the divergent red to blue theme. We both thought that the count of days spent within different deficit or surplus ranges was shown much more clearly by a continuous red to yellow color theme. Red indicated more days spent in a specified deficit or surplus range, while the lighter yellow meant fewer days spent in that range. Below is the final graph we created.

Main poster graph

I’ll confess to having used one JMP 12 axis feature to create this graph. Can you spot it?

(I added a nested week variable to the X axis so that my data would be summarized by week. The JMP 12 feature I used was the ability to selectively turn off the labels so that week numbers didn’t appear in my graph. Without a doubt, you’ll hear more about these and other JMP axis settings updates when JMP 12 comes out! Check out Xan's preview of what's coming for Graph Builder in JMP 12.)

I have collected my own calorie, weight and exercise data in notebooks during past weight loss efforts, and I've used this data-driven approach to lose 45 pounds prior to my second pregnancy. The graph above reveals what I already knew: I was in a calorie surplus during most of my pregnancy, and as a result I gained more than the recommended amount of weight. This happened even though I chose healthy foods and exercised. On the plus side, my son was born at a healthy 8 pounds, I gained less than I had during my first pregnancy, and I experienced fewer discomforts this time. But when I weighed myself three weeks after my son’s birth, I had to face the fact that about half of my pregnancy weight gain was baby and water weight and the other half was extra body fat. As you can see from the graph, it took six months of consistently eating less than I burned to lose the excess and arrive in my current maintenance weight range. The percentages of carbs, fat and protein I ate while in this post-pregnancy deficit didn't vary that greatly from the percentages noted during my pregnancy.

If you have read the first post in this series, you know that I've experienced drastic weight swings in my lifetime, and I'm determined to avoid large swings going forward. Accordingly, I have examined my data on my past and present eating and activity patterns to learn what works for me to reach (and maintain) a healthy weight. First and most importantly, I have determined that I am most successful at loss and maintenance when I actively track my data. This takes less and less time as technology improves. I used to track my weight, diet and workout data all on paper , but most of my data collection has been automated by an activity monitor, food tracking app and wireless scale. I continue to track my workouts the old-fashioned way in notebooks, but the effort is minimal. I have even begun to examine my historical workout data in JMP....Stay tuned for posts on that topic in the future!

I find that tracking my diet and fitness efforts provides daily accountability as well as data. When reviewing my historical data, I found that I would usually stop tracking during holidays or stressful times when travel and/or social commitments led me to over-indulge and skip workouts. I'd feel guilty, which would lead to negative self-talk, more excuses, and eventually to weight gain. I have worked actively for the past five years to overcome these tendencies, committing myself to enjoying social events and always logging what I eat without sinking into guilt. During busy times, I may cut down on workout length, but I always make an effort to do something rather than nothing.

After monitoring my weight nearly daily for two years with a wireless scale, I have identified which factors can cause day to day fluctuations (e.g., water changes due to hydration, workouts, sodium and carbs) and which cause true long-term weight trends (e.g., eating in a surplus or deficit). This understanding has helped me to worry less about small short-term fluctuations, pay attention to the bigger picture, and maintain my weight within a small range for the past 2 ½ years. I now understand the daily eating and activity patterns that can keep me within my maintenance zone. When I see myself drifting upward in weight or calories, I can recognize the pattern relatively quickly and adjust the numbers to get back to the range where I want to be.

To reproduce this graph with your own data in Graph Builder, open a data table containing a daily surplus/deficit measure (which I calculated from my data with the formula Calories Burned-Calories Consumed), macronutrient percentages, and body weight where each rows contains data from a single day. (Of course, you could also make smaller graphs containing only a few of these measures if you don't have them all.)

If your data table doesn't have Year and Month variables, you can create these from your Date variable by right-clicking on your date in the Graph Builder column chooser and adding them as new transform columns from the Date Time menu. This will extract the month from the Date column, and you can use a Value Ordering property on the Month variable to create a column of ordered short Month names. I made sure that Year and Month Name were both specified as Ordinal before adding them to the same axis.

To replicate this graph, open Graph Builder and drag:

  • Year to the X axis.
  • Month Name to the X Axis (just above Year, so the axis shows Month Name nested within Year).
  • Week to the X axis (just above Month name, so Week is nested within Month).
  • Weight to the Y axis.
  • Deficit/surplus to the Y axis below weight (so that they appear in separate sections). Drag the heat map element into the deficit/surplus section. Right-click on the color gradient element in the Graph Builder legend on the right to change the color theme to a continuous one; I chose Muted Yellow to Red.
  • Macronutrient percentage columns (e.g., fat, carbs, protein, alcohol) into the same section below the heat map element. Right-click (or drag in) to change the element type to bar and also add lines. In the right-click menu in that section of the graph, or in the left size variables tab, remove the bars for fat, carbs and protein. Remove the line for alcohol.

To complete the graph, adjust your Y-axis, graph title and axis titles if desired and add one or more annotations from the Tools menu. You can right-click on annotations to change their appearance to match your graph like I did. In upcoming posts, Xan and I will show how we created and optimized other favorite visualizations of my personal data that appeared on my poster.

Check out the first blog in this series to learn more about my interest in quantified self (QS) data analysis and my JMP Discovery Summit 2014 e-poster that explored 1,316 days of my activity and food log data in JMP. You can use my JMP add-in to import your own BodyMedia® files. It’s free to join the JMP User Community, where you can learn from JMP users all over the world! In the videos section here, you can watch a video of Xan showing how to reproduce part of this graph during his "Unlocking the Power of Graph Builder" talk at JMP Discovery Summit 2014.

Post a Comment

Coming in JMP 12: Graph Builder improvements

Ease-of-use and convenience functionality have been the focus of graphics work for JMP 12. The most prominent ease-of-use change is the drop zones in Graph Builder, which now light up whenever you're dragging something in.

Graph Builder Drop Zones

Now everyone can find the "secret" drop zone just inside the axis. It even has a hover tip when the mouse is over it.

A few of the functionality improvements are shown below.

  • Reference lines can now be reference ranges to highlight regions of a scale.
  • Line connections can be stepped or smooth.
  • Histograms can be overlaid with automatic transparency.

A few other important graphical enhancements are:

  • A new Axis Settings dialog with more options and a preview.
  • New scales for color gradients including quantiles.
  • A new squarify layout for treemaps.
  • Selection linking with legend items.

The new legend interactivity brings a couple productivity enhancements that I'll demonstrate with a sequence of Graph Builder snapshots of US state crime rates overlaid by region.

legend11

The initial graph is a bit busy, but in JMP 12, you can click on legend items to highlight each region. Alternately, if you click on a point or a line, you will highlight the corresponding item in the legend. In this case, you can see that the Pacific region has a markedly different regression line from the others.

legend12

If you want to focus on that difference for a report, you can turn all the other regions gray and partially transparent. With the JMP 12 legend selection, you can select the other regions and then change their settings with a single right-click.

 

legend13

Finally, you can make the Pacific line thicker to make it stand out even more. You might go further by removing or collapsing the other regions in the legend since they all look the same, but they can still be useful for interactive selection and highlighting.

legend14

Graph Builder experts will know that it's possible get the same result in JMP 11 with a bit more effort. By reducing the effort for such tasks in JMP 12, we hope to encourage greater exploration and discovery.

Editor's note: This post is part of a series of previews of JMP 12 written by the people who develop the software.

Post a Comment

JMP 12 is on the way!

All the members of the JMP Development team are anticipating the day when we can finally share JMP 12 with you, our customers! If you attended JMP Discovery Summit 2014, you were lucky enough to get an early look at many of the new features and improvements coming your way. Soon you’ll be able to visit the JMP website for in-depth descriptions, screenshots and videos of our latest release. You’ll also get the inside scoop from members of the development team as they blog here about their favorite new JMP 12 features.

Each and every release, we do our best to add features that can help you solve new problems and simplify your existing workflows. We are guided by feedback from a variety of sources, including suggestions that come in through JMP technical support, information gathered during in-person and web meetings with customers, and the many ideas we receive from dedicated JMP 12 beta testers. As you get to know JMP 12 on your own desktop, we hope that you will share your thoughts with us about how we can continue to improve JMP. (And if you haven’t yet joined the free online JMP User Community, please join in the conversations there!)

The additions we’ve made in JMP 12 further strengthen JMP as an end-to-end data analysis solution designed for scientists and engineers. We’ve made it easier for you to access your data. You can now filter and draw data from databases with the new SQL Query Builder, and we’ve extended the popular Excel Import Wizard to the Mac. We’ve simplified the process of data cleanup with a variety of new utilities and transformed the JMP Recode platform, adding time-saving new features like manual and automatic grouping options (my favorite).

We’ve also added highly requested new analytic functionality to JMP 12 with new Multiple Correspondence Analysis, Destructive Degradation, and Process Capability platforms. Many modeling platforms now offer an interactive model editing dashboard that lets you quickly assess the impact of adding and removing terms. We’ve also made major improvements to a number of popular JMP platforms including Graph Builder, Cluster, Discriminant, and Ternary.

When you’re ready to share your results, you have new options in JMP 12, allowing you to save JMP output directly to PowerPoint and also export interactive HTML versions of the Profiler and Bubble Plot for viewing in web browsers. JMP Pro includes all of the enhancements mentioned above, plus a new Covering Arrays platform, PLS-DA features, and extensive updates to existing mixed model and generalized regression features.

There are far too many new features in JMP 12 to mention them all here, but you’ll soon be able to peruse the new features guide for more details. As with every release, we have made major additions to JMP Help and JMP Books to cover new features. In addition, DOE documentation has undergone major revisions for JMP 12, adding extensive new examples and information on new capabilities. (By the way, did you know that you can search for JMP Help online? JMP Help and .PDF versions of JMP Books are available on jmp.com and can be searched there or from any search engine.)

Like many of my colleagues, I work for JMP, but I am also a JMP user. I consider myself a relative novice, but like many of you, I learn more of JMP with each release cycle. I use our software nearly every day to retrieve, visualize and analyze data that is important to my job and to my life outside of the office. Early in JMP 12 development, I began to extend my JMP skills by learning JMP Scripting Language (JSL), and presented a poster at JMP Discovery Summit 2014 detailing how I used JSL to import nearly four years my personal diet and fitness data into JMP and visualize it in Graph Builder. I even used the new Export to PowerPoint feature extensively to create my Discovery e-poster, which you can find in PDF format in the JMP User Community. I have been blogging about my project in a series, and I showed an example of how I added images to my weight tables and graphs (another JMP 12 feature) to explain the motivation for my project.

Many of the JMP 12 features I adopted when working on my Discovery Summit e-poster have proved to be incredibly useful in daily work with internal data sources. The revised JMP 12 Recode platform is just as helpful for cleaning up data about software bugs and code pushes as it was for consolidating item names in my food log data. I use new JMP 12 axis options daily to customize the appearance of my graphs, whether I’m visualizing data on my sleep, activity and eating patterns, weight training data, or creating internal graphs for colleagues to summarize progress in the software release cycle. I’ve also been playing with the new selection filtering feature that lets me use a graph to filter another graph and exploring the new Query Builder with internal SAS and SQL data sources.

Be sure to watch for more information about JMP 12 in this blog in the coming weeks. JMP 12 will be officially launched at the JMP Discovery Summit Europe conference in Brussels in March. We look forward to hearing your thoughts on the new version, and thanks again for using our software.

Editor's note: This post is part of a series of previews of JMP 12 written by the people who develop the software.

Post a Comment

Happy little trees: An updated Forest Plot Add-In

Recently, I experienced an event that brought me back to my childhood. I was having dinner at my brother's house with my sons. The television was on, and at some point Bob Ross' show "The Joy of Painting" came on the air.  If you've never experienced the show, it is very entertaining to watch. Bob starts with a blank canvas, and in 30 minutes' time, paints a beautiful landscape using a carefully selected palette of oil-based paints. The show is extremely relaxing, due in part to Bob's soft-spoken instruction and easygoing manner. Though Bob usually had a idea of what he wanted to paint in any given episode, nothing was ever set in stone, and a "happy little tree" or two (as Bob would call them) could end up anywhere in the canvas to fill some space. My brother and I couldn't help but get sucked into watching the show. Initially, our children seemed bored with it, but eventually were also pulled into Bob's world. It was a really nice moment.

Where I am going with this? Well, many happy little trees makes me think of a happy little forest, which make me think of a happy little forest plots.

In all seriousness though, shortly after revisiting the land of happy little trees with my family, I received a request from one of our JMP users about the Forest Plot Add-In. The user wanted to have the ability to size the center bubble using another variable available in the data table. This feature would be particularly useful for summarizing results from a meta-analysis, allowing you to indicate the size of each clinical trial in the plot. You may want to check out a previous blog post on forest plots. So I've updated the add-in to include this feature.

I'll illustrate this new feature with the data table in Figure 1, which shows 95% confidence intervals of the log2(relative risk) for 11 adverse events from the Nicardipine clinical trial.  Count is the number of patients that experienced the event during the study.

Figure 1. Data table of 95% CI for log2(relative risk) for adverse events from Nicardipine Trial

Figure 1. Data table of 95% CI for log2(relative risk) for adverse events from the Nicardipine Trial

The dialog of the forest plot add-in has some additional features (Figure 2). First is the Marker Size variable in the column role section of the dialog. Second is the set of Size Preferences at the bottom of the dialog. The area of the marker is proportional to the values of the Marker Size variable, Count in this example. Count is first scaled to be between 0 and 1 (Count01, say). The radii of the circles drawn are equal to square root (Scale x (Count01 + Minimum)). Minimum = 0 implies no symbol drawn for the records where Count has the minimum value. Running this example produces Figure 3. Transparency makes it possible to easily see narrow confidence intervals that are contained entirely within larger markers (such as vasoconstriction below).

Figure 2.  Forest plot dialog

Figure 2. Forest plot dialog

 

Figure 3.  Forest plot applying a size variable

Figure 3. Forest plot applying a size variable

While forest plots are useful for summarizing data from a meta-analysis, they are also useful for summarizing results from various subgroups of patients. While this particular example doesn't focus on subgroups, I'll add some more detail that can also be useful for subgroup forest plots. Click the red triangle > Show Control Panel. This allows you to further manipulate the figure in Graph Builder. Drag AE Class over to the y-axis (Figure 4).

Figure 4.  Labeling adverse events with body system

Figure 4. Labeling adverse events with body system

Clicking Done provides Figure 5.

Figure 5.  Adverse events grouped within body class

Figure 5. Adverse events grouped within body class

You can edit this a bit by right-clicking on y-axis, clicking Axis Settings, adding Lower Frame for both labels, and selecting Long Divider for Tick Mark Style for Label Row 2. This produces Figure 6. You can use either figure to summarize additional detail regarding the classification of various intervals summarized within the figure. This can be useful for summarizing subgroups, such as Gender (Male, Female). Note that ordering of the data table is important prior to using the add-in. Terms must be ordered the way you want them to be displayed in the forest plot. If classification is used as in Figures 5 and 6, terms must be sorted by class; the classes must be in the appropriate order; and the terms must be in the desired order within class. Selection of the bubbles is still possible, though you should select them in the center of each bubble. Using the drag-and-select feature of JMP (as one would use to select several points) is likely the most straightforward approach.

Figure 6.

Figure 6. Adverse events grouped within body class with long dividers

You can download this updated Forest Plot Add-In from the JMP User Community (join the community to download the add-in).

Post a Comment

An essential book for new JMP users

Cover of the book JMP Essentials Are you new to JMP or just use it occasionally to explore and analyze your data? There are many resources that can help you quickly get your work done with JMP, including documentation, webcasts and the User Community. But sometimes, the best thing is a book that guides you, step by step.

JMP Essentials, by Curt Hinrichs and Chuck Boiler, is that book. It's for the new user and occasional user of JMP who needs to get the right results from their data right away. It's designed like a cookbook, Curt and Chuck said: "Find what you need and follow the steps." After a successful initial publication of the book four years ago, the book is now in its second edition.

Curt has led the Academic team for many years and knows all about the learning needs of university students as they use JMP to understand statistics, data analysis and data visualization. Chuck leads the group of systems engineers at JMP and has the inside scoop about commercial customers' experience of the software. In addition, you may know Chuck as the founding instructor for the Getting Started With JMP live webcasts for beginners. So this pair are the perfect authors of a book for new users.

The book is available for purchase online.

Curt and Chuck told me a bit more about the new edition of book:

In your experience, who is using JMP Essentials? Who should use it?

We wrote JMP Essentials principally for the new user of JMP, with the goal of getting that person up and running and generating meaningful results quickly. In the first edition, we have found this intended audience to include commercial customers transitioning from another statistical software, students in introductory statistics courses and quite a few spreadsheet users who want to go to that next level of data visualization and analytic capability. While JMP is not difficult to use by any measure, its navigation is different from other statistics packages or spreadsheets, and we have tried to present this along with the “essentials” of JMP in the most efficient way. In fact, much of the coverage in the book has been influenced by the Getting Started with JMP weekly webinar that Chuck hosts and the needs that I typically see among new users in the academic community.

Why did you write a second edition of JMP Essentials?

We are grateful that the first edition was well-received and that readers reached out to give us feedback on the book. That feedback prompted us to begin considering how we would approach a second edition. Plus, the first edition was written with JMP 8 and now – being on the eve of version 12 – we found many areas in need of revision or addition that are reflected by the four versions of JMP that have been released since. Mind you, we have tried to keep these revisions/additions true to our essentials of JMP scope. There have been many useful  enhancements to JMP in the areas of statistical modeling, reliability, design of experiments and consumer research in these releases, but these areas are beyond the scope of the book and covered especially well by other books and JMP documentation.

What’s new and different in this edition? How different is it?

You may notice that we’ve gained a little weight in the second edition! That is, the page count has increased by about 25 percent. This wasn’t intentional but is reflected in many of the sections we revised for accuracy, expanded for clarity or added to the book. Even after extensive editing, the content needed to grow just as the audience for data analysis seems to be exploding. The additions to the second edition should serve these new audiences. We have added eight new sections to the book that we think will appeal to many users; they include Filtering Data, Creating Maps, Using the Excel Import Wizard, Combining Windows Into Dashboards, and Sharing Dynamic Graphs with HTML 5.

Can you give us an example of a technical tip that is in the new edition?

It's hard to pick just one tip, but one that we think will be a real time saver is the support in Graph Builder for background street maps with density contours. In the context of positional mapping, seeing the frequency and density of occurrences of anything with a street map can be very enlightening. This is especially relevant with the advent of handheld devices that provide location data, which are becoming ubiquitous. The San Francisco Crime map example in Chapter 4 provides an introduction to this useful feature that also employs filtering, which is covered in Chapter 2. All you need are data where latitude and longitude have been recorded.

Post a Comment

Exploring and visualizing sleep data with JMP

My BodyMedia® FIT® armband uses a variety of sensors to monitor my movements, including an accelerometer and gyroscope. When plugged into a computer via USB or connected via Bluetooth to an iPhone, the data stored on my device is uploaded to BodyMedia® servers and an algorithm runs to determine when I was active, lying down, awake or sleeping for each minute of monitoring time. I get an estimate of how long I slept, and I can view a minute-by-minute visualization of the time I spent lying down, awake and asleep. (You can get more details about BodyMedia’s sleep classification algorithm in a white paper.) I can extract the data for these daily sleep summaries from the activity summary files, as I described in an earlier blog post.

I wanted to analyze and create some effective data visualizations of my sleep data. An early version of my JMP Discovery Summit e-poster included a multivariate scatterplot of sleep duration, sleep efficiency and time spent lying down during~1,300 nights. I colored points red indicating nights with sleep durations <4 hours. Since I rarely sleep so few hours, I suspected some or all of those low outliers to be infrequently occurring data collection errors. On the other hand, I knew that the highest sleep day was an accurate measurement from a recent bout of the stomach flu!

Scatterplot Matrix

By the time I met with Xan Gregg, lead developer of Graph Builder and head of the JMP Data Visualization group, I had switched to using box plots to visualize my sleep data. The box plot view told a much more interesting story since it showed data over time and gave me a better sense of the variability of my sleep measurements within months. Also, the addition of a time dimension clearly showed that the big drop in my sleep efficiency at the end of September 2011 coincided perfectly with my son’s birth!blue and orange sleep box plot2

When I showed this graph to Xan, he recommended:

  • Lightening the colors used in the box plots.
  • Darkening the smoother lines.
  • Adding annotations to point to important trends.
  • Changing my X axis labels to a horizontal orientation.

We both thought that the final version of my sleep graph was more visually appealing and easier to interpret.

Teal and purple with anno final

I was surprised to see how much my sleep patterns varied with the time of year, as you can see from the box plots and the up-and-down pattern of the smoother line in the sleep duration graph on the bottom. In contrast to my activity measurements, which I know are strongly affected by the fact that I wear my armband less during the day in summer, I wear my armband to bed almost without fail. Seasonal variation in sleep patterns is actually a commonly recognized phenomenon in human sleep, yet I had completely missed it in my own data up till now! This pattern was not obvious when I viewed the standard summary reports of my sleep information within days, weeks and months.

It's easy to see how useful long-term data on nightly sleep patterns could be if you are chronically tired. Although it may not be as accurate as a sleep study done by a medical professional, accumulating nightly data over a long period of time with a sleep-monitoring device could help you assess how much sleep you need to awake feeling rested. If you have insomnia, you can use your sleep data to see when and how long you are awake in the night. If you battle sleepiness during the day, you can change your behavior in various ways and assess the outcome on your sleep.

Experts recommend various sleep improvement strategies such as cutting out or limiting caffeine, going to bed earlier, getting up when you wake up rather than hitting snooze repeatedly, adding exercise to your day and avoiding the use of screens close to bedtime. Sleep monitoring gives you actual outcome data to assess which of these strategies may work to optimize your own sleep patterns. Some sleep monitors even can wake you at an optimum time given where you are in your sleep cycle. Since starting to monitor my sleep, I have rarely used a morning alarm. When I wake up, I assess whether I am still tired, and if not, I check how many hours I slept. If I believe I have had enough uninterrupted sleep based on these two assessments, I get up and start my day.

By examining my food log and sleep data, I have discovered that in addition to seasonal variations, a number of other factors appear to affect how well I sleep. I sleep much less when I am experiencing stress at work, and a few days without exercise can hurt my sleep. What and when I eat can also impact my sleep quality.  I have observed that the stimulant kick provided by chocolate Greek yogurt (my favorite breakfast) disrupts my sleep if I eat it as an evening snack. Perhaps because of my background in biochemistry and genomics, I wasn't satisfied by simply observing this connection. I had habituated to drinking caffeine late at night in the past without such a negative impact on my sleep, so I wondered what was different here?  I did some reading about cocoa powder, and it turns out that it contains caffeine and a related stimulant called theobromine. The same cellular pathway deactivates both these chemicals, and I know from other test results that I have the slow form of a major enzyme involved. I suspect that when I eat chocolate Greek yogurt close to bedtime, I can't metabolize enough of the stimulant chemicals it contains before I go to sleep, and as a result, I sleep poorly throughout the night.

To reproduce my sleep graph with your own data in Graph Builder, open a data table containing sleep duration and/or sleep efficiency measures in hr:m format by date. If your data table doesn't have Year and Month variables, you can create these from the Date variable in your table by right-clicking on it in the Graph Builder variable chooser and adding new transform columns from the Date Time menu. I used a Value Ordering property on the Month variable to create a Month Name column and made sure that Year and Month Name were specified as Ordinal.

To create the graph, open Graph Builder and drag:

  • Year to the X axis.
  • Month Name to the X axis (just above Year, so the axis shows Month Name nested within Year).
  • Sleep Efficiency to the Y axis.
  • Sleep Duration to the Y axis just below Sleep Efficiency so they appear in separate sections.

To complete the graph, change the element type to Box Plot using the icon at the top of the window, adjust your Y-axis, graph title and axis titles if desired, and add one or more annotations from the Tools menu. You can right-click on annotations to change their appearance to match your graph like I did. Stay tuned for the next few posts where Xan and I show how he helped improve other of visualizations of my data!

You can learn more about my interests in quantified self data analysis in this blog post here, see an e-poster on my activity and food log data import project in the JMP Discovery Summit 2014 User Community here, and read about how I imported my BodyMedia® Activity Summary files from Excel here and Food Log files from text here. I used JMP to recode food item names and classify foods into categories, and then used my data to characterize my activity and meal logging patterns. You can download a JMP add-in from the JMP File Exchange here to import your own BodyMedia® activity summary files and food log files or CSV formatted food log files from the free MyFitnessPal website.

Post a Comment

Hear from analytics thought leaders (and get their books too)

Analytics thought leaders are busy folks these days as more organizations are finding analytics are key to their success. But once a month, a different analytics expert makes time to have a conversation with my colleague Anne Milley for the Analytically Speaking webcast series. Their discussions cover statistics, design of experiments, quality engineering, data visualization, and consumer and market research.

We’ve compiled some of the highlights from the series in a webcast we're calling Analytically Speaking Featuring the Best in Show. It will be shown Wednesday, Dec. 10, 1 – 2:30 p.m. ET. This webcast premiered on December 10. It is now available on demand. The best in show includes:

  • SAS co-founder John Sall on the importance of the statistical discipline.
  • Behavioral economist and author Dan Ariely on why more businesses don’t experiment.
  • Statistical Thinking authors Ronald Snee and Roger Hoerl on the relationship between quality and reliability.
  • Popular bloggers Kaiser Fung and Alberto Cairo on the elements of good data visualization.
  • Words of wisdom for aspiring data analysts from esteemed statisticians David Salsburg and Stu Hunter.
  • And, of course, Professor Dick De Veaux on how he became the official statistician of The Grateful Dead!

There were so many good moments that this webcast runs longer than the usual hour. You have the option to watch the webcast in its entirety or segment by segment, as your time permits.

If you do watch, we hope you’ll leave a thoughtful comment below about your favorite moment from the webcast. The first 16 to participate will qualify to receive a free book from the following selection of titles (some of which are signed by the authors):

Numbersense: How to Use Big Data to Your Advantage, Kaiser Fung

Optimal Design of Experiments: A Case Study Approach, Peter Goos and Bradley Jones

The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century, David Salsburg

Statistics for Experimenters: Design, Innovation, and Discovery, George E.P. Box, J. Stuart Hunter and William H. Hunter

The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day, David J. Hand

The Innovator’s Hypothesis, Michael Schrage

Data Mining Techniques: For Marketing, Sales, and Customer Relationship Management, Michael J.A. Berry and Gordon S. Linoff

Statistics for Business: Decision Making and Analysis, Robert A. Stine and Dean Hunter

Here's how the book giveaway works: After watching the Best in Show webcast, tell us your favorite moment from the webcast, here in a comment. Your comment must be between 50 and 75 words long. Be sure to enter your e-mail address when you write your comment so we can contact you if you are a winner. Only one book per commenter. Commenters must reside in the US to be eligible to receive a book. The first 16 eligible commenters will win a book!

Post a Comment

John Sall named AAAS Fellow for contributions to statistical sciences, software

John Sall, co-founder and Executive VP of SAS, and lead architect of JMP softwareJohn Sall, SAS co-founder and Executive Vice President, has been named a Fellow of the American Association for the Advancement of Science, the world’s largest general scientific society.

Sall was elected “for distinguished and visionary contributions to statistical sciences and software development, having the greatest impact on businesses, education, engineering and all other sciences,” the AAAS said.

Sall leads the JMP business unit, which creates interactive and visual statistical discovery software designed for scientists and engineers. He frequently speaks about statistics at universities and conferences. Sall developed many of the earliest analytical procedures for Base SAS software and had significant roles in creating other SAS products, including SAS/ETS, SAS/IML, SAS/OR and SAS/QC. Sall also developed JMP software and remains its lead architect.

"John has performed extraordinarily broad and deep work at SAS for more than 35 years – work that has had a powerful impact on science across the globe,” said Russ Wolfinger, Director of Research and Development for Genomics in the JMP business unit and a 2012 AAAS Fellow.

Sall said the work of his team at JMP is to make statistical modeling as friendly, accessible and informative as possible, and to make experimentation as efficient as possible.

“Doing well at these goals will contribute greatly to scientific discovery and engineering breakthroughs,” said Sall. Along with SAS Analytics software, JMP helps organizations deliver the value of analytics by putting data to work for solving problems, making better decisions and improving our world. “The value of statistics is realized when it is translated, taught and delivered with accessible computing environments to the scientists and analysts who use it," Sall added.

Sall recognized the leadership in statistics at North Carolina State University for creating the environment from which SAS launched. "Thank you especially to three statisticians whose exemplary leadership at NCSU put them on the path to becoming university deans: Dan Solomon, Blan Godfrey and Sastry Pantula," Sall said.

Pantula, Dean of the College of Science at Oregon State University, called Sall a true visionary and a great promoter of science. "Statistical sciences are at times invisible, but are having an impeccable impact on innovation and discoveries. The software behind it is even more invisible, but the software developed by John and his colleagues at JMP is having a great impact on business, education, engineering and sciences throughout the world,” Pantula said.

J. Stuart Hunter, Professor Emeritus at Princeton University’s School of Engineering and Applied Science and an expert in design of experiments, said: "John Sall has spent his entire career immersed in the advancement of science and engineering. His career of many decades contains myriad contributions to the statistical solution of engineering and social problems. His personal contributions to computer software programs adapted to statistical data analysis are simply outstanding. His recent innovations in graphical displays of masses of multivariate data has added meaningfully to the art of finding signals in noise."

Post a Comment

Holiday book gift ideas for the analytically minded

The gift-giving season is approaching, and it’s time to start thinking about the quantitatively inclined people on your list. A few years ago, I wrote a post offering a list of books. Many new analytical books have been published since then, and there are some classics worth revisiting. So I wanted to list some more recent books and make sure you know about the recommended reading page of the Analytically Speaking webcast series (now in its third year). Many thought leaders from this webcast series — several of whom are authors — share books they recommend, and we continually update that reading list.

The first three books on my list below are from three of our featured keynotes at Discovery Summit this past September. You can view the plenary talks for David Hand and Michael Schrage to get a sense of what their recent books cover. The recording of the speech by Jonah Berger will air in January.

For those interested in analytical concepts — “listening for the melody” versus making the music:

For those seeking more recent views on data visualization:

For big data enthusiasts:

And for those who enjoy books on methods, techniques and more focused topics:

Among the holiday catalogs (arriving since October) was a book catalog with an appropriate plaque: “Life is short. Read fast.”  If you have some books you’d like to suggest, leave me a comment. Happy reading!

Post a Comment